I installed nginx and passenger with the command:
rvmsudo passenger-install-nginx-module
Is there any nifty way to remove it easily?
nginx should be self-contained in /opt/nginx or whatever path you chose during install.
You can just remove the folder and everything should be ok.
Edit : Or remove passenger altogether : gem uninstall passenger