Right now I'm running 3 resque workers to run the background jobs. My system has around 50K users and there are many jobs pending in the Queue.
You can scale up the number of workers you run to as many as you would like as long as the computer you're running them on can handle the load. This number is going to vary from machine to machine and application to application; you'll have to do some testing to see what the limits are for your application and hardware (though in general, if your workers load your Rails app, memory tends to be the main constraint).
Keep in mind that the workers don't have to run on the same box that runs the Rails application. We are currently running 11 workers over about 9 queues (9 of the workers are dedicated to one or more specific queues, while 2 of them are wildcard workers) on a dedicated server, and we are considering upping our worker count by either getting a beefier EC2 instance or firing up a second instance to run additional workers on.