How to scale activejob-google_cloud_pubsub worker?

Question

Overview

I have a Rails ActiveJob that uses the gem activejob-google_cloud_pubsub. (Short usage tutorial). My application is deployed to AppEngine on a ruby flex environment. While my application works as expected, I am having difficulty scaling the worker services that consume the messages from Google Cloud PubSub. The gem's README docs mention a way to specify a Worker's max_threads from the terminal but not from within the rails app. As I am reading the gem's source code I see the initialize for the worker takes in that variable max_threads (which default to the number of cores). The problem is that I don't see how to initialize the worker with a default configuration so I can increase the thread number.

Questions Summary:

1 - How to initialize the activejob-google_cloud_pubsub Worker with a higher number of threads?

2 - Would it be unadvisable to initialize the worker with more threads than processors in the machine?

Other Details:

• I am deploying the worker on a separate service from the Rails Server by using a worker.yaml apart from the app.yaml
• The worker service is being scaled manually to 3 instances, still this are barely handling 10 messages per second (lean message processor, only log to output and return)

Any Input into how I can scale this type of queue processing would be helpful.

Answer 1

1 - How to initialize the activejob-google_cloud_pubsub Worker with a higher number of threads?

worker.yaml

Add the terminal flag --max_threads as described in the gem's README

runtime: ruby
env: flex
service: <xxx>
entrypoint: bundle exec activejob-google_cloud_pubsub-worker --max_threads=5
...

This should create 5 worker threads in every instance. The default is the number of logical cores.