I'm processing many Sidekiq jobs (tens of millions) which return 1 string and 2 integers as a result. There is a central computer which is hosting the Redis server/storing the results, and multiple computers acting as workers, fetching data from this central computer.
Inside the worker, I do the logic needed for processing and after the result is called, I just call:
Sidekiq::Client.push('class' => ResultsWorker, 'args' => [arg1, arg2, arg3])
and on the central computer, I just have another worker which just checks this queue and stores the results in a database.
I haven't seen such a pattern of "two-way communication" being mentioned anywhere in this Sidekiq community. So I was wondering, is there a reason for this? Is there some side-effect of doing this? I've tried putting a simple web app on the redis server comp which accepts the results and stores them, but I figured that for such small data, why not re-use the same Redis server with a different queue?
I haven't seen such a pattern of "two-way communication" being mentioned anywhere in this Sidekiq community.
This doesn't need any special mentioning or discussion, because there's nothing special or unusual about this. A job can indeed enqueue some other jobs. This happens left and right in big apps.
Although you should measure the overhead of this concrete situation in your case. It might turn out that persisting to the database directly from the first job will be much faster (no losses on making network requests, etc.)