I have a task that will take a long time so I split it into 3 parts and want to launch three threads that will work on it concurrently (I made sure there isn't any accessing of the same variables or anything, don't worry, they strictly handle their own datasets).
As far as I can tell sidekiq launches a new thread for each worker, so I made three workers importer,importer2,importer3, all in app/workers. In one of my controllers I have this code:
Importer.perform_async(arrays[0], date)
Importer2.perform_async(arrays[1], date)
Importer3.perform_async(arrays[2], date)
render json: 1
My question is: Is that the best way to handle this?
It seems odd that a) the request to the controller would take so long to render the 1 and in the sidekiq log I can see Importer JID-639e67d2aa20cce885690dc7 INFO: start as well as the same for Importer2 but not 3 and then then sidekiq just exits with killed
When I relaunch sidekiq, I get the Importer3 ... start and it then is the only one working (it updates a DB value and it is the only one changing`
Any ideas why?
Are you sure you have enough memory? Maybe this can be helpful: Debugging Mystery Sidekiq Shutdowns