I would like to know if fileobject can be sent to sidekiq worker, I have tried this out and it works fine, I am concerned about the performance, for example if I have a CSV with 1000 rows, then should I send the file object for e.g.
SidekiqWorker.perform_async(params[:bulk_update_csv_file].read)
This approach works but as the file consists of a lot of rows, I am concerned in terms of performance if that fileobject should be passed as an argument to sidekiq worker.
I have tried a few other things like using
CSV.parse(params[:bulk_update_csv_file].read, headers: true) and then slicing the object in around 500 rows and then sending that CSV parser object with 500 rows to the sidekiq worker but I am getting an error SystemStackError (stack level too deep):
Another approach that I have tried is to convert each line into a hash and then send it over to sidekiq worker which also works fine. It would be great to know what would be the best approach here, even if there are 10000 rows.
My last resort would be to save the file and send the link of the path to sidekiq worker, but it would be great if this could be done without this method.
From the Sidekiq Wiki:
Don't save state to Sidekiq, save simple identifiers.
The contents of your file would be state, the path to the file would be a simple identifier.
Conclusion: Send the path to the file and let the Sidekiq worker read the file.