I'm building an app that parses the contents of a CSV into a number of "readings". It then posts these readings to an REST API server where it's added to a database.
So far so good (and the above is working). However, I'm conscious that there may occasionally be connectivity issues with the server (delays, or if the API goes down for any reason).
To address this, I planned on using Sidekiq to perform the transmission asynchronously. However, I have a couple of questions:
Am I best off creating an ActiveJob to perform the post and then just queuing this job when the CSV is parsed? I guess that, if there is a connection issue, it would eventually resume in the same order as they are added?
Do I need to tell Sidekiq to "retry" the job, or will that just happen automatically?
Finally, as Sidekiq uses Redis, should enable persistence so that if the app server crashes while there are items in the queue they won't be lost?
Is there anything else to consider? Or a better way of tackling this?
There are two scenarios
1) you are going to put the complete processing of the CSV file in on Job, in which case, I think it will not work perfectly as the if the job fails it will again loop over all the rows. unless you mark the rows by adding another column in the CSV say read.
2) You add each row as a separate Job in Sidekiq, I can only think one downside with this is that you may end up creating too many jobs, in case you have large CSV files, but it will save a lot of processing on CSV side.