I have some rake tasks that need to run every hour approximately, each one of them has to contact a web site, download a file and import the file into my db.
I was wondering what the best approach would be in terms of efficiency and speed of execution. Am I correct if I think that making a thread for each task will save me some time? By doing so I should be able to minimize the time spent waiting for the server response to that of the slowest one among the threads.
Once downloaded the files, I was thinking of using gem "parallel" for importing data into the db from large csv and xml files.
Please let me know should you have any suggestions.
When number of such files is low, you do not care for order of execution and can afford some extra memory - simpliest solution is just to run them in different processes by cron (for example - gem 'whenever').
If there're more - use some http gems for parallel downloading - typhoeus, curb, em-http-request etc