Search code examples
node.jsmongodbexpresscron

Best practice for automatically updating database everyday?

I have an original source, Source A, where I am fetching data from. I am reformatting and filtering the data from Source A, and storing it in Source B.

I have created a REST API using Node/Express that uses Source B.

Source A gets updated once every day. I want to update Source B at the same rate. What is the best way to do it?

Ideas so far:

  1. For every API call I get to my server, before returning the data, check if the data was last updated within a day. If not then update the data and send it. This would mean that one API call per day would be extremely slow.
  2. Perform Cron Scheduling

I would like to know if there are more ways to do this and I would like a comparison of different ways? I would also like if any of you guys do something like that in production and what method has worked?

Note: In my case Source A is a CSV file on a github repo, and Source B is MongoDB collection.

Solution

  • The best case you can achieve here is automatic updates to the MongoDB collection whenever that github CSV file is updated. If you can hook your job execution into whatever event is triggering the CSV upload, you would be golden. If you have to go through github, look into github hooks and see if you can subscribe your workload to one of those events

    There is a nice option 3 that you can do with mongo, by the way. Mongo queues are great for scheduling jobs at precise intervals.