What is the best way to consume APIs required for data processing/ data warehousing.
I have multiple data sources that I would like to obtain via APIs. Some of the modules have over 20 API calls others have 5-6. How do I get all this data for each system/ data source.
Would the best approach be to use Console Application like Azure Web Job Service, etc. Is that the most amount of work since it will require programming from scratch? Or should I use a middleware like Talend, Pentaho to consume these APis and transfer to a database?
Any help will be immensely appreciated.
The only benefit that WebJobs will give you is the ability to execute code on a schedule. You'll be writing your own code to pull data from various APIs and store it whenever you choose to warehouse the data.
In my opinion, if you've already got .NET code that hits these APIs and saves the necessary data written, then WebJobs would be a reasonable choice. If you've got to write the code to call APIs from scratch, I'd first look to a third party data integration tool such as Talend and see if it will work for you.