To start off, I'm completely new to Azure development. I'm working on a school/enterprise project regarding knowledge-based management, and we'll be working with big(ish) data in an Azure environment.
Basically we want to collect data from different APIs, and possibly also use crawled data. Everything will be set up on a web application that is easy to use for the company we are working with. Basically we need to integrate the data we get into a user interface deployed in Azure. I'm not sure if we need to have a database/storage, or is there some Azure service that can just fetch and display the data using given parameters.
So my question is, which tools in Azure should I look into? There are so many different ones and they get kind of confusing, since it looks like many do "kind of" the same thing. What would be the best way to set up a crawler in Azure? Or those API requests and displaying them on a webpage?
Thanks!
I get it can be quite daunting at first!
As a starter for 10 you might want to look at Logic Apps for pulling data from a 3rd party API.
You also need to think about how "big" the data is, the structure of the data and how your web app will need to consume it.
Is it highly structured data that needs to go into a SQL DB, key/value data you could store in Storage Tables or something else entirely?
In terms of the web crawler are you going to roll your own code for this? You could look at a WebJob sitting on your web app or if you want to get fancy you could look at doing it with a timer based Durable Function.
There are many other things to consider but hopefully the above gives you at least something to start looking at.
There will always be at least a half dozen ways to do the same thing in Azure so don't assume you have to find the "right" solution, if you can find a solution that works for you in terms of ease of development, cost etc. then go for it!