everybody,
We're working on a project for university. The project should run in the cloud and be highly scalable. We outsourced all functions to Google Cloud Functions, which works great. Now we're looking for alternative implementations that work even better for our use case or can scale better.
That's why we're looking for some different approaches to solve this use case: It's a web scraper where many parallel requests from different users go to that system. The system then scrapes the data parallel from different websites and parses the content. Subsequently, the results are reassembled and sent back to the users. What I've described here as "the system" is a huge collection of cloud functions/ micro services which are doing their part.
I have no experience with Kafka, but we have thought about using stream processing. Are there other frameworks that we should consider? Any suggestions?
Thanks in advance!
For what you are describing you should check:
All 3 products should be okay for your use-case but each having its own pros/cons.
Once you read through the description of each one and/or watch the intro videos, you should be able to determine which one is the best for you.