I've recently started using an App Engine on Google Cloud Platform and have set up some cron jobs to get some scheduled work done. However recently one of my tasks took more than 10 minutes and it timed out... obviously I could break this work into batches or find another way around the problem, however I'm keen to not always be mindful of how long a job might take and want future jobs to run until completed or failed.
I've looked into various services that Google offer but with no success; Task Queue is Python 2.x only and Cloud Tasks has the same 10 minute limit unless you manually manage scaling (which I would prefer to stay automatic as that's the point of App Engine for me).
Am I missing something? This 10 minute limit seems like a big unnecessary blocker and I have no idea where to look.
https://cloud.google.com/tasks/docs/creating-appengine-handlers
Thanks for your time.
Google services such as App Engine are designed to model a web server HTTP Request / Response design. You are trying to use them as task/execute engines.
Use the correct service if you require long execution times, which usually means requests that take longer than a few minutes to complete. Use Cloud Tasks and Compute Engine. Otherwise you will need to architect your application to fit with App Engine's requirements and limitations.