Here is the error line
ERROR - Failed to run aws glue job, error: An error occurred (ResourceNumberLimitExceededException) when calling the StartJobRun operation: Reached the limit while starting jon run while starting a run for job with name: my_glue_job_test
The error occurs when StartJobRun is called and all I found about this error from docs is "A resource numerical limit was exceeded" which doesn't make any sense to me because last week it was working totally fine. I did not change anything and this error show up out of nowhere.
Also I think this is a different case from How to fix ResourceNumberLimitExceededException when creating Glue job in AWS? since I tried to create a new job and create_job is working totally fine. However, when StartJobRun is called the exact same error show up.
Does anyone know how is this happened?
I faced the similar issue today, Just go to aws service quota and change all the quotas that got changed to 0 to its default value. It worked for me! This issue is caused by aws internally.