I have a web application that accepts csv files (large files > 2000000 rows), performs statistical analysis and creates a csv file with an equal number of lines as an output. The whole process of the lines loop is inside a job. I am using laravel 6.9. My queue driver is set to database. this is what happens:
I submit my file and parameters for my analysis via jquery. The server picks the request, dispatches the job using something like this in my controller:
IndexEstimatorJob::dispatch($indexEstimator)->delay(now()->addSeconds(3));
The browser receives a response, which means it runs async, and redirects me to a page where i want to view the jobs that are in progress. The thing is, while the job starts async and i get redirected to a different page, when i try to visit another page my browser is waiting for the server to respond forever.
Is that a server resources problem? Can i set the job priority somehow so that it is slower but does not render the server unresponsive?
Any ideas?
Well thank you for your comments people. I posted this question mainly to get a comment about my use of the queue since I have not used it alot so far. The culprit was the server after all. It seemed that cpu usage maxed during the parsing of the file so the server was unresponsive until the calculations wre complete. By doubling the cpus allocated, the problem was resolved so I guess I used the queue correctly.
Thanks again for your time.
P.S. @AnujShretha no it was not a memory issue.I have payed alot of attention to how i parse the file and each line so i wouldnt waste resources, it was the cpu which cannot be calculated as easily. Thanks again