I have an APNS notification server sent up, which would in theory every day send about 50,000 to 100,000 users a processed notification (based on the amount of users of our web app that ties in with the iOS app).
The notification would go out around 2, but it must send it to each user individually (using Urban Airship) and is triggered by curl on a cron job.
It iterates through each user and has to use an HTML scraper (simple_html_dom to be exact) which takes about 5-10s per user, and is obviously very memory intensive. A simple GET request cant be the right way to come about doing this, in fact im positive it will fail. What is the best way to handle this long, memory intensive task on a cron job?
If You will reuse same variables or set ones You are not going to use any more to null You won't run out memory. Just don't load all data at once and free it(set to null) or replace with new data after You process it.
And make sure You can't improve speed of Your task 5-10s sounds really long.