Our system generates various invoices every hour and uploads those to the cloud. Also it is possible to create the invoice on demand by clicking a button on our frontend.
When manually requesting to create the said invoice then it never fails to upload. As for the cron generated invoices after some time all uploads fail with:
Client error response
[status code] 401
[reason phrase] Unauthorized
[url] https://storage101.dfw1.clouddrive.com/v1/MossoCloudFS_930575/ <...> .pdf" @ Guzzle\Http\Exception\BadResponseException->factory
Which should mean that the token has expired which it probably is. Rackspace tokens last for 24 hours but the Laravel's Storage should automatically refresh the token.
Now here for some fun facts:
1) Every time our code is deployed with Capistrano the token seems to get refreshed and cron uploads work again for some time. Important thing to notice here is that every deploy creates a new folder similar to this /releases/201605190925 pulls the code, installs dependencies from scratch and if all goes well then symlinks this folder with the one Apache is showing.
2) Laravel jobs get handled on a different process than www-data
3) It's hard to track down if after deploy the upload works more than 24 hours or not. I suspect that sometimes it works more than that. But it's hard to track down since not every hour there are invoices that need to be generated. There are more developers who deploy than just me etc.
4) When cron fails and I get the failing notification I can immediately go and successfully generate the invoice. After that the cron still keeps failing. So it seems like these two instances have different tokens stored somewhere.
5) Cache cleaning with php artisan cache:clear doesn't seem to have any impact
6) Probably have tried restarting Apache service but without results
Since it has been going on for some time now I have tested various things and even contacted Rackspace at one point but they couldn't find anything odd from their end... Reminded me simply to catch the 401 error, update the token and try again. But Laravel and Flysystem should handle it somewhere by itself
Since nobody else seems to be having similar problems with Flysystem nor Laravel nor Rackspace I suspect that it's some kind of unique problem with the cron process. I was simply hoping that soon we have a refactored version of our system ready and the problem simply goes away. As for now it's still in development and might take another month.
Don't think it's related to code but anyway here's the upload line:
Storage::put($folder . '/' . $filename, file_get_contents($filePath));
Here's our config:
'default' => 'rackspace',
'disks' => [
'rackspace' => [
'driver' => 'rackspace',
'username' => ' ... ',
'key' => ' ... ',
'container' => ' ... ',
'endpoint' => 'https://identity.api.rackspacecloud.com/v2.0/',
'region' => 'DFW',
]
]
Any thoughts on the matter are appreciated.
Here I was once again trying to find a solution to the problem almost 2.5 months later. And I think it really clicked this time when I reread @Tommmm update.
So you restarted the worker queue and I started to think about the way I was running my supervisor/workers.
php artisan queue:work database --daemon --sleep=10 tries=3
--Daemon (as it was really nicely explained in this StackOverflow post by @the-shift-exchange )
In Laravel >=4.2 there was a --daemon command added. The way it works is simply keeps running the queues directly, rather than rebooting the entire framework after every queue is processed. This is an optional command that significantly reduces the memory and cpu requirements of your queue.
The important point with the --daemon command is that when you upgrade your application, you need to specifically restart your queue with queue:restart, otherwise you could potentially get all sorts of strange errors as your queue would still have the old code in memory.
Since the application never gets booted up again I feel that the token is also never renewed. As for the rest of the application still renews the token and uses the new one. That explains why all on-click actions were working and only the background tasks were failing. As to why deploying temporarily fixed our problem was that we had queue:restart in there after every deploy.
I am quite certain that this is the case and now there are two possibilities to start with. Either you restart your workers after some time (<24h for Rackspace) or you don't run the workers together with --daemon.
Edit:
Confirmed that adding following to Kernel.php restarts daemon workers after every 6 hours.
$schedule->command('queue:restart')->cron('0 */6 * * *')