Cannot access a file from worker, but works in the view - [Errno 2] No such file or directory

I have followed the tutorial on https://devcenter.heroku.com/articles/python-rq

Let's say I have a function that converts PDF Files to JPG and stores them in the app's tmp/file/ and that it later uploads it to S3.

However, since the request timeout is 30s on heroku, I am not able to put this in a View, so I decided to use a background worker to execute the task. So I moved the code to the task but the worker is not able to find the right path.

[Errno 2] No such file or directory. I tried printing without luck, but I'm able to print in the function.

The function I tried calling that gives an empty string in the task but gives the right value in the view issorted(os.listdir('tmp/file/')

I searched around and thought that maybe PYTHONPATH variable could be wrong, but everything works via the manage.py

My worker.py looks like this.

import os
import redis
import sys
from rq import Worker, Queue, Connection

listen = ['high', 'default', 'low']
os.environ['DJANGO_SETTINGS_MODULE'] = 'cinnamon_rest.settings'

redis_url = os.getenv('REDISTOGO_URL', 'redis://localhost:6379')

conn = redis.from_url(redis_url)

if __name__ == '__main__':
    with Connection(conn):
        worker = Worker(map(Queue, listen))
        worker.work()

And my manage.py looks like this.

#!/usr/bin/env python
import os
import sys

if __name__ == "__main__":
    os.environ.setdefault("DJANGO_SETTINGS_MODULE", "cinnamon_rest.settings")

    from django.core.management import execute_from_command_line

    execute_from_command_line(sys.argv)

Solution

The worker and web dynos do not share the file system; they are like containers (I Heroku uses LXC containers behind the scenes, just like Docker). Files that are present in the git repo or generated by a build process will be present (since they are assembled into the "slug" used to boot the container), but any files that are written dynamically will only exist in the file system seen by that particular container. You will need to use an external storage mechanism to share file/blob data between containers, such as https://django-storages.readthedocs.io/en/latest/