Collectstatic failing - botocore.exceptions.ClientError: An error occurred (404) when calling the HeadObject operation: Not Found

Question

I'm attempting to run collectstatic on a stage deploy to AWS. I'm getting the following error, and no files are being placed in the bucket:

Traceback (most recent call last):
  File "/home/hcc/.local/lib/python3.6/site-packages/storages/backends/s3boto3.py", line 523, in _open
    f = S3Boto3StorageFile(name, mode, self)
  File "/home/hcc/.local/lib/python3.6/site-packages/storages/backends/s3boto3.py", line 74, in __init__
    self.obj.load()
  File "/home/hcc/.local/lib/python3.6/site-packages/boto3/resources/factory.py", line 505, in do_action
    response = action(self, *args, **kwargs)
  File "/home/hcc/.local/lib/python3.6/site-packages/boto3/resources/action.py", line 83, in __call__
    response = getattr(parent.meta.client, operation_name)(*args, **params)
  File "/home/hcc/.local/lib/python3.6/site-packages/botocore/client.py", line 316, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/home/hcc/.local/lib/python3.6/site-packages/botocore/client.py", line 635, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (404) when calling the HeadObject operation: Not Found

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "manage.py", line 15, in <module>
    execute_from_command_line(sys.argv)
  File "/home/hcc/.local/lib/python3.6/site-packages/django/core/management/__init__.py", line 401, in execute_from_command_line
    utility.execute()
  File "/home/hcc/.local/lib/python3.6/site-packages/django/core/management/__init__.py", line 395, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/hcc/.local/lib/python3.6/site-packages/django/core/management/base.py", line 328, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/home/hcc/.local/lib/python3.6/site-packages/django/core/management/base.py", line 369, in execute
    output = self.handle(*args, **options)
  File "/home/hcc/.local/lib/python3.6/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 161, in handle
    if self.is_local_storage() and self.storage.location:
  File "/home/hcc/.local/lib/python3.6/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 215, in is_local_storage
    return isinstance(self.storage, FileSystemStorage)
  File "/home/hcc/.local/lib/python3.6/site-packages/django/utils/functional.py", line 224, in inner
    self._setup()
  File "/home/hcc/.local/lib/python3.6/site-packages/django/contrib/staticfiles/storage.py", line 501, in _setup
    self._wrapped = get_storage_class(settings.STATICFILES_STORAGE)()
  File "/home/hcc/.local/lib/python3.6/site-packages/django/contrib/staticfiles/storage.py", line 379, in __init__
    self.hashed_files = self.load_manifest()
  File "/home/hcc/.local/lib/python3.6/site-packages/django/contrib/staticfiles/storage.py", line 389, in load_manifest
    content = self.read_manifest()
  File "/home/hcc/.local/lib/python3.6/site-packages/django/contrib/staticfiles/storage.py", line 383, in read_manifest
    with self.open(self.manifest_name) as manifest:
  File "/home/hcc/.local/lib/python3.6/site-packages/django/core/files/storage.py", line 36, in open
    return self._open(name, mode)
  File "/home/hcc/.local/lib/python3.6/site-packages/storages/backends/s3boto3.py", line 526, in _open
    raise IOError('File does not exist: %s' % name)
OSError: File does not exist: static/staticfiles.json

My static files settings are as follows in settings.py:

STATICFILES_FINDERS = [
    "django.contrib.staticfiles.finders.FileSystemFinder",
    "django.contrib.staticfiles.finders.AppDirectoriesFinder",
]

STATICFILES_DIRS = [join(BASE_DIR, "assets", "build")]

DEFAULT_FILE_STORAGE = "staticfiles.storage.MediaStorage"
STATICFILES_STORAGE = "staticfiles.storage.ManifestStaticFilesStorage"

STATIC_FILES_BUCKET = "stage-bucket"
STATIC_FILES_LOCATION = "static"

STATIC_ROOT = join(BASE_DIR, "static")
STATIC_URL = f"https://{STATIC_FILES_BUCKET}.s3.amazonaws.com/{STATIC_FILES_LOCATION}/"

MEDIA_FILES_BUCKET = ""
MEDIA_FILES_LOCATION = ""

MEDIA_ROOT = join(BASE_DIR, "media")
MEDIA_URL = "/media/"

The custom storage classes that I'm using are:

from django.contrib.staticfiles.storage import ManifestFilesMixin
from django.conf import settings
from storages.backends.s3boto3 import S3Boto3Storage


class ManifestStaticFilesStorage(ManifestFilesMixin, S3Boto3Storage):
    bucket_name = settings.STATIC_FILES_BUCKET
    location = settings.STATIC_FILES_LOCATION


class MediaStorage(S3Boto3Storage):
    bucket_name = settings.MEDIA_FILES_BUCKET
    location = settings.MEDIA_FILES_LOCATION
    file_overwrite = False

Requirements.txt includes:

Django==3.0.3
gunicorn==20.0.4

django-model-utils==4.0.0
django-storages==1.9.1
boto3==1.13.15
htmlmin==0.1.12

sentry-sdk==0.14.3
slackclient==2.5.0
structlog==18.2.0
ddtrace==0.31.0
colorama==0.3.9
json-logging-py==0.2

When I removed the ManifestFilesMixin from the ManifestStaticFilesStorage class, collectstatic worked, but obviously not with hashed files or the staticfiles.json that I need.

I've spent multiple days trying to figure this out and I'm stumped as to what could be the cause. Any information would be appreciated.

Answer 1

Found an answer on another site from someone that was having a similar issue. It seems S3Boto3Storage is throwing IOError but ManifestFilesMixin is expecting FileNotFound. This solved my issue and I'm now seeing the hashed files in my S3 bucket along with a staticfiles.json.

class ManifestStaticFilesStorage(ManifestFilesMixin, S3Boto3Storage):
    bucket_name = settings.STATIC_FILES_BUCKET
    location = settings.STATIC_FILES_LOCATION

    def read_manifest(self):
        try:
            return super(ManifestStaticFilesStorage,self).read_manifest()
        except IOError:
            return None