Using AWS CloudFormation I set up a CloudFront distribution to serve content from a private S3 bucket. I do not have the bucket configured as an S3 website — rather, I'm using the latest-and-greatest technique: Origin Access Control (OAC). See Restricting access to an Amazon S3 origin. I'm using Route53 and Certificate Manager to serve the CloudFront distribution over TLS with a custom domain example.com.
So far the basics are working fine for URLs that reference objects that exist in the S3 bucket. I can access https://example.com/foobar.html just fine, for example. But if I request a file that does not exist, such as https://example.com/missing.html, CloudFront returns a 403 "Access Denied" instead of a 404 "Not Found".
I can make a wild guess that some communication between CloudFront and S3 makes CloudFront think its access is denied if the object doesn't exist. (Still that doesn't explain why.) Is this a bug? Is this expected behavior? How are we expected to use CloudFront+S3+OAC with this odd behavior—does AWS expect us to set up a CloudFront custom error response to convert 403 to 404? (But why would we want to assume all access denied errors in CloudFormation really indicate a missing object on S3?)
Note that I found various other CloudFront questions related to 403, but none related to an OAC configuration, and most of the other questions were regarding a CloudFront distribution that always returned 403, not just for missing files.
Unless you have the s3:ListBucket permission, S3 returns the 403 Forbidden status and the AccessDenied error for missing objects, by design. This is because without s3:ListBucket, the principal doesn't have permission to know whether the object is missing or if it exists but they aren't allowed access.
Note that unlike s3:GetObject, an object-level permission where the resource ARN is arn:aws:s3:::bucket-name/*, s3:ListBucket is a bucket-level permission, so the resource is arn:aws:s3:::bucket-name without the trailing /*.
After updating the bucket policy, you should find that the 404s work as expected, but you also need to set the Cloudfront Default Root Object for the distribution to whatever you want returned when / is requested, otherwise a bucket listing will be returned, which is probably not what you want.
Also be aware of the Error Caching Minimum TTL, which causes CloudFront to cache those 403s for 5 minutes, separate from the other TTL settings for the cache behavior.