Different S3 object listing when uploading via Java SDK and AWS Console

Question

I have a bucket in S3 with the following structure and contents:

javaFolderA/
└── javaFolderB/
    └── javaFile.tmp
consoleFolderA/
└── consoleFolderB/
    └── consoleFile.tmp

The java* folders and file were uploaded via the Java SDK:

final File file = new File("C:\\javaFolderA\\javaFolderB\\javaFile.tmp");
client.putObject("testbucket", "javaFolderA/javaFolderB/javaFile.tmp", file);

The console* folders and file were created/uploaded from the web console (Clicking the "+ Create folder" button for each folder, then uploading the file with public read permissions).

In the web console, after clicking to create a new bucket, the following message is shown:

When you create a folder, S3 console creates an object with the above name appended by suffix "/" and that object is displayed as a folder in the S3 console.

So, as expected, with the folders and files above, we get 3 objects created in the bucket with the following keys:

• consoleFolderA/
• consoleFolderA/consoleFolderB/
• consoleFolderA/consoleFolderB/consoleFile.tmp

Tthe result of the SDK upload is a single object with the key: javaFolderA/javaFolderB/javaFile.tmp. This makes sense, as we are only putting a single object, not three. However, this results in inconsistencies when listing the contents of a bucket. Even though there is only one actual file in each directory, listing the contents returns multiple for the console scenario.

My question is why is this the case, and how can I achieve consistent behavior? There doesn't seem to be a way to "upload a directory" via the SDK (In quotes because I know there aren't actually folders/directories).

From the CLI I can verify the number of objects and their keys:

C:\Users\avojak>aws s3api list-objects --bucket testbucket
{
    "Contents": [
        {
            "LastModified": "2018-01-02T22:43:55.000Z",
            "ETag": "\"d41d8cd98f00b204e9800998ecf8427e\"",
            "StorageClass": "STANDARD",
            "Key": "consoleFolderA/",
            "Owner": {
                "DisplayName": "foo.bar",
                "ID": "2c401638471162eda7a3b48e41dfb9261d9022b56ce6b00c0ecf544b3e99ca93"
            },
            "Size": 0
        },
        {
            "LastModified": "2018-01-02T22:44:02.000Z",
            "ETag": "\"d41d8cd98f00b204e9800998ecf8427e\"",
            "StorageClass": "STANDARD",
            "Key": "consoleFolderA/consoleFolderB/",
            "Owner": {
                "DisplayName": "foo.bar",
                "ID": "2c401638471162eda7a3b48e41dfb9261d9022b56ce6b00c0ecf544b3e99ca93"
            },
            "Size": 0
        },
        {
            "LastModified": "2018-01-02T22:44:16.000Z",
            "ETag": "\"968fe74fc49094990b0b5c42fc94de19\"",
            "StorageClass": "STANDARD",
            "Key": "consoleFolderA/consoleFolderB/consoleFile.tmp",
            "Owner": {
                "DisplayName": "foo.bar",
                "ID": "2c401638471162eda7a3b48e41dfb9261d9022b56ce6b00c0ecf544b3e99ca93"
            },
            "Size": 69014
        },
        {
            "LastModified": "2018-01-02T22:53:13.000Z",
            "ETag": "\"d41d8cd98f00b204e9800998ecf8427e\"",
            "StorageClass": "STANDARD",
            "Key": "javaFolderA/javaFolderB/javaFile.tmp",
            "Owner": {
                "DisplayName": "foo.bar",
                "ID": "2c401638471162eda7a3b48e41dfb9261d9022b56ce6b00c0ecf544b3e99ca93"
            },
            "Size": 0
        }
    ]
}

Answer 1

If you prefer the console implementation then you need to emulate it. That means that your SDK client needs to create the intermediate 'folders', when necessary. You can do this by creating zero-sized objects whose key ends in forward-slash (if that's your 'folder' separator).

The AWS console behaves this way, allowing you to create 'folders', because many AWS console users are more comfortable with the notion of folders and files than they are with objects (and keys).

It's rare, in my opinion, to need to do this, however. Your SDK clients should be implemented to handle both the presence and absence of these 'folders'. More info here.