how to use Amazon S3 connector in wso2 ei 6.6 to get all the objects in particular subfolder?

Question

I have multiple csv files in object 'bbb' which is in bucket 'aaa'. I am able to download single file as shown below using the Amazon S3 connector with 'getObject' functionality local entry looks like this:

<?xml version="1.0" encoding="UTF-8"?>
<localEntry key="AMAZON_S3_CONNECTION_1" xmlns="http://ws.apache.org/ns/synapse">
    <amazons3.init>
        <awsAccessKeyId>**</awsAccessKeyId>
        <name>AMAZON_S3_CONNECTION_1</name>
        <region>**</region>
        <connectionType>amazons3</connectionType>
        <awsSecretAccessKey>**</awsSecretAccessKey>
    </amazons3.init>
</localEntry>

<?xml version="1.0" encoding="UTF-8"?>
<proxy name="AmazonConnectivity" startOnLoad="true" transports="http https vfs" xmlns="http://ws.apache.org/ns/synapse">
    <target>
        <inSequence>
            <log level="full">
                <property name="log" value="====S3 to Local===="/>
            </log>
            <amazons3.getObject configKey="AMAZON_S3_CONNECTION_1">
                <bucketName>aaa</bucketName>
                <objectKey>bbb/xyz.csv</objectKey>
            </amazons3.getObject>
            <log level="full">
                <property name="log" value="====After Amazon S3 Init===="/>
            </log>
        </inSequence>
        <outSequence/>
        <faultSequence/>
    </target>
    <parameter name="transport.PollInterval">5</parameter>
    <parameter name="transport.vfs.FileURI">vfs:file:///C:/amazon-s3</parameter>
    <parameter name="transport.vfs.ContentType">text/plain</parameter>
    <parameter name="transport.vfs.ActionAfterProcess">MOVE</parameter>
    <parameter name="transport.vfs.MoveAfterFailure">vfs:file:///C:/failure</parameter>
    <parameter name="transport.vfs.ActionAfterFailure">MOVE</parameter>
    <parameter name="transport.vfs.FileNamePattern">.*.rdy</parameter>
    <parameter name="transport.vfs.MoveAfterProcess">vfs:file:///C:/out</parameter>
</proxy>

but now I want to bulk download all the files in object 'bbb'. Can someone help me with it. I am using Amazon S3 connector -Version 2.0.2.

Answer 1

I would suggest you to refer to WSO2 S3 Connector Doc here for reference.

However, an example use case of the getObjectsInBucket that can be used for this scenario is below.

You can set the value in <prefix></prefix> for the targeted sub-folder you want to list.

Sample Request to List:

<getObjectsInBucket>
    <accessKeyId>ACCESS_KEY</accessKeyId>
    <secretAccessKey>ACCESS_SECRET</secretAccessKey>
    <region>REGION</region>
    <delimiter>DELIMITER</delimiter>
    <encodingType>ENCODING_TYPE</encodingType>
    <methodType>GET</methodType>
    <bucketName>aaa</bucketName>
    <bucketUrl>BUCKET_URL</bucketUrl>
    <host>HOST_URL</host>
    <prefix>bbb</prefix>
</getObjectsInBucket>

Edit:

Sample Request to Download:

<getObject>
    <accessKeyId>AKIXXXXXHXQXXG5XX</accessKeyId>
    <secretAccessKey>qHXXBXXXXASYQc4oMCEOj+343HD82s</secretAccessKey>
    <methodType>GET</methodType>
    <contentType>application/xml</contentType>
    <region>us-east-2</region>
    <host>s3.us-east-2.amazonaws.com</host>
    <bucketUrl>http://s3.us-east-2.amazonaws.com/signv4test</bucketUrl>
    <bucketName>signv4test</bucketName>
    <isXAmzDate>true</isXAmzDate>
    <xAmzSecurityToken/>
    <contentMD5/>
    <objectName>Tree2.png</objectName>
    <rangeBytes/>
    <responseContentType/>
    <responseContentLanguage/>
    <responseExpires/>
    <responseCacheControl/>
    <responseContentDisposition/>
    <range/>
    <ifModifiedSince/>
    <ifUnmodifiedSince/>
    <ifMatch/>
    <ifNoneMatch/>
</getObject>

However, downloading multiple object is not supported as for the AWS S3 Docs. There are suggested workarounds as similar to here