How to download a file from an endpoint and upload that file to S3 ?(GITLAB)

Question

I'm actually trying to download a zip file from a Gitlab REST endpoint that is supposed to return the the repository for a given projectID.

I used axios to call the endpoint and tried to directly upload the response data to S3, but it seems to be returning a corrupt file as the zip that it returns says it cannot be opened.

I am doing the downloading of this file in a serverless function and attempting to return an S3 URL to the client.

Headers for the response

  res.headers {
  date: 'Wed, 19 Jan 2022 13:44:42 GMT',
  'content-type': 'application/zip',
  'transfer-encoding': 'chunked',
  connection: 'close',
  'cache-control': 'max-age=0, private, must-revalidate',
  'content-disposition': 'attachment; filename="third-project-eac3ea41c782df4bee4fe07ecc3bf356f7f74f47-eac3ea41c782df4bee4fe07ecc3bf356f7f74f47.zip"',
  'content-transfer-encoding': 'binary',
  etag: 'W/"12ae32cb1ec02d01eda3581b127c1fee"',
  vary: 'Origin',
  'x-content-type-options': 'nosniff',
  'x-frame-options': 'SAMEORIGIN',
  'x-request-id': '01FSS9A6W1RM77TYMFM6G7HKZ8',
  'x-runtime': '0.063985',
  'strict-transport-security': 'max-age=31536000',
  'referrer-policy': 'strict-origin-when-cross-origin',
  'ratelimit-observed': '2',
  'ratelimit-remaining': '1998',
  'ratelimit-reset': '1642599941',
  'ratelimit-resettime': 'Wed, 19 Jan 2022 13:45:41 GMT',
  'ratelimit-limit': '2000',
  'gitlab-lb': 'fe-17-lb-gprd',
  'gitlab-sv': 'localhost',
  'cf-cache-status': 'MISS',
  'expect-ct': 'max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"',
  'report-to': '{"endpoints":[{"url":"https:\\/\\/a.nel.cloudflare.com\\/report\\/v3?s=r6XcPGaiU7JhlicrSC9iBZgXXCOMoBMXjU8kvxjZGb5UkUQBIjemmAOOX39m1ijVCnQROVhNNxc6B%2B4x%2FNf5ZG9cc8GLY%2BfMYUE29gJkHN624QKJRSX8HBrMqEQ%3D"}],"group":"cf-nel","max_age":604800}',
  nel: '{"success_fraction":0.01,"report_to":"cf-nel","max_age":604800}',
  server: 'cloudflare',
  'cf-ray': '6d007fcb7a8a8e63-PDX'
}

This it the code I'm using

const requestURL = `https://gitlab.com/api/v4/projects/${repoID}/repository/archive.zip?sha=${commitSHA}`;
 let url = await Axios({
    method: "GET",
    url: requestURL,
    headers: {
      Accept: "*/*",
      Authorization: "Bearer " + authToken,
    },
  })
    .then(async (res) => {
      
      const Key = "path/to/file";
      const params = {
        Bucket: BUCKET_URL,
        Key,
        Body: res.data,
        ContentType: "application/zip",
      };
      await s3.upload(params).promise();

      const URL = await s3.getSignedUrl("getObject", {
        Bucket: BUCKET_URL,
        Key,
        ResponseContentDisposition:'attachment; filename = test.zip"',
      });
      return URL ;
    })
    .catch((error) => {
      console.log(error);
    });
   return url;

But when I try and access this URL on my browser from the frontend, It returns a file test.zip which is larger and corrupt.

This is a screenshot of the data that I received on postman, if I click save response and name the file filename.zip it shows the actual contents of the file.

Any help would be appreciated!

Answer 1

This worked!

const requestURL = `https://gitlab.com/api/v4/projects/${repoID}/repository/archive.zip?sha=${commitSHA}`;
 let url = await Axios.get(requestURL,{
    headers: {
      Accept: "*/*",
      Authorization: "Bearer " + authToken,
    },
    responseType:'arraybuffer'
  })
    .then(async (res) => {
      
      const bufferData = Buffer.from(res.data)
      const Key = "path/to/file";
      const params = {
        Bucket: BUCKET_URL,
        Key,
        Body: bufferData,
        ContentType: "buffer",
      };
      await s3.upload(params).promise();

      const URL = await s3.getSignedUrl("getObject", {
        Bucket: BUCKET_URL,
        Key,
        ResponseContentDisposition:'attachment; filename = test.zip"',
      });
      return URL ;
    })
    .catch((error) => {
      console.log(error);
    });
   return url;