The bulk download from AWS s3 is failing

Question

This code was working fine until a few days ago. However, we noticed that it is failing exactly after downloading 40 files. I don't remember changing any configuration parameters in the recent past.

Tried isolating by validating these scenarios:

1. Different images
2. Different folders
3. Different users
4. Different S3 buckets
5. Different servers - staging and production - Issue is occuring in both
6. No memory (storage and RAM), file permission issues as presigned urls are accessible and sufficient storage and memory is available on the servers

What's puzzling is - why the download is failing exactly after the 40th file? The client (browser) is getting the error "Unable to download the zip file. Please retry on a high speed internet connection or contact support, if issue persists!". This is the response returned to the client from catch block.

Of course, this error can be fixed. But, it doesn't help the user to download beyond 40 files even if there are more.

• Is it possible that the process moved to populating $ImagesZip array before the files are download?
• If so, how come it is failing exactly after 40 files every time? There can be different sizes of files and some may take longer/ shorter as per their size.

I'm out of my wits here. Any help is greatly appreciated.

Here below the sanitized code:

public function downloadZip(Request $request) { 

    $reqdata = $request->all();
    $ordref = @$reqdata['ordref'];
    $usrnumber = @$reqdata['usrnumber'];

    if (!$ordref || !$usrnumber) {
    //check if parameters passed valid
        echo json_encode($data);
        exit();
    } 

    $this->db = DB::table("orders AS o");
    $details = $this->db->select("o.photos", "o.ref", "o.id", "o.folder_id", "o.price","o.is_download")
            ->where(....)
            ->first();

    if (!$details) {
        return redirect("/");
    }

    //check if json is valid
    $images = $this->processphotos($details->photos);

    // find photo ids from DB        
    ....
    ....
    $dbimages = $this->getimagesbyids($imageids);

    $imageNames = [];
    foreach ($dbimages as $item) {
        $thumbName = $item->imgurl;
        array_push($imageNames, array('image' => $thumbName, 'category' => $item->category, 'id' => $item->id));
    }


    if (!$imageNames) {
    //return No images found error.

    }

    $zipimages = [];
    $public_dir = public_path() . DIRECTORY_SEPARATOR . 'downloads' . DIRECTORY_SEPARATOR;
    $zipimagesarray = array();
    $imagepatharray = explode("/", $imageNames[0]['image']);
    $imagename = end($imagepatharray);

    $ctr = 100;
    foreach ($imageNames as $row) {
        $imgtype = 'jpg';
        $prefix = $usrnumber.'_';
        $imagename = $prefix . $ctr . $imgtype;
        $signedurl = $this->getpresignedurl($row['image']);
        @ File::copy($signedurl, $public_dir . $imagename);

        array_push($zipimagesarray, array('path' => $public_dir, "image" => $imagename));
        $ctr++;
    }

    $imageZip = [];
    $certZip = [];
    foreach ($zipimagesarray as $imagePublic) {
        $path = explode(DIRECTORY_SEPARATOR,$imagePublic['path']);

        $zipImage = $imagePublic['path'] . $imagePublic['image'];
        array_push($imageZip, $zipImage);
    }

    try {
        $zipName = $usrnumber . date('G:i:s') . ".zip";

        Zipper::make(public_path('downloadOrder/' . $zipName))->add($imageZip)->close();

        File::delete($imageZip);
        $file = "/download/" . $zipName;

        $data['result'] = array();
        $data['result'] = url('/') . $file ;
        $data['statusCode'] = 200;
        $data['hasError'] = false;
        $data['message'] = "Images retrieval is successful.";
        return response($data)->header('Content-Type', 'application/json');
    } catch (\Exception $ex) {
        $data['result'] = array();
        $data['statusCode'] = 400;
        $data['hasError'] = true;
        $data['message'] = "Unable to download the zip file. Please retry on a high speed internet connection or contact support, if issue persists!";
        Log::error("Api::downloadZip - orderId: " . $data['Message'] . " " . $ex->getMessage());
        return response($data)->header('Content-Type', 'application/json');
    }
}

and

function getpresignedurl($key, $s3Client = null) {

$s3_env = [];
$Bucket = "xxxxx";

if (is_null($s3Client)) { 
   $s3_env = array('region' => config('aws.AWS_REGION'),
       'credentials' => ['key' => config('aws.AWS_ACCESS_KEY_ID'), 'secret' => config('aws.AWS_SECRET_ACCESS_KEY')]);
   $s3Client = \App::make('aws')->createClient('s3', $s3_env);
}

//Creating a presigned URL
$cmd = $s3Client->getCommand('GetObject', [
    'Bucket' => $Bucket,
    'Key' => $key
]);

$request = $s3Client->createPresignedRequest($cmd, '+30 minutes');
$signedurl = (string) $request->getUri();
return $signedurl;
}

Answer 1

Well, This is not the exact solution for the issue reported.

I have addressed it by implementing entirely a new approach. Introduced a new method downloadFile and used AWS SDK GetObject command to download the files. Even saveAs parameter in GetObject did not come to the rescue. This solution getObject with SaveAs not working in sdk v3 helped.

function downloadFile($key, $destination, $s3Client = null) {
     $s3_env = [];
     $Bucket = "xxxxx";

     try {
        if (is_null($s3Client)) { 
            $s3_env = array('region' => config('aws.AWS_REGION'),
                'credentials' => ['key' => config('aws.AWS_ACCESS_KEY_ID'), 'secret' => config('aws.AWS_SECRET_ACCESS_KEY')]);
            $s3Client = \App::make('aws')->createClient('s3', $s3_env);
        }

        //Creating a presigned URL
        $cmd = $s3Client->getCommand('GetObject', [
           'Bucket' => $Bucket,
           'Key' => $key,
           '@http' => ['sink' => $destination]
        ]);

        $result = $s3Client->execute($cmd);

        if ($result['@metadata']['statusCode'] === 200) {
            return true;
        } else {
            Log::error("Api::downloadFile - Failed to download to $destination " . $result['@metadata']['statusCode']);
            return false;
        }
    } catch (\Exception $ex) {
        Log::error("Api::downloadFile - exception: $ex->getMessage()");
        return false;
    }
}

And in the main method downloadZip, replaced these lines

    $signedurl = $this->getpresignedurl($row['image']);
    @ File::copy($signedurl, $public_dir . $imagename);

with

    $destination = $public_dir . $imagename;
    $result = $this->downloadFile($row['image'], $destination);

Also, to handle download failures, added an if condition

if (file_exists($public_dir . $imagename) {        
    array_push($zipimagesarray, array('path' => $public_dir, "image" => $imagename));
}

I'll wait for a few more days if someone can offer a solution on why automatic html encoding is happening after 40th file and then accept this as a solution.