PHP - Google App Engine Uploading large files to Google Drive as chunks (Resumable Upload) - 500 Error

Question

I'm uploading a file from the Google Cloud storage bucket to Google Drive folder using a php code that runs in the background. My problem is when the size of the file increase, the Google Drive(or GAE) gives a 500 error.

Error: Server Error
The server encountered an error and could not complete your request.
Please try again in 30 seconds.

I'm using the following dependency.

"nao-pon/flysystem-google-drive": "^1.1",

Following is my code part that gives this error. This works perfectly for small files:

$fileN = $this->clean($att->ATT_TITLE);

$fileData = file_get_contents("https://storage.googleapis.com/xxxxxx/".$att->ATT_FILE);

$dir = sys_get_temp_dir();
$tmp = tempnam($dir, $fileN);
file_put_contents($tmp, $fileData);
Storage::cloud()->putFileAs( $folderPath, new File($tmp),$fileN);

I have set the configurations in .env file

FILESYSTEM_CLOUD=google
GOOGLE_DRIVE_CLIENT_ID=xxxxapps.googleusercontent.com
GOOGLE_DRIVE_CLIENT_SECRET=xxxxx
GOOGLE_DRIVE_REFRESH_TOKEN=xxxxxx
GOOGLE_DRIVE_FOLDER_ID=xxxx
#GOOGLE_DRIVE_TEAM_DRIVE_ID=xxx

I have setup the disk as below in the filesystem.php

'google' => [
        'driver' => 'google',
        'clientId' => env('GOOGLE_DRIVE_CLIENT_ID'),
        'clientSecret' => env('GOOGLE_DRIVE_CLIENT_SECRET'),
        'refreshToken' => env('GOOGLE_DRIVE_REFRESH_TOKEN'),
        'folderId' => env('GOOGLE_DRIVE_FOLDER_ID'),
    // 'teamDriveId' => env('GOOGLE_DRIVE_TEAM_DRIVE_ID'),
    ],

I have set the php.ini file as below

upload_max_filesize = 300M
post_max_size = 300M
memory_limit = 3000M

App.yaml

runtime: php72
handlers: 
- url: /assets
  static_dir: public/assets
env_variables:
   APP_KEY: xxxxx
   APP_STORAGE: /tmp
   CACHE_DRIVER: file
   VIEW_COMPILED_PATH: /tmp
   SESSION_DRIVER: database
   DB_DATABASE: xxx
   DB_USERNAME: xx
   DB_PASSWORD: xxxx
   DB_SOCKET: "/cloudsql/xxxxx"
runtime_config:
    document_root: public

Can anyone provide me a hint or give me another alternative way to do this? My requirement is basically to upload a large file (like 250MB) to Google Drive by a background job in Google App Engine.

Answer 1

For this situation, Uploading in chunks solve the issue.

Refer to the following example from Google API PHP Client https://github.com/googleapis/google-api-php-client/blob/master/examples/large-file-upload.php

Sample Working Code:

$fileData = file_get_contents("https://storage.googleapis.com/xxxx/".$att->ATT_FILE);

$dir = sys_get_temp_dir();
$tmp = tempnam($dir, $fileN);
file_put_contents($tmp, $fileData);       
     
$fileToUpload = new File($tmp);
$filename = $fileN;
$mimeType = mime_content_type($tmp);
                   
$driveService = Storage::cloud();
$client = new \Google_Client();
$client->setClientId('xxxx');
$client->setClientSecret('xxx');
$client->refreshToken('xxx');
$client->setApplicationName('xxx');

$service = new \Google_Service_Drive($client);


// Test 1
$file = new \Google_Service_Drive_DriveFile();
$file->title = $fileN;
$file->name = $fileN;
$file->parents = array($basePath);
$chunkSizeBytes = 1 * 1024 * 1024;

// Call the API with the media upload, defer so it doesn't immediately return.
$client->setDefer(true);
$request = $service->files->create($file);

// Create a media file upload to represent our upload process.
$media = new \Google_Http_MediaFileUpload(
  $client,
  $request,
  $mimeType,
  null,
  true,
  $chunkSizeBytes
);
$media->setFileSize(filesize($tmp));

// Upload the various chunks. $status will be false until the process is
// complete.
$status = false;
$handle = fopen($tmp, "rb");
while (!$status && !feof($handle)) {
  $chunk = fread($handle, $chunkSizeBytes);
  $status = $media->nextChunk($chunk);
 }

// The final value of $status will be the data from the API for the object
// that has been uploaded.
$result = false;
if($status != false) {
  $result = $status;
}

fclose($handle);