upload_many_blobs_with_transfer_manager() returned TimeoutError('The write operation timed out'))

Question

The below code was successful for uploading CSVs and an XLSX file to Google Cloud Storage - as expected since it is from Upload objects from a file system

However a CSV around 2,000KB returned the error message:

('Connection aborted.', TimeoutError('The write operation timed out'))

There are other functions that have a timeout option but it is not the case here, I've considered amending the bucket method based on reading list below.

from google.cloud.storage import Client, transfer_manager

def upload_many_blobs_with_transfer_manager(input_logger,
    bucket_name, filenames, source_directory="", workers=8):
    """Upload every file in a list to a bucket, concurrently in a process pool.

    Each blob name is derived from the filename, not including the
    `source_directory` parameter. For complete control of the blob name for each
    file (and other aspects of individual blob metadata), use
    transfer_manager.upload_many() instead.
    """

    storage_client = Client()
    bucket = storage_client.bucket(bucket_name)

    results = transfer_manager.upload_many_from_filenames(
        bucket, filenames, source_directory=source_directory, max_workers=workers
    )

    for name, result in zip(filenames, results):
        # The results list is either `None` or an exception for each filename in
        # the input list, in order.

        if isinstance(result, Exception):
            input_logger.info("Failed to upload {} due to exception: {}".format(name, result))
        else:
            input_logger.info("Uploaded {} to {}.".format(name, bucket_name))

All help appreciated.

NateBI

I have read:

• Why does upload_from_file Google Cloud Storage Function Throws timeout error?

• https://googleapis.dev/python/storage/latest/retry_timeout.html

• Why does upload_from_file Google Cloud Storage Function Throws timeout error? and I'm considering changing the bucket = storage_client.bucket(bucket_name) to bucket = client.get_bucket(BUCKET_NAME, timeout=300.0) # five minutes based on this but asking here first.

Answer 1

As John Hanley stated it was a networking issue. Solved!