Search code examples
google-cloud-platformgoogle-bigquerydata-warehouseamazon-ses

Uploading data received via email to data warehouse


I am going to receive csv files of data from a vendor weekly via email, we use Outlook. I want to upload the attachments into our data warehouse, which is Google BigQuery.

In looking for the easiest way to do this I found Amazon's Simple Email Service which looks like it can automatically save the email and attachment to a storage bucket. This would be great... if only it was on Google Cloud Platform.

Is there a GCP equivalent to this? Or if not is there a standard approach to uploading data files received via email to a warehouse?


Solution

  • That's what you are looking for is App Engine, which gives you the possibility to listen to incoming emails via Mail API. You can read and write attachments to GCS bucket, by specifying the bucket while writing in Cloud Storage.

    from google.appengine.api import app_identity
    bucket_name = os.environ.get('BUCKET_NAME',app_identity.get_default_gcs_bucket_name())
    

    To process receiving email messages, you associate email addresses with script handlers in your app configuration, then include the handlers in your app's code.

    Please, refer to official documentation and read about quotas and limits, e.g. maximum size of incoming mail messages, including attachments equals to 31.5 MB.