Why is the export to the Drive empty, when using Google Earth Engine in Python with authentication through service account?

Question

I'm using the Google Earth Engine in Python to get a Sentinel-2 composition and download it to my google drive. When doing a manual authentication everything is working fine:

ee.Authenticate()
ee.Initialize()

However, as I want to use my code in a workflow and don't use all the time the manual authentication, I am using a service account, like described here. It works fine and I can use GEE without doing anything manually:

# get service account
service_account = 'test@test.iam.gserviceaccount.com'

# get credentials
credentials = ee.ServiceAccountCredentials(service_account, gee_secret.json)

ee.Initialize(credentials)

In order to export my File to Google Drive I'm using following code:

# export options
export_config = {
   'scale':10,
   'region':aoi, #aoi is a polygon
   'crs': 'EPSG:3031',
}
file_name = "test"

# export to drive
task = ee.Batch.Export.iamge.toDrive(image, file_name, **export_config)
task.start()

With both authentication methods this task is successfully finished (The status of the task is 'Completed'). However, only when using the manual authentication, I can see my exported image in my Drive. When using the automatic authentication, my Drive is empty.

Someone else already asked a similar question here. A possible idea here was that the image file is exported to the Google Drive of the service account and not to my personal Google Drive. However, I am not sure how to access this other Drive.

Does anyone have an idea how to solve this (=how to access the exported file?). Or have another solution for automatic authentication in which the file will be at my personal Google Drive?

Answer 1

Many thanks to the hints of DaImTo and Yancy Godoy! With these I could find a solution. I will post it here, so that it is maybe useful for others as well.

Indeed the export to the Google Drive worked perfectly, however it was not exported to my personal Google Drive, but to the Google Drive of the service account. It was therefore important to add the access to Google Drive for my service account (see here).

In the following you can find a complete workflow. For the downloading I am using PyDrive

from pydrive.auth import GoogleAuth
from pydrive.drive import GoogleDrive
from oauth2client.service_account import ServiceAccountCredentials

# this file contains the E-mail adress of the service account
with open(key_path + '/service_worker_mail.txt', 'r') as file:
   service_account_file = file.read().replace('\n', '')

# get the credentials
credentials = ee.ServiceAccountCredentials(service_account_file, key_path + "/" + "gee_secret.json")

# authenticate and initialize Google Earth Engine
ee.Initialize(credentials)

#  The following operations of Google Earth Engine, including the Export to the Drive
# ...
# ...
# ...

# authenticate to Google Drive (of the Service account)
gauth = GoogleAuth()
scopes = ['https://www.googleapis.com/auth/drive']
gauth.credentials = ServiceAccountCredentials.from_json_keyfile_name(key_path + "/" + "gee_secret.json", scopes=scopes)

drive = GoogleDrive(gauth)

# get list of files
file_list = drive.ListFile({'q': "'root' in parents and trashed=false"}).GetList()
    for file in file_list:

        filename = file['title']

        # download file into working directory (in this case a tiff-file)
        file.GetContentFile(filename, mimetype="image/tiff")

        # delete file afterwards to keep the Drive empty
        file1.Delete()