I currently have a notebook on Colab Enterprise that accesses environment variables from a .env file to get access into a database to do ETL. When I run the notebook manually using a runtime instance, with the .env file uploaded into the working directory, it works fine; however, when I schedule this notebook to execute, using Colab Enterprise's schedule feature, the notebook is unable to access the environment variables (variables = None) and I am not sure where or how I should be passing these variables into notebook on a scheduled execution (the outputs currently go into a GCP bucket). I've read only that I may need to set up the environment using a python script before the execution, but I am not sure how to go about this.
Below is a code snippet I use to access the environment variables from .env:
# Now you can access your environment variables using os.getenv
load_dotenv() #function loads environment variables from .env
folder
#BE DB
BE_DEV_DBNAME = os.getenv('BE_DEV_DBNAME')
BE_DEV_USER = os.getenv('BE_DEV_USER')
BE_DEV_PASSWORD = os.getenv('BE_DEV_PASSWORD')
BE_DEV_HOST = os.getenv('BE_DEV_HOST')
BE_DEV_PORT = os.getenv('BE_DEV_PORT')
BE_DEV_ENGINE= create_engine(f'postgresql+psycopg2://{BE_DEV_USER}:{BE_DEV_PASSWORD}@{BE_DEV_HOST}:{BE_DEV_PORT}/{BE_DEV_DBNAME}')
When printing out each of the above variables, I get the result "None" and the error occurs when using the SQLAlchemy "create_engine()" function with the following error:
invalid literal for int() with base 10: 'None'
I have tried uploading the .env file in the GCP bucket, hoping the scheduled execution's directory would be inside the bucket to access the file, but to no avail. The current working directory output I get from the scheduled notebook execution is "/mnt/executor/scratch", however, I am not sure how to load the .env file in this location, and whether this is the right way to go about it.
SOLUTION: I ended up using the Google Cloud Secret Manager feature to access secret variables following this documentation: https://codelabs.developers.google.com/codelabs/secret-manager-python#6