Trying to write files into dags folder

Question

I'm trying to create folders and write files into the dags folder which is located in a Google Cloud Storage bucket. This using Airflow using the following python code:

Path(f'/home/airflow/gcs/dags/API/config').mkdir(parents=True, exist_ok=True) 
with open(file='/home/airflow/gcs/dags/API/config/config.json', mode = 'w') as out_file:
    out_file.write(json_string)

No error is raised and yet no folder nor file is created, anywhere. I tried the same method with the data directory which actually works

Answer 1

You are writing the files locally, the folder and the files are created only on the worker machine.

By design, you provide a DAG to Airflow to execute things. In Composer, the the DAG directory is only read from GCS and duplicated into Composer VMs. If you write on it, you won't write back to GCS. It's not the same thing for the data.

If you want to write into the DAG directory, write in GCS and not in the local directory.