So I am trying to start a Glue crawler from my DAG and waiting for it to complete before I proceed with my workflow. Cant figure out what Airflow expects in the config argument.
Here is my code:
run_crawler = AwsGlueCrawlerOperator(task_id="run_crawler",
aws_conn_id=aws_secret, config={???},
crawler_name ="name_of_crawler",
poll_interval=60, priority_weight=3)
Initially, I entered {"crawler_name":"name_of_crawler", "iam_role":"iam_role_name} with other settings constant. However it complained about me referencing crawler name in the dictionary. Took that out and still no dice.
Does anyone know what goes in the config dictionary for this operator or alternatively initiate a glue crawler from local docker instance on aws account? I have the credentials in a Connection that I am referencing in the variable aws_secret.
Thx in advance
From the code it looks like the name in config should be under Name key not crawler_name. Also I think this operator does not accept crawler_name argument which you pass in your example.