RecrawlPolicy issue while using AWS Glue Crawler

Question

I'm creating a AWS glue crawler using API, I have created a crawler already and it is working fine. Now i want to implemt IncrementalCrawler functionality to it. When I try to give Recrawler parameter for update_crawler() it is throwing me with the error. The same thing is happening is happening with create_crawler() also.

mycode

`crawler_args = {"Name": args['CRAWLER_NAME'], "Role": args['IAM_ROLE'], "DatabaseName": args['DB_NAME'],"RecrawlPolicy": {"RecrawlBehavior": "CRAWL_NEW_FOLDERS_ONLY"}
                "Targets":
                    {"S3Targets": [{"Path": f"s3:/../{src_obj}/"}]},
                "TablePrefix": ""}
    glue.create_crawler(**crawler_args)`

"Parameter validation failed: Unknown parameter in input: "RecrawlPolicy", must be one of: Name, Role, DatabaseName, Description, Targets, Schedule, Classifiers, TablePrefix, SchemaChangePolicy, Configuration, CrawlerSecurityConfiguration"

I believe it is because of the boto3 version im using, if that is the case how to update boto3 to latest version.

Answer 1

It looks like you are missing a comma, can you try:

crawler_args = {"Name": args['CRAWLER_NAME'], "Role": args['IAM_ROLE'], "DatabaseName": args['DB_NAME'],"RecrawlPolicy": {"RecrawlBehavior": "CRAWL_NEW_FOLDERS_ONLY"},
                "Targets":
                    {"S3Targets": [{"Path": f"s3:/../{src_obj}/"}]},
                "TablePrefix": ""}