Search code examples
amazon-web-servicespysparkaws-gluedistributed-systemaws-glue-spark

How to pass input parameter to AWS Glue Map.apply function

I am working on an AWS Glue job where I have a function "some_function" that I want to apply on DynamicFrame dy_f, but I also want to pass an input param to some_function.

Map.apply(frame=products_combination, f=search)

where some_function's definition is:

some_function(record, k)

What I've tried so far: Works:

Map.apply(frame=products_combination, f=search) ##provided i'm not taking k as input in some_function def as well

What is giving error:

Map.apply(frame=products_combination, f=search(k=10))

This returns "TypeError: search() missing 1 required positional argument: 'record'"

How can I pass a parameter to Map.apply function? I have gone through the documentation [here] but couldn't find my solution there.1

Solution

  • I eventually used Spark udf instead of DynamicFrame.Map.apply. Worked for me.