Search code examples
pysparkapache-spark-sqlaws-glue

Problem when converting DataFrame to DynamicFrame


I have a code that converts DataFrame to DynamicFrame and I get this weird error when trying to execute return statement, any clues what's going on?

Error: {AttributeError}'str' object has no attribute '_jvm'

# record is DynamicFrame
def extractCustomFields(record, ctx):
    rec = record.toDF()
    rec = rec.withColumn("lastname", rec["customfields"][0].value)
    rec.show()
return DynamicFrame.fromDF(rec, ctx, "recordTransform")

Solution

  • fromDF() expects the GlueContext as second argument. You need to pass that:

    return DynamicFrame.fromDF(rec, ctx, "recordTransform")