Search code examples
pythonamazon-web-servicesnumpyamazon-dynamodbaws-data-wrangler

AWS Wrangler S3 reading parquet, writing to DynamoDB - Unsupported type numpy.ndarray


I am trying to read parquet into dataframe with AWS wrangler, while writing this data to DynamoDB its erroring out with unsupported type error - Unsupported type numpy.ndarray for value.......

wr.s3.read_parquet(path=s3_path, dataset=dataset, chunked=True)

and writing like 

wr.dynamodb.put_df(df=df, table_name=table_name)

Is there a way, I can convert the ndarray type to dynamoDB list ? I dont want to lose the readability of the array if I use np.tobytes() to write and np.frombuffer() to read again. Also, users will not have access Numpy to read the data from DynamoDB


Solution

  • Finally able to get the answer, so it worked like this for me using tolist()-

    df["key"] = df["key"].apply(lambda x: x.tolist())
    
    wr.dynamodb.put_df(df, "test_table")