Search code examples
apache-nifi

QueryRecord unable to write attribute


Original FlowFile content (stored as JSON):

[
  {
    "report_date": "20210620",
    "metric_name": "mcf:assistedConversions",
    "metric_value": "0"
  },
  {
    "report_date": "20210620",
    "metric_name": "mcf:lastInteractionConversions",
    "metric_value": "9"
  }
]

I want to convert it to AVRO and add additional fields from attributes.

  1. account.view.id = 187712
  2. utm.marks = utm_campaign=Beeline_2021&utm_content=banner_320х480&utm_medium=Media&utm_source=ad&utm_term=msc

Using QueryRecord:

SELECT '${account.view.id}' AS account_view_id,
       '${utm.marks}' AS utm_marks,
        *
  FROM FLOWFILE

It throws error:

 org.apache.nifi.processor.exception.ProcessException: java.sql.SQLException: 
Error while preparing statement [SELECT 
'187712' AS account_view_id,
'utm_campaign=Beeline_2021&utm_content=banner_320х480&utm_medium=Media&utm_source=ad&utm_term=msc' AS utm_marks,
*
FROM FLOWFILE]

If I'll remove '${utm.marks}' AS utm_marks from SQL script, it will work. But why QueryRecord can't parse value from utm.marks attribute? Unexpected char or what? JoltTransform working fine as well.

How to fix it?

UPDATE

QueryRecord config queryrecord

AvroRecordSetWritter Avro Writer JsonTreeReader jsonreader


Solution

  • The error happens because you select for two fields and then you select for everything (*). Instead, you can push your attributes account.view.id and utm.marks into flow file using UpdateRecord and then you can just query for everything using *. Attaching possible configuration for UpdateRecord processor. enter image description here

    Or more dynamic approach: enter image description here