Search code examples
apache-spark-sqlyaml

Illegal pattern character: n while reading yaml

I have a yaml with the following definition, but on reading the file i am getting an error # Error: Failed while running read report config. Illegal pattern character: n

  - source: internal
    instance: internal
    query: "select distinct b.HSETTLEDATE,a.ACCOUNT_NO,a.FREEZE_STATUS,a.DESCRP,a.FREEZE_MARK_UNMARK_DT,a.NARRATION,a.EXTENDED_NARR from Demo2 a left join Demo1 b on a.ACCOUNT_NO=INT(b.ACC_NO)"
    fetch_size: 10000
    partition: 1
    df_name: 'monthly_report_for_dbt_freeze_account_error'
    output_file_type: excel
    delimiter: na
    output_flag: T
    output_location: ""
    output_filename: "monthly_report_for_dbt_freeze_account_error"

this query is syntactically correct - not sure what the error is here...

Solution

  • Update - while the error reads that the issue is in the read config , it is on execution of the query - the error was caused by previous blocks of the yaml

    the error arises when the date format is nor parseable by spark

    please check the to_date statements in your code

    steps to follow

    1. run each query (print the dataframe created)
    2. check the date formats for each query