Search code examples
snowflake-cloud-data-platformsnowflake-schemasnowpipe

How to get Value column in AVRO file format


I’m trying to verify whether the column data has been mapped correctly. Typically, when using an external table, there’s a column named “value” that contains the row data in JSON format. Is it possible to have that same “value” column when using Snowpipe?

CREATE OR REPLACE PIPE fuel_analytics_pipe
  AUTO_INGEST = TRUE
AS 
  COPY INTO DATNS.PRING.FUNNEL_PIPE 
  FROM @finance_analytics_stage
  MATCH_BY_COLUMN_NAME=CASE_INSENSITIVE
   INCLUDE_METADATA = (  FILENAME=METADATA$FILENAME, 
  FILE_LAST_MODIFIED=METADATA$FILE_LAST_MODIFIED )
   FILE_FORMAT = (TYPE = 'AVRO')
  pattern ='ETL/2025/02/04/futrics_.*\\.avro'
; 

Solution

  • There is a mention of selecting columns in documentation to support transformation, unsure if there is an option of value, but I guess with this option you can still select column using $

    COPY INTO mytable(C1, C2)
      FROM (SELECT $5, $4 FROM @mystage)
    

    I guess based on above you could change your code

    COPY INTO DATNS.PRING.FUNNEL_PIPE(C1,C2) 
      FROM (select $3, $4 @finance_analytics_stage)