Search code examples
azureazure-data-factoryazure-data-lake-gen2

How to achieve incremental load (SCD1) from Oracle to Parquet

I want to perform SCD 1.

My watermark column is Rma_date. I need to use COPY activity because Oracle is not supported as Source in Dataflow in order to use Alter Row.

How to achieve SCD1 from Oracle db to parquet using COpy Activity?

Solution

  • Here is an example of Oracle to Parquet with what could be incremental based on your where statement, and the sink I'm making BLOB storage but Azure Data Lake is pretty much same thing:

    SOURCE:

    Source

    SINK:

    Sink

    MAPPING: Don't put anything here, it will automatically pull all columns from table and contain in Parquet.

    Output Stats:

    Output Stats

    Parquet File sitting in Blob:

    Sitting

    I hope this helps, I'm not super familiar with SDC, but if things do change on the Oracle side, I know this will be reflected in the parquet since I don't have a custom mapping configured.