In Azure Data Factory, is it possible to subtract the matching contents contents of 1 dataset from another to create a new dataset?
For example...
Dataset A
| Date | |
|---|---|
| [email protected] | 2023-01-05 |
| [email protected] | 2023-01-07 |
| [email protected] | 2023-01-09 |
Dataset B
| Date | |
|---|---|
| [email protected] | 2023-01-05 |
| [email protected] | 2023-01-07 |
| [email protected] | 2023-01-09 |
| [email protected] | 2023-01-11 |
| [email protected] | 2023-01-15 |
| [email protected] | 2023-01-17 |
I want dataset C to be all the records in dataset B, minus the matching records in dataset A
The result would look like the following....
| Date | |
|---|---|
| [email protected] | 2023-01-11 |
| [email protected] | 2023-01-15 |
| [email protected] | 2023-01-17 |
How might I be able to achieve this in ADF?
You can use join transformations in dataflow within ADF to achieve this. https://learn.microsoft.com/en-us/azure/data-factory/data-flow-join
You can use left join between the datasets and then via filter transformation remove the rows with null value for the other column thereby getting only the delta values in final set