Search code examples
sqlreporting-servicesssisssis-2012dts

Check for duplicate rows while transferring data from text file to excel

Platform : SSIS

I am new to SSIS and trying to check for duplicate rows while transferring the data from text file to excel file. Heard about cache transform can be used but I am not really sure about it. Any suggestions?

Solution

  • One simple way to handle this is use an Aggregate transform between the source and destination. In it, group by all the columns in the source to eliminate duplicates. I have used this technique, and it works well.

    This could be slow if the source is large.