Search code examples
powershellcsvreconciliation

how can I correct my reconciliation of .csv files to remove dupes/nulls


I have been using code from this answer to check for additions/changes to class rosters from MS Teams:

$set = [System.Collections.Generic.HashSet[string]]::new(
    [string[]] (Import-CSV -Path stundent.csv).UserPrincipalName,
    [System.StringComparer]::InvariantCultureIgnoreCase
)
Import-Csv ad.csv | Where-Object { $set.Add($_.UserPrincipalName) } |
    Export-Csv path\to\output.csv -NoTypeInformation

Ideally, I want to be able to check if there have been removals when compared to a new file, swap the import file positions, and check for additions. If my files look like Source1 and Source2 (below), the check for removals would return Export1, and the check for additions would return Export2. Since there will be multiple instances of students across multiple classes, I want to include TeamDesc in the filter query to make sure only the specific instance of that student with that class is returned.

Source1.csv

TeamDesc UserPrincipalName Name
Team 1 [email protected] john smith
Team 1 [email protected] nancy drew
Team 2 [email protected] harvey dent
Team 3 [email protected] john smith

Source2.csv

TeamDesc UserPrincipalName Name
Team 1 [email protected] nancy drew
Team 2 [email protected] harvey dent
Team 2 [email protected] tim tams
Team 3 [email protected] john smith

Export1.csv

TeamDesc UserPrincipalName Name
Team 1 [email protected] john smith

Export2.csv

TeamDesc UserPrincipalName Name
Team 2 [email protected] tim tams

Solution

  • Try the following, which uses Compare-Object to compare the CSV files by two column values, simply by passing the property (column) names of interest to -Property; the resulting output is split into two collections based on which input side a differing property combination is unique to, using the intrinsic .Where() method:

    $removed, $added = (
      Compare-Object (Import-Csv Source1.csv) (Import-Csv Source2.csv) -PassThru `
                     -Property TeamDesc, UserPrincipalName
    ).Where({ $_.SideIndicator -eq '=>' }, 'Split')
    
    $removed | 
      Select-Object -ExcludeProperty SideIndicator |
      Export-Csv -NoTypeInformation Export1.csv
    
    $added |
      Select-Object -ExcludeProperty SideIndicator |
      Export-Csv -NoTypeInformation Export2.csv