In my script I try to separate a csv that I imported into an array, into 11 different arrays. I do this by checking a id in each array object which decides into which new array the data should be transferred. I achieve this by creating new ps-objects with the relevant information and then adding them to the other arrays.
The source file I have is 700.000 lines long and the problem is that my foreach loop in which I separate the data gets progressively slower. I tested with 5000 lines and they were through in a few seconds, but with the complete data set I had to cancel the execution of the script after 1:15h.
This is my script:
$Nest1 = @()
$Nest2 = @()
$Nest3 = @()
$Nest4 = @()
$Nest5 = @()
$Nest6 = @()
$Nest7 = @()
$Nest8 = @()
$Nest9 = @()
$Nest10 = @()
$Errornest = @()
$counter = 0
$Pfad = 'C:\Users\Measurements\Documents\Data.csv'
$Header = 'Type','Date','Time','Nestid','Value','goempty'
$Data = Import-Csv $pfad -Delimiter ";" -Header $Header
$Data = $Data | where-Object {$_.Value -ne "Waage antwortet nicht" -and $_.Date -ne ""}
$total = $Data.count
Foreach ($Dat in $Data){
$counter ++
Write-Progress -Activity 'Processing Data' -PercentComplete (($counter / $total) * 100)
if($Dat.Nestid -eq 1){
$Nestobject = New-Object psobject
$Nestobject | Add-Member -MemberType NoteProperty -name 'Type' -Value $Dat.type
$Nestobject | Add-Member -MemberType NoteProperty -name 'Date' -Value $Dat.Date
$Nestobject | Add-Member -MemberType NoteProperty -name 'Time' -Value $Dat.Time
$Nestobject | Add-Member -MemberType NoteProperty -name 'Value' -Value $Dat.Value
$Nest1 += $Nestobject
}
elseif($Dat.Nestid -eq 2){
$Nestobject = New-Object psobject
$Nestobject | Add-Member -MemberType NoteProperty -name 'Type' -Value $Dat.type
$Nestobject | Add-Member -MemberType NoteProperty -name 'Date' -Value $Dat.Date
$Nestobject | Add-Member -MemberType NoteProperty -name 'Time' -Value $Dat.Time
$Nestobject | Add-Member -MemberType NoteProperty -name 'Value' -Value $Dat.Value
$Nest2 += $Nestobject
}
elseif($Dat.Nestid -eq 3){
$Nestobject = New-Object psobject
$Nestobject | Add-Member -MemberType NoteProperty -name 'Type' -Value $Dat.type
$Nestobject | Add-Member -MemberType NoteProperty -name 'Date' -Value $Dat.Date
$Nestobject | Add-Member -MemberType NoteProperty -name 'Time' -Value $Dat.Time
$Nestobject | Add-Member -MemberType NoteProperty -name 'Value' -Value $Dat.Value
$Nest3 += $Nestobject
}
elseif($Dat.Nestid -eq 4){
$Nestobject = New-Object psobject
$Nestobject | Add-Member -MemberType NoteProperty -name 'Type' -Value $Dat.type
$Nestobject | Add-Member -MemberType NoteProperty -name 'Date' -Value $Dat.Date
$Nestobject | Add-Member -MemberType NoteProperty -name 'Time' -Value $Dat.Time
$Nestobject | Add-Member -MemberType NoteProperty -name 'Value' -Value $Dat.Value
$Nest4 += $Nestobject
}
elseif($Dat.Nestid -eq 5){
$Nestobject = New-Object psobject
$Nestobject | Add-Member -MemberType NoteProperty -name 'Type' -Value $Dat.type
$Nestobject | Add-Member -MemberType NoteProperty -name 'Date' -Value $Dat.Date
$Nestobject | Add-Member -MemberType NoteProperty -name 'Time' -Value $Dat.Time
$Nestobject | Add-Member -MemberType NoteProperty -name 'Value' -Value $Dat.Value
$Nest5 += $Nestobject}
elseif($Dat.Nestid -eq 6){
$Nestobject = New-Object psobject
$Nestobject | Add-Member -MemberType NoteProperty -name 'Type' -Value $Dat.type
$Nestobject | Add-Member -MemberType NoteProperty -name 'Date' -Value $Dat.Date
$Nestobject | Add-Member -MemberType NoteProperty -name 'Time' -Value $Dat.Time
$Nestobject | Add-Member -MemberType NoteProperty -name 'Value' -Value $Dat.Value
$Nest6 += $Nestobject}
elseif($Dat.Nestid -eq 7){
$Nestobject = New-Object psobject
$Nestobject | Add-Member -MemberType NoteProperty -name 'Type' -Value $Dat.type
$Nestobject | Add-Member -MemberType NoteProperty -name 'Date' -Value $Dat.Date
$Nestobject | Add-Member -MemberType NoteProperty -name 'Time' -Value $Dat.Time
$Nestobject | Add-Member -MemberType NoteProperty -name 'Value' -Value $Dat.Value
$Nest7 += $Nestobject}
elseif($Dat.Nestid -eq 8){
$Nestobject = New-Object psobject
$Nestobject | Add-Member -MemberType NoteProperty -name 'Type' -Value $Dat.type
$Nestobject | Add-Member -MemberType NoteProperty -name 'Date' -Value $Dat.Date
$Nestobject | Add-Member -MemberType NoteProperty -name 'Time' -Value $Dat.Time
$Nestobject | Add-Member -MemberType NoteProperty -name 'Value' -Value $Dat.Value
$Nest8 += $Nestobject}
elseif($Dat.Nestid -eq 9){
$Nestobject = New-Object psobject
$Nestobject | Add-Member -MemberType NoteProperty -name 'Type' -Value $Dat.type
$Nestobject | Add-Member -MemberType NoteProperty -name 'Date' -Value $Dat.Date
$Nestobject | Add-Member -MemberType NoteProperty -name 'Time' -Value $Dat.Time
$Nestobject | Add-Member -MemberType NoteProperty -name 'Value' -Value $Dat.Value
$Nest9 += $Nestobject}
elseif($Dat.Nestid -eq 10){
$Nestobject = New-Object psobject
$Nestobject | Add-Member -MemberType NoteProperty -name 'Type' -Value $Dat.type
$Nestobject | Add-Member -MemberType NoteProperty -name 'Date' -Value $Dat.Date
$Nestobject | Add-Member -MemberType NoteProperty -name 'Time' -Value $Dat.Time
$Nestobject | Add-Member -MemberType NoteProperty -name 'Value' -Value $Dat.Value
$Nest10 += $Nestobject}
else{
$Nestobject = New-Object psobject
$Nestobject | Add-Member -MemberType NoteProperty -name 'Type' -Value $Dat.type
$Nestobject | Add-Member -MemberType NoteProperty -name 'Date' -Value $Dat.Date
$Nestobject | Add-Member -MemberType NoteProperty -name 'Time' -Value $Dat.Time
$Nestobject | Add-Member -MemberType NoteProperty -name 'Value' -Value $Dat.Value
$Errornest += $Nestobject}
}
I already thought about writing a function for creating the accordig object, but i guess that would only bring better readability to the script and not really increase the performance.
Just in case the Data structure i have also is relevant, the source csv looks something like this:
I;17.07.2023;11:16:32;9;000734d9c1;
W;17.07.2023;11:16:31;7;120.79;g
W;17.07.2023;11:16:31;3;Waage antwortet nicht;
W;17.07.2023;11:16:32;4;5.08;g
W;17.07.2023;11:16:33;10;148.49;g
W;17.07.2023;11:16:32;2;49.91;g
W;17.07.2023;11:16:33;9;117.13;g
I;17.07.2023;11:16:33;9;000734d9c1;
I;17.07.2023;11:16:34;9;000734d9c1;
For starters, it's a very bad practice to paste examples as screenshots. Why don't you use a design like:
$Nest1 = $Data | Where-Object {$_.Nestid -eq '1'} | Select Type, Date, Time, Value
This should work faster. But I think you first need to understand why 11 separate arrays are needed in the first place.