I'm trying to create a powershell script to copy only the new entries that match a pattern (can be one or multiple lines) from one file to another, every time the script is run. The source file is updated randomly by an application but the request is to copy the last entries every hour.
The solution I'm working on is to take the last entry from previous run stored in a file and the compare with the last entry from the file, and if these don't match, then start copying the new lines after that one. That is the part when I'm stuck, I can't figure it how to indicate that, instead I'm copying the whole content every time.
This is what I got so far:
Write-Host "Declaring the output log file ..... " -ForegroundColor Yellow
$destinationfile = 'C:\file\out\output.log'
Write-Host "Getting the last line of the source file ..... " -ForegroundColor Yellow
$sourcefile = 'C:\app\blade\inputlogs.log'
$sourcefilelastline = Get-Content $originfile | Select-Object -last 1
$sourcefilelastline
Write-Host "Getting the last line of the destination file ..... " -ForegroundColor Yellow
$destinationfilelastline = Get-Content $destinationfile | Select-Object -last 1
$destinationfilelastline
if ($sourcefilelastline -eq $destinationfilelastline){
Write-Host "Skipping the process ..... " -ForegroundColor Yellow
}
else{
Write-Host "Reading the source log file and updating destination file ..... " -ForegroundColor Yellow
$sourcefilecontent = Get-Content -Path $sourcefile | Where-Object { $_ -ne '' } | Select-String -Pattern 'error' -CaseSensitive -SimpleMatch
$sourcefilecontent | Add-Content $destinationfile
}
Any ideas on how to get this done ? Thanks.
Get-content have a switch 'tail' which lets you read the last rows from a file. Per Microsoft own words:
The Tail parameter gets the last line of the file. This method is faster than retrieving all the lines in a variable and using the [-1] index notation.
You can use it in your case start from the bottom line and go up until they match.
<# The Function.
.SYNOPSIS
Recive X last number of lines from a file
.DESCRIPTION
Using the "Tail" parameter of get-content we can get the X number of lines from a file.
This should dramatically improve performance instead of reading the entire file.
.PARAMETER FilePath
Mandatory parameter.
Path to the file to get the X number of lines from.
.PARAMETER LastLines
Optional parameter.
How many lines to read from the end of the file.
Default value = 1
.EXAMPLE
Get-FileLastContent -FilePath "C:\Windows\System32\drivers\etc\hosts" -LastLines 1
.NOTES
General notes
#>
function Get-FileLastContent {
param (
[Parameter(
Position = 0,
Mandatory = $true,
HelpMessage = 'Path to File')]
[ValidateScript({ Test-Path -Path $_ })]
[string]
$FilePath,
[Parameter(
Position = 1,
HelpMessage = 'Number of lines lines to check')]
[ValidateNotNullOrEmpty()]
[int]
$LastLines = 1
)
$FileContent = Get-Content -Path $FilePath -Tail $LastLines
Return $FileContent
}
## Inisialization of Variables, as sometimes re-running the script is using values from last run.
$LastLines = 1
$destinationfilelastline = ''
$sourcefilelastline = ''
##Declaring the Destnation File
Write-Host -Object "Declaring the output log file ..... " -ForegroundColor Yellow
$destinationfile = 'C:\file\out\output.log'
## Using the function to recive it's last line
Write-Host -Object "Getting the last line of the destination file ..... " -ForegroundColor Yellow
$destinationfilelastline = Get-FileLastContent -FilePath $destinationfile
$destinationfilelastline
## Declaring the source file and using the function to get it's last line
Write-Host -Object "Getting the last line of the source file ..... " -ForegroundColor Yellow
$sourcefile = 'C:\app\blade\inputlogs.log'
$sourcefilelastline = Get-FileLastContent -FilePath $sourcefile -LastLines 1
Write-Host -Object $sourcefilelastline
## if source file is empty, empty destination file as well
if ($sourcefilelastline.Length -eq 0) {
Write-Host -Object "Soruce file is empty, clearing destination" -ForegroundColor Yellow
Set-Content -Path $destinationfile -Value "" -Force -Encoding UTF8 -NoNewline
}
## If source file is not empty, but not matching the destination file
elseif (($sourcefilelastline -ne $destinationfilelastline)) {
Write-Host -Object "Reading the source log file and updating destination file ..... " -ForegroundColor Yellow
## if destination file is not empty, loop in the source file bottom-up until it is finding a line matching the destinaion file
if ($destinationfilelastline.Length -gt 0) {
while (($sourcefilelastline[0] -ne $destinationfilelastline) -and ($LastLines -le $sourcefile.Length)) {
$LastLines = $LastLines + 1
$sourcefilelastline = Get-FileLastContent -FilePath $sourcefile -LastLines $LastLines
}
## If found a match in source file compared to the destination file
if (($sourcefilelastline[0] -eq $destinationfilelastline)) {
## Prepare sourcefilelastline variable for export, skip the first result as it is already in the destination file (they match).
$sourcefilelastline = $sourcefilelastline | Select-Object -Skip 1 | Where-Object { $_ -ne '' } | Select-String -Pattern 'error' -CaseSensitive -SimpleMatch
# adding new line at the start
$sourcefilelastline[0] = "`n" + $sourcefilelastline[0]
## Export the sourcefilelastline to the destination file
$sourcefilelastline | Out-File -FilePath $destinationfile -Force -Encoding UTF8 -Append
}
else {
## it means that the sourcefile was overwriteen since last check with complete new data
## Overwrite destination file with the last line of source file
$sourcefilelastline[($sourcefilelastline.Length-1)] | Out-File -FilePath $destinationfile -Force -Encoding UTF8
}
}
#If no match found in sourcefile compare to the destination file
# if destination file is empty, copy the last line of the source file to the destination file.
else {
$sourcefilelastline | Out-File -FilePath $destinationfile -Force -Encoding UTF8
}
}
## Skip if source and destination are equal
else {
Write-Host -Object "Skipping the process ..... " -ForegroundColor Yellow
}