Search code examples
batch-filesftpwinscp

Skip duplicate lines when downloading text file from SFTP server with WinSCP

Is there a command for a WinSCP batch file to only append non duplicate values in the remote file to the local file?

This is my batch command:

@echo off

"C:\Program Files (x86)\WinSCP\WinSCP.com" ^
  /log="C:\Users\U37615\Documents\POCLOSEBATCHLOG\WinSCP.log" /ini=nul ^
  /command ^
    "open xxxxxx -hostkey=""ssh-rsa 2048 xx:xx:xx:xx:xx:xx:xx....""" ^
    "get -append -delete xxxxxx"" ""xxxxxxx""" ^
    "exit"

set WINSCP_RESULT=%ERRORLEVEL%
if %WINSCP_RESULT% equ 0 (
  echo Success
) else (
  echo Error
)

pause
exit /b %WINSCP_RESULT%

I have a text file on the remote server that I want to append to the local file, but I want to only append items not already on the local list. Any ideas on a command or work around to accomplish this?

Solution

  • Absolutely not. This is such a custom requirement, that you cannot reasonably expect any SFTP client to support it out of the box.

    Download the remote file (no -append) to a temporary local file (temp.txt).

    And then do something like:

    powershell -Command "type local.txt, temp.txt | sort -unique > combined.txt"

    Some references: