How to stream from one SFTP to another SFTP using python and pysftp

Question

I want to transfer files from one SFTP server to another.

I don't want to write locally, and I don't want to read the whole file into memory.

Pysftp has getfo and putfo methods.

But getfo doesn't give you a reader -- it gives just writes to an opened fileobj.

I essentially want to pipe getfo to some file-like that can be read from, so I can provide this as the reader for putfo, to send to target.

Pseudocode:

buffer_obj = SomeBufferedIOReader()
source_sftp.getfo('/remote/path.txt.gz', buffer_obj)
target_sftp.putfo('/target/path.txt.gz', buffer_obj) # streaming commences

If I was doing this in bash I would create a fifo file, write to the fifo asynchronously, and then upload from that file. When the file was uploaded, the fifo would be exhausted and process would complete. Not sure how to do the equivalent in python.

Answer 1

OK ... duh.... pysftp provides an open method.

with source_sftp.open('source/path.txt.gz', 'rb') as f:
    f.prefetch()
    target_sftp.putfo('target/path.txt.gz', f)