Search code examples
rubydownloadftpstream

How to only read a few lines from a remote file?


Before downloading file, I need to set up a way it (the .csv typically, but not always) will be parsed.

I don't want to download the whole file especially if the "headers" do not match what is expected.

Is there a way to only download up until a certain number of byes and then gracefully kill the connection?


Solution

  • There's no explicit support for this in an FTP protocol.

    There's an expired draft for RANG command that would allow this:
    https://datatracker.ietf.org/doc/html/draft-bryan-ftp-range-08 But that's obviously supported by only new FTP servers.


    Though there's nothing that prevents you from initiating a normal (full) download and forcefully break it as soon you get the amount of data you need.

    All you need to do is to close the data transfer connection. This is basically what all FTP clients do, when an end user decides to abort the transfer.

    This approach might result in few error messages in an FTP server log.


    If you can use an SFTP protocol, then it's easy. The SFTP supports this natively.