Apache FTPClient fails to retrieve a file when it's almost downloaded

Question

Apache's FTPClient fails to download a file which is perfectly downloaded by FileZilla.

Basically, what I'm trying to do after a successfull login and listing is to download one specific file:

FTPClient client = new FTPClient();
client.setDataTimeout(20000);
client.setConnectTimeout(20000);
client.setBufferSize(65536);
//...
client.connect(host);
client.login(user, pswd);
// response validation
client.enterLocalPassiveMode();
// some listings with validations

InputStream in = new BufferedInputStream(client.retrieveFileStream(ftpFilePath), 16384);
// ...
byte[] buffer = new byte[8192];
while ((rd = in.read(buffer)) > 0) {
// .. reading the file and updating download progress

The last lines could be easily replaced with FTPClient's file download with virtually the same result, but then we can't track the download progress:

client.setControlKeepAliveTimeout(30);
client.retrieveFile(ftpFilePath, new org.apache.commons.io.output.NullOutputStream());

As a result of all those actions I can see the file is downloading until something very close to 100% and then the exception occurs:

java.net.SocketTimeoutException: Read timed out
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.socketRead(Unknown Source)
        at java.net.SocketInputStream.read(Unknown Source)
        at java.net.SocketInputStream.read(Unknown Source)
        at java.io.FilterInputStream.read(Unknown Source)
        at java.io.BufferedInputStream.fill(Unknown Source)
        at java.io.BufferedInputStream.read1(Unknown Source)
        at java.io.BufferedInputStream.read(Unknown Source)
        at java.io.FilterInputStream.read(Unknown Source)
        at <my code from here on>

There seems to be no firewall, but when the internet connection speed is better the download succedes (probably, some sort of timeout is hit). I would think that the problem is with the connection, but the thing is FileZilla succedes to download the same file.

So, I can reformulate my question like this: how do I make my FTPClient behave like FileZilla when downloading a file. Probably, there's some complicated retry on ping logic that I'm not aware of.

Commons Net: commons-net-3.6

FTP Server: proftpd-1.3.3g-6.el5 on CentOS 5.8 with a default configuration, does not support FTP over TLS.

Answer 1

I don't know what is the actual reason for this phenomena (time out on the last chunk of a file), but I've checked with Wireshark what it is that FileZilla does to download a file and found that it suffers from the very same problem with the same timeout and it is reconnecting to a server and sending a REST FTP query to restart the download of this specific file from when it was aborted that is to download only the last chunk.

So, the solution would be to add some sort of retry logic to the download process, so that this chunk of code:

InputStream in = new BufferedInputStream(client.retrieveFileStream(ftpFilePath), 16384);
// ...
byte[] buffer = new byte[8192];
while ((rd = in.read(buffer)) > 0) {

Becomes this:

InputStream in = new BufferedInputStream(client.retrieveFileStream(ftpFilePath), 16384);
// ...
byte[] buffer = new byte[8192];
long totalRead = 0;
for (int resumeTry = 0; resumeTry <= RESUME_TRIES; ++resumeTry) {
    try {
        while ((rd = in.read(buffer)) > 0) {
            //...
            totalRead += rd;
        }
        break;
    } catch (SocketTimeoutException ex) {
        // omitting exception handling
        in.close();
        client.abort();
        client.connect(...);
        client.login(...);
        client.setFileType(FTPClient.BINARY_FILE_TYPE);
        client.enterLocalPassiveMode();
        client.setRestartOffset(totalRead);
        in = client.retrieveFileStream(...);
        if (in == null) {
            // the FTP server doesn't support REST FTP query
            throw ex;
        }
        in = new BufferedInputStream(in, 16384);
    }
}