Apache FTP Client running in Docker has problem with retrieving files contents from external server

Question

I use Apache FTP Client to retrieve contents of files from FTP server. It runs on passive mode, retrieving with retrieveFileStream or retrieveFile and it works properly when I run it without Docker. There is a problem when I run this application from Docker container. It properly connects, log in, retrieving list of files. It has problem with retrieving content of file - it cannot retrieve full content of file, but only part of it. I thought that might be an buffer issue, I set values of all buffer settings to huge number and also tried to use buffer to read from stream (as was proposed on StackOverFlow when someone has problem with retrieving file), but without success. I also set the MTU settings of Docker to the value that my normal network uses, but also it did not help.

Here is some code:

    FTPClient ftpClient = new FTPClient();
    ftpClient.setControlKeepAliveTimeout(120);
    ftpClient.setDataTimeout(120000);
    ftpClient.setBufferSize(1024000);
    ftpClient.setReceiveBufferSize(1024000);
    ftpClient.setReceieveDataSocketBufferSize(1024000);
    LOGGER.trace("Trying to connect to host {}.", connectionSettings.getHost());
    ftpClient.connect(connectionSettings.getHost());
    int replyCode = ftpClient.getReplyCode();
    if (!FTPReply.isPositiveCompletion(replyCode)) {
        ftpClient.disconnect();
        LOGGER.error("Received negative reply code from host {}. Disconnected.", connectionSettings.getHost());
        throw new FtpClientNotReadyException();
    }

    ftpClient.enterLocalPassiveMode();

    LOGGER.trace("Received positive reply from host {}. Trying to log in.", connectionSettings.getHost());
    try {
        if (!ftpClient.login(connectionSettings.getUsername(), connectionSettings.getPassword())) {
            LOGGER.warn("FTP Client login failed for user {}", connectionSettings.getUsername());
            ftpClient.disconnect();
            throw new FtpClientNotReadyException();
        }
    } catch (IOException e) {
        ftpClient.disconnect();
        throw e;
    }
    LOGGER.trace("Successfully logged in to host {} on account {}.", connectionSettings.getHost(),
            connectionSettings.getUsername());
    try {
        if (Strings.isNotBlank(connectionSettings.getWorkingDirectory()) &&
                !ftpClient.changeWorkingDirectory(connectionSettings.getWorkingDirectory())) {
            LOGGER.warn("FTP Client could not change worked directory to {}.", connectionSettings.getWorkingDirectory());
            throw new FtpClientNotReadyException();
        }
    } catch (IOException e) {
        ftpClient.logout();
        ftpClient.disconnect();
        throw e;
    }
 InputStream retrievedFileContentInputStream = ftpClient.retrieveFileStream(fileName);

        byte[] fileContentByteArray = retrievedFileContentInputStream.readAllBytes();
        retrievedFileContentInputStream.close();
        try {
            checkFileSize(fileName, expectedFileSize, fileContentByteArray.length);
        } catch (FileCorruptedException | FileBeingCurrentlyUploadedException e) {
            throw e;
        } finally {
            if (!ftpClient.completePendingCommand()) {
                LOGGER.trace("Complete pending command was not successful.");
            }
            LOGGER.trace("Completed pending command.");
        }

These are solutions that I tried: 1) Buffer to read from InputStream:

InputStream ins = ftpClient.retrieveFileStream(fileName);
    ByteArrayOutputStream swapStream = new ByteArrayOutputStream();
    byte[] buff = new byte[2048];
    int rc = 0;
    int value =2048;
    while ((rc = ins.read(buff, 0, value)) > 0) {
        swapStream.write(buff, 0, rc);
    }
    byte[] fileByte = swapStream.toByteArray();
    ftpClient.completePendingCommand();

2) Changing output stream to input stream

        ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
        ftpClient.retrieveFile(fileName, outputStream);
        outputStream.close();
        byte[] fileContentByteArray = outputStream.toByteArray();

My typical log output when problem occurs:

Retrieved file size is lower than expected. Was: 228098. Expected: 236826.

Retrieved file size is lower than expected. Was: 1199. Expected: 1246.

Answer 1

Turns out the client on different platform than Windows received less bytes, because of different line endings (carriage return) that FTP server sent in ASCII MODE.