"ftplib.error_perm: 550 Operation not permitted" when trying to download the second file from the same directory using FTP_TLS

Got this error while running my script: ftplib.error_perm: 550 Operation not permitted

ftpdownloader.py :

import ftplib
import threading
import logging
import os


class Client:
    def __init__(self, host, port, username, passwd):
        self.host = host
        self.port = port
        self.username = username
        self.passwd = passwd
        self.ftp = None
        self.set_ftp()
        self.login_to_ftp()

    def set_ftp(self):
        self.ftp = ftplib.FTP_TLS(self.host)
        self.ftp.set_debuglevel(True)
        self.ftp.set_pasv(True)

    def login_to_ftp(self):
        self.ftp.login(self.username, self.passwd)
        self.ftp.prot_p()

    def download_file(self, filename, local_filename=None):
        logging.info('Will download this file : "%s"', filename)
        if local_filename is None:
            local_filename = filename
        self.ftp.voidcmd('TYPE I')
        sock = self.ftp.transfercmd('RETR ' + filename)

        def background():
            with open(local_filename, 'wb') as f:
                while True:
                    block = sock.recv(1024 * 1024)
                    if not block:
                        break
                    f.write(block)
                logging.debug('will close sock ...')
                sock.close()

        t = threading.Thread(target=background)
        t.start()
        while t.is_alive():
            t.join(20)
            self.ftp.voidcmd('NOOP')
        # self.ftp.close()
        logging.info("File downloaded with success : %s", local_filename)

    def get_directories_list(self, directory_path):
        logging.debug('Getting directories of directory : %s', directory_path)
        directories_list = []
        for elem in self.ftp.mlsd(directory_path):
            if elem[1]['type'] == 'dir':
                directories_list += [elem[0]]
        return directories_list

    def get_files_list(self, directory_path):
        logging.debug('Getting files of directory : %s', directory_path)
        files_list = []
        for elem in self.ftp.mlsd(directory_path):
            if elem[1]['type'] == 'file':
                files_list += [elem[0]]
        return files_list

    def get_all_files_of_directory(self, ftp_directory_root, ftp_directory_path, output_directory_path):
        current_position = ftp_directory_root + ftp_directory_path
        current_output_directory = output_directory_path + ftp_directory_path
        # self.set_ftp()
        # self.login_to_ftp()
        files_list = self.get_files_list(current_position)
        directories_list = self.get_directories_list(current_position)

        os.makedirs(current_output_directory, exist_ok=True)

        for file in files_list:
            self.download_file(current_position + '/' + file, current_output_directory + '/' + file)
            # self.set_ftp()
            # self.login_to_ftp()

        for directory in directories_list:
            if ftp_directory_path == '/':
                ftp_directory_path = ''
            self.get_all_files_of_directory(ftp_directory_root, ftp_directory_path + '/' + directory,
                                            output_directory_path)

Test script :

test.py :

from ftpdownloader import Client

ftp_host_name = 'ftp-hostname'
ftp_port = 21
ftp_username = 'username'
ftp_password = '*******'

client = Client(ftp_host_name, ftp_port, ftp_username, ftp_password)
client.get_all_files_of_directory('/directory1', '/', 'downloads/test')

FTP content :

 - directory1:
   - directory11:
     - file111.xml
     - file112.zip
   - directory12:
     - file121.xml
     - file122.zip

When I run my test.py script (with commented lines on ftpdownloader.py), the script :

Got the list of directories and files
Download the first founded file (file111.xml)
ERROR when trying to download the second file (file112.zip)-> ftplib.error_perm: 550 Operation not permitted

FTP_TLS debug logs :

*cmd* 'TYPE A'
*resp* '200 Type set to A'
*cmd* 'PASV'
*resp* '227 Entering Passive Mode (81,252,220,78,234,106).'
*cmd* 'MLSD /directory1'
*resp* '150 Opening ASCII mode data connection for MLSD'
*resp* '226 Transfer complete'
*cmd* 'TYPE A'
*resp* '200 Type set to A'
*cmd* 'PASV'
*resp* '227 Entering Passive Mode (81,252,220,78,234,187).'
*cmd* 'MLSD /directory1'
*resp* '150 Opening ASCII mode data connection for MLSD'
*resp* '226 Transfer complete'
*cmd* 'TYPE A'
*resp* '200 Type set to A'
*cmd* 'PASV'
*resp* '227 Entering Passive Mode (81,252,220,78,234,99).'
*cmd* 'MLSD /directory1/directory11'
*resp* '150 Opening ASCII mode data connection for MLSD'
*resp* '226 Transfer complete'
*cmd* 'TYPE A'
*resp* '200 Type set to A'
*cmd* 'PASV'
*resp* '227 Entering Passive Mode (81,252,220,78,234,131).'
*cmd* 'MLSD /directory1/directory11'
*resp* '150 Opening ASCII mode data connection for MLSD'
*resp* '226 Transfer complete'
*cmd* 'TYPE I'
*resp* '200 Type set to I'
*cmd* 'PASV'
*resp* '227 Entering Passive Mode (81,252,220,78,234,126).'
*cmd* 'RETR /directory1/directory11/file111.xml'
*resp* '150 Opening BINARY mode data connection for /directory1/directory11/file111.xml (879 bytes)'
*cmd* 'NOOP'
*resp* '226 Transfer complete'
*cmd* 'TYPE I'
*resp* '550 Operation not permitted'

Traceback (most recent call last):
  File "..../envs/back/lib/python3.9/code.py", line 90, in runcode
    exec(code, self.locals)
  File "<input>", line 11, in <module>
  File "..../back/tools/ftpdownload.py", line 86, in get_all_files_of_directory
    self.get_all_files_of_directory(ftp_directory_root, ftp_directory_path + '/' + directory,
  File "..../back/tools/ftpdownload.py", line 86, in get_all_files_of_directory
    self.get_all_files_of_directory(ftp_directory_root, ftp_directory_path + '/' + directory,
  File "..../back/tools/ftpdownload.py", line 79, in get_all_files_of_directory
    self.download_file(current_position + '/' + file, current_output_directory + '/' + file)
  File "..../back/tools/ftpdownload.py", line 30, in download_file
    self.ftp.voidcmd('TYPE I')
  File ".../envs/back/lib/python3.9/ftplib.py", line 286, in voidcmd
    return self.voidresp()
  File "..../envs/back/lib/python3.9/ftplib.py", line 259, in voidresp
    resp = self.getresp()
  File "..../envs/back/lib/python3.9/ftplib.py", line 254, in getresp
    raise error_perm(resp)
ftplib.error_perm: 550 Operation not permitted

So, I tried to run the script using login after downloading each file (uncomment lines for ftpdownloader.py script)! It's work but It's too slow comparing FileZilla or other FTP client!

Can you help please ?

Solution

There are at least two problems imo:

You are not reading a response to RETR.
You are not reading the responses in the order of the requests, so you misinterpret the responses. Consequently, you have many wrong assumptions which make it hard to understand the root cause.

I believe this is what is happening:

You send RETR and start the transfer.
You send NOOP and wait for a response.
But the server (correctly) responds to the requests in the order they are received. So it cannot respond NOOP, until RETR completes.
When the transfer finishes, the servers responds with 226 to the RETR.
Your code misinterprets the 226 as a response to NOOP
The server probably sends 550 as a response to NOOP
Your code does not read response to RETR (If it did, you will notice the mismatch between responses and answers at the first file already)
You send TYPE command for the second file
You read queued response 550 to RETR and misinterpret it as a response to TYPE.

I do not think the answer you have based your code on is correct. It indeed cannot handle more than one file, as it breaks pairing of requests and responses. If you believe the NOOP solves the freeze, then the correct solution might be to just send the NOOP command, without waiting for a response, using FTP.putcmd:

ftp.putcmd('NOOP')
noops_sent += 1

And then at the end of the the transfer, you have to wait for the RETR response using FTP.voidresp (see the code of FTP.retrbinary) and then wait for responses of all queued NOOP commands:

ftp.voidresp()
for _ in range(noops_sent):
    ftp.voidresp()

If the server is responding with 550 to NOOP, you will have to use FTP.getresp, as FTP.voidresp would throw on 550.