Search code examples
python-3.xsftpmessage-queuemessaging

File transfer between apps

We are building a Python app to receive file from another app, process the received file and send the processed file back to that app.

For the transfer of file we are considering options SFTP and messaging queue.Below are our requirements

  1. Securely transfer files
  2. Acknowledge the file and notify the sender apps for any failures
  3. Load balance the requests for file processing
  4. To be able verify the status of file sent by the sender app

Which do you think is better suited given this scenario SFTP or message queue? I know each one has its own merits and demerits but wanted to get some insight and also find if something is being overlooked

Solution

  • Message-driven systems depend almost entirely on the quality and reliability of their broker, and as I have no experience with QPID's brokers I cannot comment on that crucial facet of the decision. However, Qpid uses AMQP, and should you run into trouble with the selected broker, switching to another should not be overly complex.

    Now that is is out of the way, let's have a look at your requirements. In my opinion your requirements are mostly "natural" properties of a good messaging system, so you get properly working, reliable implementations of your requirements without having to layer mechanisms on top of a file transfer protocol.

    • Securely transfer files

    AMQP and SFTP are probably quite comparable wrt security, if properly implemented

    • Acknowledge the file and notify the sender apps for any failures

    Almost automatic using the appropriate delivery guarantee mode for QPID, messy to do via sftp.

    • Load balance the requests for file processing

    Also effortless by having all consumers attached to the same queue, avoiding double consumption in a way that avoids message loss is hard to get right with just sftp.

    • To be able verify the status of file sent by the sender app

    Not 100% sure what you mean here, but AMQP has mechanisms to ensure that the sender knows with certainty that a message has been properly accepted by the broker, and that a file has been correctly received and processed by the receiver (manual acknowledgement mode).

    One argument that wasn't mentioned yet is high availability: message brokers, being crucial elements of many enterprise processing chains, have extensive high availability features to avoid downtime and message loss.