Python subprocess - OSError: [Errno 7] Argument list too long: 'perl'

Question

I have a Python script, which loops over an id list. Within a for loop, messages are downloaded from S3 bucket based on the particular id. Then a Perl script is called and the messages are decrypted one by one. The average size of a message is a few 100kB, not that big.

When run the script, only the first 2 messages decrypted successfully, and I got "OSError: [Errno 7] Argument list too long: 'perl'" error after that.

Python script

from subprocess import run


for id_ in ids:
    # this function calls "s3.get_object" and returns "StreamingBody()" object
    # from the "StreamingBody()" object the message is get as response["Body"].read()
    # the message is "bytes" type
    message = get_message_from_s3(id_)

    # call a Perl script to decrypt the message
    command = ["perl", "decrypt_message.pl", "decrypt_message", message]
    output = run(command, check=True, capture_output=True, text=True)
    ...

Perl script

#!/usr/bin/perl

use strict;
use warnings;


my $function = shift;
my $message = shift;

print decrypt_message($message);


sub decrypt_message {
    my $message = shift;

    my $decrypted_message = '';
    ...
    # decrypt here
    ...
    return $decrypted_message;
}

The size of the first 2 successful messages is about 20kB.The size of the unsuccessful ones are as mentioned a few 100kB. Can it be the problem? How to handle this?

Answer 1

100KB is pretty excessive for a command line; a typical limit might be as small as 32-64KB, but this is very system-dependent.

Modify your script to take input from stdin instead:

my $message = do { local $/; <> };

then launch the process as

command = ["perl", "decrypt_message.pl", "decrypt_message"]
output = subprocess.run(command, input=message, check=True, capture_output=True)
output = output.decode("utf8") # or whatever encoding you prefer