I have a python program like this:
raw_data = sys.stdin.buffer.read(nbytes) # Read from standard input stream
# Do something with raw_data to get output_data HERE...
output_mask = output_data.tostring() # Convert to bytes
sys.stdout.buffer.write(b'results'+output_mask) # Write to standard output stream
Then I get the my_py.exe of this python program using Pyinstaller. I test the my_py.exe using subprocess.run() in Python. It is fine.
However, I need to call this my_py.exe in IDL. IDL has this tutorial on how to use its SPAWN command with pipes. So my IDL program which calls the my_py.exe is like this:
SPAWN['my_py.exe', arg], COUNT=COUNT , UNIT=UNIT
WRITEU, UNIT, nbytes, data_to_stream
READU, UNIT, output_from_exe
Unfortunately, the IDL program above hang at READU. Does anyone know the issue I have here? Is the problem in my python read and write?
You are missing a comma in the SPAWN command, although I imagine if that typo was in your code, IDL would issue a syntax error before you ever got to READU. But, if for some reason IDL is quietly continuing execution with an erroneous SPAWN call, maybe READU is hanging because it's trying to read some nonsense logical unit. Anyway, it should read:
SPAWN,['my_py.exe', arg], UNIT=UNIT
Here's the full syntax for reference:
SPAWN [, Command [, Result] [, ErrResult] ]
Keywords (all platforms): [, COUNT=variable] [, EXIT_STATUS=variable] [ ,/NOSHELL] [, /NULL_STDIN] [, PID=variable] [, /STDERR] [, UNIT=variable {Command required, Result and ErrResult not allowed}]
UNIX-Only Keywords: [, /NOTTYRESET] [, /SH]
Windows-Only Keywords: [, /HIDE] [, /LOG_OUTPUT] [, /NOWAIT]
I've eliminated the COUNT keyword, because, according to the documentation, COUNT contains the number of lines in Result, if Result is present, which it is not. In fact, Result is not even allowed here, since you're using the UNIT keyword. I doubt that passing the COUNT keyword is causing READU to hang, but it's unnecessary.
Also, check this note from the documentation to make sure that the array you are passing as a command is correct:
If Command is present, it must be specified as follows:
On UNIX, Command is expected to be scalar unless used in conjunction with the NOSHELL keyword, in which case Command is expected to be a string array where each element is passed to the child process as a separate argument.
On Windows, Command can be a scalar string or string array. If it is a string array, SPAWN glues together each element of the string array, with each element separated by whitespace.
I don't know the details of your code, but here's some further wild speculation:
You might try setting the NOSHELL keyword, just as a shot in the dark.
I have occasionally had problems with IDL not seeming to finish writing to disk when I haven't closed the file unit, so make sure that you are using FREE_LUN, UNIT after READU. I know you said it hangs at READU, but my thinking here is that maybe it's only appearing to hang, and just can't continue until the file unit is closed.
Finally, here's something that could actually be the problem, and is worth looking into (from the tutorial you linked to):
A pipe is simply a buffer maintained by the operating system with an interface that makes it appear as a file to the programs using it. It has a fixed length and can therefore become completely filled. When this happens, the operating system puts the process that is filling the pipe to sleep until the process at the other end consumes the buffered data. The use of a bidirectional pipe can lead to deadlock situations in which both processes are waiting for the other. This can happen if the parent and child processes do not synchronize their reading and writing activities.