I get Ran out of input using multiprocessing and not when using threading

Question

BACKSTORY

I'm making a gui version of a crypto miner. I've completed the UI and i have a command line miner but when i try to start it from python there where problems start. I love threading so i added it to start the mining process and capturing the output to get info from it and display it to the user while still being able to use and interact with the app. At first it was working as intended but a problem came and that's is to stop the thread but as far as I know there isn't an official way of doing so. I switched to multiprocessing thinking it would work litle did I know it didn't. It says raises an error saying EOFError: Ran out of input, meaning that there is no output even tho when using threading it was working great.

IN SUMMARY

can't get output of a process when using multiprocessing even though it works with threading the error is raised when starting the multiprocessing.Process()

here is a snippet of the code

# start mining
    def start_mining(self):
        if self.mining_thread and self.mining_thread.is_alive():
            print("Already started mining")
            return

        settings["mining"] = True
        self.miningpage_frame.pack(fill="both")
        self.homepage_frame.pack_forget()

        mining_command = self.mining_cmd.get()

        # Function to capture and display the miner output
        def capture_output():
            process = subprocess.Popen(mining_command, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, shell=True,
                                       text=True)
            for line in process.stdout:
                self.output_text.insert(tk.END, line)
                self.output_text.see(tk.END)  # Scroll to the end to show the latest output

                # Check if mining should be stopped
                if settings["mining"] == False:
                    break  # Exit the loop and stop mining

            process.wait()
            return_code = process.returncode
            if return_code == 0:
                print("Mining process completed successfully.")
            else:
                print(f"Mining process failed with return code {return_code}.")


        # Start the mining thread
        self.mining_thread = threading.Thread(target=capture_output)
        self.mining_thread.start()
        #self.mining_process = multiprocessing.Process(target=capture_output)
        #self.mining_process.start()

Answer 1

First, there is no need to run capture_output in a new process since it is running the mining_command in a child process and just doing simple processing of the stdout output for which threading should be adequate.

But as to why you get the exception when using multiprocessing, I would guess you are running this under Windows or some platform which uses the spawn method to create child processes. Under Windows I would expect the exception you get because your class instance (whatever the class name is) needs to be pickled to the child process but classes with functions contained within methods such as compute_method cannot be pickled.

But even if you run this under Linux, which does not have this limitation, the code still cannot work. The new child process will be updating a copy of your class instance that is running in a different address space than the main process; the main process's copy of the class remains unchanged. That is, self.output_text in the main process's copy of the class instance will not have been modified.

Consider the following minimal, reproducible example:

from multiprocessing import Process
import subprocess

class Foo:
    def run(self):
        self.stdout_data = None

        def capture_output():
            process = subprocess.Popen("echo Hello!", stdout=subprocess.PIPE, shell=True,
                                       text=True)

            self.stdout_data, _ = process.communicate()

            print('child process self.stdout_data = ', self.stdout_data, end='')

        p = Process(target=capture_output)
        p.start()
        p.join()

        print('main process self.stdout_data = ', self.stdout_data)



# To support platforms that create child processes using
# the "spawn" method (e.g. Windows):
if __name__ == '__main__':
    foo = Foo()
    foo.run()

When this is run under Windows, we get an exception:

...
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Program Files\Python38\lib\multiprocessing\spawn.py", line 116, in spawn_main
    exitcode = _main(fd, parent_sentinel)
  File "C:\Program Files\Python38\lib\multiprocessing\spawn.py", line 126, in _main
    self = reduction.pickle.load(from_parent)
EOFError: Ran out of input
    ForkingPickler(file, protocol).dump(obj)
AttributeError: Can't pickle local object 'Foo.run.<locals>.capture_output'

Under Linux we get no exception. However, the main process's copy of foo has not been updated. The output is:

child process self.stdout_data =  Hello!
main process self.stdout_data =  None