Search code examples
pythonterminaljupyter-notebookmultiprocessingpython-multiprocessing

Running file on Terminal not doing the same thing as running on Jupyter Notebook?

I have a python file that basically runs a main() function which scrapes data given a given keyword. I am trying to do multiprocessing so my main function in my Jupyter Notebook looks like

p1 = multiprocessing.Process(target=main, args=['cosmetics'])
p2 = multiprocessing.Process(target=main, args=['airpod pro case'])

if __name__ == '__main__':
    p1.start()
    p2.start()

Now this runs perfectly, it basically scrapes data given a keyword, saves it to a csv file (in this case, 'cosmetics.csv' and 'airpod pro case.csv' then calls another function to go through each column that contains a url from the saved csv file.

However, I wanted to run this on my terminal/cmd and I changed the above code to:

def multi_process(item_1, item_2):
    p1 = multiprocessing.Process(target=main, args=[item_1])
    p2 = multiprocessing.Process(target=main, args=[item_2])
    #p3 = multiprocessing.Process(target=main, args=['hair dryer'])

    if __name__ == '__main__':
        p1.start()
        p2.start()
        #p3.start()
    
item_1 = sys.argv[1]
item_2 = sys.argv[2]
multi_process(item_1, item_2)

then I saved the file as a .py file and ran this line on my terminal/cmd

> python3 /Users/Name/Desktop/DE/Scrape.py "cosmetics" "airpod pro case"

FileNotFoundError: [Errno 2] File b'/Users/Name/Desktop/DE/cosmetics 1.csv' does not exist: b'/Users/Name/Desktop/DE/cosmetics 1.csv'

FileNotFoundError: [Errno 2] File b'/Users/Name/Desktop/DE/airpod pro case 1.csv' does not exist: b'/Users/Name/Desktop/DE/airpod pro case 1.csv'

and I get this error saying that it can't find the csv files which led me to check the folder and find out that the csv files are not being saved which should be.

Does running python file on termina/cmd not allow saving csv files? I can't seem to figure out what the problem is.

Solution

  • My bad guys, it was a stupid mistake on my part. I didn't actually do cd Desktop/DE to go to the DE folder on my terminal/cmd line and ran the code when I just opened the terminal/cmd, making the files automatically save somewhere else.