Search code examples
pythonmultithreadinghttpresponsegil

How does the GIL in python affect the downloading of webpages in parallel?

My understanding of the background of this question:

  • The GIL limits python to one thread running at a time.
  • Because of the GIL, multithreading long calculations is not useful
  • Threading can still be useful
  • Threading may be useful with I/O operations

Therefore my question is:

How would the GIL affect the downloading of a requested webpage? Would making parallel webpage request be a good use of python threading? Because downloading a webpage is an I/O operation, would this mean that threading is useful?

I would imagine that one thread would make a request > another thread would get passed control at some point and make its own request > another thread would get passed control, etc. And then data would start streaming in, but how would this be handled? Would downloads get interrupted? I suppose I am lacking the low-level understanding of response handling by the OS, the python interpreter, and the OS.

Solution

  • The GIL won't hurt you here.

    For I/O bound tasks (like downloading webpages), the GIL is not a problem. Python releases the GIL when I/O is happening, which means all the threads will be able execute the requests in parallel. Whenever you're doing processing of the downloaded pages, this is where the GIL can hurt you.

    You're right about the general rule of thumb: you can do I/O and the GIL doesn't hurt you, but with processor-bound tasks, you should try to use multiprocessing instead.

    For more info about the GIL, you can check out David Beazley's talk