Search code examples
pythonpipsetuptoolspython-wheel

`'pip wheel .` much slower than 'setup.py bdist_wheel`


With this setup.py (not yet committed to the root of this project) Python 2.7 , pip wheel . takes almost 20 minutes

$:~/projects/cfgov-refresh$ time pip wheel .                                                                                                     
   Processing /Users/karchnerr/projects/cfgov-refresh
   Building wheels for collected packages: cfgov
   Running setup.py bdist_wheel for cfgov ... done
   Stored in directory: /Users/karchnerr/projects/cfgov-refresh
   Successfully built cfgov

    real    19m50.061s
    user    0m29.641s
    sys     1m21.651s

While setup.py bdist_wheel takes mere seconds.

rk@:~/projects/cfgov-refresh$ time python setup.py bdist_wheel
    [... ommited a bunch of noise ...]
    real    0m2.575s
    user    0m0.586s
    sys     0m0.309s

Any idea what accounts for such a big difference? Is there something I should be doing differently?


Using the ts command from moreutils, I can see that a huge chunk of the time is spent before pip even starts processing setup.py

[2017-01-17 15:41:36] Processing /Users/karchnerr/projects/cfgov-refresh
[2017-01-17 15:54:42]   Running setup.py     (path:/private/var/folders/_f/3lrtcs492pnbf6zcql9t0n380000gp/T/pip-p6TG9c-    build/setup.py) egg_info for package from file:///Users/karchnerr/projects/cfgov-refresh

Solution

  • Here's what I came up with:

    • pip wheel insists on copying the entire project into a temp directory, using shutil.copytree
    • copytree is pretty slow
    • this repo has a lot of files.