Search code examples
pythonpython-3.xpippackagepypi

Python package won't work in real-life, only in virtual env

Background

I have a package, stacksearch which should work, at least in my virtual environment. (Here's the testpypi.)

(venv) RealName@theXliners-Mac stacksearch % python setup.py sdist bdist_wheel
running sdist
running egg_info
creating stacksearch.egg-info
writing stacksearch.egg-info/PKG-INFO
writing dependency_links to stacksearch.egg-info/dependency_links.txt
writing requirements to stacksearch.egg-info/requires.txt
writing top-level names to stacksearch.egg-info/top_level.txt
writing manifest file 'stacksearch.egg-info/SOURCES.txt'
reading manifest file 'stacksearch.egg-info/SOURCES.txt'
writing manifest file 'stacksearch.egg-info/SOURCES.txt'
running check
warning: Check: missing required meta-data: url

creating stacksearch-1.2.1.dev6

creating stacksearch-1.2.1.dev6/scripts
creating stacksearch-1.2.1.dev6/stacksearch.egg-info
copying files to stacksearch-1.2.1.dev6...
copying README.md -> stacksearch-1.2.1.dev6
copying setup.py -> stacksearch-1.2.1.dev6
copying scripts/stacksearch -> stacksearch-1.2.1.dev6/scripts
copying stacksearch.egg-info/PKG-INFO -> stacksearch-1.2.1.dev6/stacksearch.egg-info
copying stacksearch.egg-info/SOURCES.txt -> stacksearch-1.2.1.dev6/stacksearch.egg-info
copying stacksearch.egg-info/dependency_links.txt -> stacksearch-1.2.1.dev6/stacksearch.egg-info
copying stacksearch.egg-info/requires.txt -> stacksearch-1.2.1.dev6/stacksearch.egg-info
copying stacksearch.egg-info/top_level.txt -> stacksearch-1.2.1.dev6/stacksearch.egg-info
Writing stacksearch-1.2.1.dev6/setup.cfg
creating dist
Creating tar archive
removing 'stacksearch-1.2.1.dev6' (and everything under it)
running bdist_wheel
running build
running build_scripts
creating build
creating build/scripts-3.8
copying and adjusting scripts/stacksearch -> build/scripts-3.8
changing mode of build/scripts-3.8/stacksearch from 644 to 755
installing to build/bdist.macosx-10.9-x86_64/wheel
running install
running install_egg_info
Copying stacksearch.egg-info to build/bdist.macosx-10.9-x86_64/wheel/stacksearch-1.2.1.dev6-py3.8.egg-info
running install_scripts
creating build/bdist.macosx-10.9-x86_64/wheel/stacksearch-1.2.1.dev6.data
creating build/bdist.macosx-10.9-x86_64/wheel/stacksearch-1.2.1.dev6.data/scripts
copying build/scripts-3.8/stacksearch -> build/bdist.macosx-10.9-x86_64/wheel/stacksearch-1.2.1.dev6.data/scripts
changing mode of build/bdist.macosx-10.9-x86_64/wheel/stacksearch-1.2.1.dev6.data/scripts/stacksearch to 755
adding license file "LICENSE" (matched pattern "LICEN[CS]E*")
creating build/bdist.macosx-10.9-x86_64/wheel/stacksearch-1.2.1.dev6.dist-info/WHEEL
creating 'dist/stacksearch-1.2.1.dev6-py3-none-any.whl' and adding 'build/bdist.macosx-10.9-x86_64/wheel' to it
adding 'stacksearch-1.2.1.dev6.data/scripts/stacksearch'
adding 'stacksearch-1.2.1.dev6.dist-info/LICENSE'
adding 'stacksearch-1.2.1.dev6.dist-info/METADATA'
adding 'stacksearch-1.2.1.dev6.dist-info/WHEEL'
adding 'stacksearch-1.2.1.dev6.dist-info/top_level.txt'
adding 'stacksearch-1.2.1.dev6.dist-info/RECORD'
removing build/bdist.macosx-10.9-x86_64/wheel
(venv) RealName@theXliners-Mac stacksearch %

So, to simulate installing, I run pip install -e . (In my virtual enviorment) in my upper stacksearch folder.

(venv) RealName@theXliners-Mac stacksearch % pip install -e .
Obtaining file:///Users/RealName/projects/stacksearch
Requirement already satisfied: requests in /Users/bryanhu/venv/lib/python3.8/site-packages (from stacksearch==1.2.1.dev6) (2.24.0)
Requirement already satisfied: bs4 in /Users/bryanhu/venv/lib/python3.8/site-packages (from stacksearch==1.2.1.dev6) (0.0.1)
Requirement already satisfied: blessings in /Users/bryanhu/venv/lib/python3.8/site-packages (from stacksearch==1.2.1.dev6) (1.7)
Requirement already satisfied: lxml in /Users/bryanhu/venv/lib/python3.8/site-packages (from stacksearch==1.2.1.dev6) (4.5.2)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /Users/bryanhu/venv/lib/python3.8/site-packages (from requests->stacksearch==1
.2.1.dev6) (1.25.10)
Requirement already satisfied: certifi>=2017.4.17 in /Users/bryanhu/venv/lib/python3.8/site-packages (from requests->stacksearch==1.2.1.dev6) (2020.6.20
)
Requirement already satisfied: idna<3,>=2.5 in /Users/bryanhu/venv/lib/python3.8/site-packages (from requests->stacksearch==1.2.1.dev6) (2.10)
Requirement already satisfied: chardet<4,>=3.0.2 in /Users/bryanhu/venv/lib/python3.8/site-packages (from requests->stacksearch==1.2.1.dev6) (3.0.4)
Requirement already satisfied: beautifulsoup4 in /Users/bryanhu/venv/lib/python3.8/site-packages (from bs4->stacksearch==1.2.1.dev6) (4.9.1)
Requirement already satisfied: six in /Users/bryanhu/venv/lib/python3.8/site-packages (from blessings->stacksearch==1.2.1.dev6) (1.15.0)
Requirement already satisfied: soupsieve>1.2 in /Users/bryanhu/venv/lib/python3.8/site-packages (from beautifulsoup4->bs4->stacksearch==1.2.1.dev6) (2.0

.1)
Installing collected packages: stacksearch
  Attempting uninstall: stacksearch
    Found existing installation: stacksearch 1.2.1.dev6
    Uninstalling stacksearch-1.2.1.dev6:
      Successfully uninstalled stacksearch-1.2.1.dev6
  Running setup.py develop for stacksearch
Successfully installed stacksearch
(venv) RealName@theXliners-Mac stacksearch %

Which then I run stacksearch (the command):

(venv) RealName@theXliners-Mac stacksearch % stacksearch
usage: StackSearch [-h] [-j] [-o [OUTPUT]] [-s] [--sites SITES [SITES ...]] [-v] [query [query ...]]

    For searching StackOverflow and getting results that you can use.

    There are many other libraries/modules available that do the same
    thing. The reason you should use this is because this returns results that you can
    use. If ran from the command line, it'll return human readable results. If ran from
    another python script, it'll return some parsable JSON. Assuming you are utilizing
    this script's wonderful functions and objects.

positional arguments:
  query                 The query to search.

optional arguments:

  -h, --help            show this help message and exit
  -j, --json, --raw-data, -r, --raw
                        For outputting JSON data that you can use.
  -o [OUTPUT], --output [OUTPUT]
                        The output file.
  -s, --silent          Don't print the progress.
  --sites SITES [SITES ...]
                        The StackExchange sites to search.
  -v, -V, --version     Print the version number and exit.


 Judge a man by his questions rather than by his answers" - Voltaire

(venv) RealName@theXliners-Mac stacksearch %

It works! (To see what's in the files, check out the GitHub repo)

But sadly, that's not the reality: it doesn't work on the one I get from testpypi.

(venv) RealName@theXliners-Mac ~ % pip install -i https://test.pypi.org/simple/ stacksearch==1.2.1.dev6; stacksearch
Looking in indexes: https://test.pypi.org/simple/
Collecting stacksearch==1.2.1.dev6
  Downloading https://test-files.pythonhosted.org/packages/a4/09/5040ea9750192d64ccb634e022967793759592a47637731a845dbb2ab636/stacksearch-1.2.1.dev6-py3
-none-any.whl (4.4 kB)
Requirement already satisfied: bs4 in ./venv/lib/python3.8/site-packages (from stacksearch==1.2.1.dev6) (0.0.1)
Requirement already satisfied: blessings in ./venv/lib/python3.8/site-packages (from stacksearch==1.2.1.dev6) (1.7)
Requirement already satisfied: lxml in ./venv/lib/python3.8/site-packages (from stacksearch==1.2.1.dev6) (4.5.2)
Requirement already satisfied: requests in ./venv/lib/python3.8/site-packages (from stacksearch==1.2.1.dev6) (2.24.0)
Requirement already satisfied: beautifulsoup4 in ./venv/lib/python3.8/site-packages (from bs4->stacksearch==1.2.1.dev6) (4.9.1)
Requirement already satisfied: six in ./venv/lib/python3.8/site-packages (from blessings->stacksearch==1.2.1.dev6) (1.15.0)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in ./venv/lib/python3.8/site-packages (from requests->stacksearch==1.2.1.dev6) (1
.25.10)
Requirement already satisfied: certifi>=2017.4.17 in ./venv/lib/python3.8/site-packages (from requests->stacksearch==1.2.1.dev6) (2020.6.20)
Requirement already satisfied: chardet<4,>=3.0.2 in ./venv/lib/python3.8/site-packages (from requests->stacksearch==1.2.1.dev6) (3.0.4)

Requirement already satisfied: idna<3,>=2.5 in ./venv/lib/python3.8/site-packages (from requests->stacksearch==1.2.1.dev6) (2.10)
Requirement already satisfied: soupsieve>1.2 in ./venv/lib/python3.8/site-packages (from beautifulsoup4->bs4->stacksearch==1.2.1.dev6) (2.0.1)
Installing collected packages: stacksearch
Successfully installed stacksearch-1.2.1.dev6
Traceback (most recent call last):
  File "/Users/bryanhu/venv/bin/stacksearch", line 156, in <module>
    main(sys.argv[1:])
  File "/Users/bryanhu/venv/bin/stacksearch", line 25, in main
    from stacksearch import __version__
ModuleNotFoundError: No module named 'stacksearch'

So then I test it:

(venv) RealName@theXliners-Mac ~ % python
Python 3.8.2 (v3.8.2:7b3ab5921f, Feb 24 2020, 17:52:18)
[Clang 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import stacksearch
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'stacksearch'
>>>

My question

Why is when I do pip install -e ., I can install the package but it will only install the binaries if I install it from PyPi?

Solution

  • this might be the answer to your problem. posting it as an answer allows me to use the space and formatting i need.

    The basic problem is that you are not including the stacksearch folder in your dist file.

    One way I discovered this was downloading your tar ball from pypi.test . you can see what's in the folder and it only has the scripts. another way is looking at the results of your pip install -e . command. you can see it here:

    copying README.md -> stacksearch-1.2.1.dev6
copying setup.py -> stacksearch-1.2.1.dev6
copying scripts/stacksearch -> stacksearch-1.2.1.dev6/scripts
copying stacksearch.egg-info/PKG-INFO -> stacksearch-1.2.1.dev6/stacksearch.egg-info

    it's only copying the scripts folder

    you can run commands directly from setup.py by python -m setup.py <some command> and see what's getting built.

    possible solution it might be that find_packages is not finding everything you think it is. try importing that function into an interpreter and running it. here's the results in my own repository:

    find_packages()
Out[9]: 
['dicetables',
 'tests',
 'time_trials',
 'dicetables.eventsbases',
 'dicetables.factory',
 'dicetables.tools',
 'tests.eventsbases',
 'tests.factory',
 'tests.tools']
find_packages(include='dicetables')
Out[10]: []
find_packages(include=['dicetables'])
Out[11]: ['dicetables']

    not necessarily what you would expect. so, experiment with find_packages and what exactly is getting copied when you pip install -e .

    p.s. you didn't reveal any problems in your local directory because your scripts could import from the folders already present. if you had moved to a different directory, it should fail.