Search code examples
pythonsetuptoolssetup.py

Python code doesn't see data files after I packaged it with setuptools and installed with pip

I have packaged some python code with setuptools. The code uses a data file, but after installing the package with pip, the code cannot find the data file. What am I doing wrong?

Below are the file structure and file contents. The files may be found here. The python code setup.py is trying to use the mesa/a.txt data file. The code works fine if I don't package+pip it.

|
|- tm/
|  |- __init__.py
|  |- test2.py
|- mesa/
|  |- __init__.py
|  |- a.txt
|- MANIFEST.in
|- setup.py

MANIFEST.in: 

include mesa/a.txt

setup.py:

from setuptools import setup
setup(name='bobab',
      version='0.1',
      py_modules=['tm.test2'],
      author_email='[email protected]',
      package_data = {
        'tm': ['mesa/a.txt']
        },
    )

'mesa/a.txt:

hello world!

tm/test2.py:

import os

def main():
    print 'hi' 
    print open(os.path.join('..', 'mesa', 'a.txt'), 'r').read() # print file content

if __name__ == "__main__":
    main()

I use the command python setup.py sdist to create the package. I use the following command to install the package:

unzip bobab-0.1.zip
cd bobab-0.1
python setup.py install

The package doesn't see the data file mesa/a.txt, as illustrated in the error message below:

Python 2.7.12 (default, Dec  4 2017, 14:50:18)
[GCC 5.4.0 20160609] on linux2
>>> import tm.test2
>>> tm.test2.main()
hi
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "tm/test2.py", line 5, in main
    print open(os.path.join('..', 'mesa', 'a.txt'), 'r').read()
IOError: [Errno 2] No such file or directory: '../mesa/a.txt'

I use Python 2.7.

Below are the packaging and installation logs. Interestingly, the packaging log mentions that the data file mesa/a.txt was added to the package, but the installation log doesn't mention the presence of it.

Packaging log:

C:\Users\Franck\Documents\GitHub\misc\src\test>python setu
p.py sdist
running sdist
running egg_info
creating bobab.egg-info
writing bobab.egg-info\PKG-INFO
writing top-level names to bobab.egg-info\top_level.txt
writing dependency_links to bobab.egg-info\dependency_links.txt
writing pbr to bobab.egg-info\pbr.json
writing manifest file 'bobab.egg-info\SOURCES.txt'
reading manifest file 'bobab.egg-info\SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'bobab.egg-info\SOURCES.txt'
warning: sdist: standard file not found: should have one of README, README.rst,
README.txt

running check
warning: check: missing required meta-data: url

warning: check: missing meta-data: either (author and author_email) or (maintain
er and maintainer_email) must be supplied

creating bobab-0.1
creating bobab-0.1\bobab.egg-info
creating bobab-0.1\mesa
creating bobab-0.1\tm
copying files to bobab-0.1...
copying MANIFEST.in -> bobab-0.1
copying setup.py -> bobab-0.1
copying bobab.egg-info\PKG-INFO -> bobab-0.1\bobab.egg-info
copying bobab.egg-info\SOURCES.txt -> bobab-0.1\bobab.egg-info
copying bobab.egg-info\dependency_links.txt -> bobab-0.1\bobab.egg-info
copying bobab.egg-info\pbr.json -> bobab-0.1\bobab.egg-info
copying bobab.egg-info\top_level.txt -> bobab-0.1\bobab.egg-info
copying mesa\a.txt -> bobab-0.1\mesa
copying tm\__init__.py -> bobab-0.1\tm
copying tm\test2.py -> bobab-0.1\tm
Writing bobab-0.1\setup.cfg
creating 'dist\bobab-0.1.zip' and adding 'bobab-0.1' to it
adding 'bobab-0.1\MANIFEST.in'
adding 'bobab-0.1\PKG-INFO'
adding 'bobab-0.1\setup.cfg'
adding 'bobab-0.1\setup.py'
adding 'bobab-0.1\bobab.egg-info\dependency_links.txt'
adding 'bobab-0.1\bobab.egg-info\pbr.json'
adding 'bobab-0.1\bobab.egg-info\PKG-INFO'
adding 'bobab-0.1\bobab.egg-info\SOURCES.txt'
adding 'bobab-0.1\bobab.egg-info\top_level.txt'
adding 'bobab-0.1\mesa\a.txt'
adding 'bobab-0.1\tm\test2.py'
adding 'bobab-0.1\tm\__init__.py'
removing 'bobab-0.1' (and everything under it)

Installation log:

(sedona) dernonco@ilcompn0:/mnt/ilcompn0d1/user/dernonco/temp/bobab-0.1$ python setup.py install
running install
running bdist_egg
running egg_info
writing bobab.egg-info/PKG-INFO
writing top-level names to bobab.egg-info/top_level.txt
writing dependency_links to bobab.egg-info/dependency_links.txt
reading manifest file 'bobab.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'bobab.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_py
creating build
creating build/lib.linux-x86_64-2.7
creating build/lib.linux-x86_64-2.7/tm
copying tm/__init__.py -> build/lib.linux-x86_64-2.7/tm
copying tm/test2.py -> build/lib.linux-x86_64-2.7/tm
creating build/bdist.linux-x86_64
creating build/bdist.linux-x86_64/egg
creating build/bdist.linux-x86_64/egg/tm
copying build/lib.linux-x86_64-2.7/tm/__init__.py -> build/bdist.linux-x86_64/egg/tm
copying build/lib.linux-x86_64-2.7/tm/test2.py -> build/bdist.linux-x86_64/egg/tm
byte-compiling build/bdist.linux-x86_64/egg/tm/__init__.py to __init__.pyc
byte-compiling build/bdist.linux-x86_64/egg/tm/test2.py to test2.pyc
creating build/bdist.linux-x86_64/egg/EGG-INFO
copying bobab.egg-info/PKG-INFO -> build/bdist.linux-x86_64/egg/EGG-INFO
copying bobab.egg-info/SOURCES.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying bobab.egg-info/dependency_links.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying bobab.egg-info/pbr.json -> build/bdist.linux-x86_64/egg/EGG-INFO
copying bobab.egg-info/top_level.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
zip_safe flag not set; analyzing archive contents...
creating dist
creating 'dist/bobab-0.1-py2.7.egg' and adding 'build/bdist.linux-x86_64/egg' to it
removing 'build/bdist.linux-x86_64/egg' (and everything under it)
Processing bobab-0.1-py2.7.egg
Copying bobab-0.1-py2.7.egg to /mnt/ilcompn0d1/user/dernonco/pyenv/sedona/lib/python2.7/site-packages
Adding bobab 0.1 to easy-install.pth file

Installed /mnt/ilcompn0d1/user/dernonco/pyenv/sedona/lib/python2.7/site-packages/bobab-0.1-py2.7.egg
Processing dependencies for bobab==0.1
Finished processing dependencies for bobab==0.1
Solution

  • Several things should be adapted:

    1. MANIFEST.in will be used only when you package source dists, so python setup.py sdist will include mesa/a.txt even with your current setup. This is not the case with binary distributions, neither bdist_egg nor bdist_wheel will read the MANIFEST.in.

    2. package_data is usually used for non-python files placed inside the package, but mesa is not placed inside the tm dir, so the relative path is wrong. You can circumvent this by setting the dotted path:

      package_data={'tm': ['../mesa/a.txt']}

      However, if mesa is part of tm package, it makes sense to put it where it belongs.

    3. os.path.join('..', 'mesa', 'a.txt') will resolve the path relative to current directory, so this line can find the file only when you're in the tm directory - after installing the package, it will be hardly the case. You need to resolve the path properly, for example against the __file__ attribute:

      os.path.join(os.path.dirname(__file__), '..', 'mesa', 'a.txt')

    Additional notes:

    1. You can safely replace py_modules=['tm.test2'] with packages=['tm'] and don't need to care about updating py_modules when adding further modules to tm package.

    2. setuptools offers a module named pkg_resources that offers useful management functions for non-python files installed via package_data. For example,

      open(os.path.join('..', 'mesa', 'a.txt'), 'r').read()

      could be replaced with

      pkg_resources.resource_string('tm', '../mesa/a.txt')

      Refer to ResourceManager API section in the setuptools docs if you want to learn more.