Search code examples
pythonpymongomongoenginebson

mognoengine and bson package not work together

I have one project which is depend on mongoengine and I am using one library which was using bson library.

It structure like below.

bsNotify/
├── setup.cfg
├── setup.py
└── src
    └── bsnotify
        ├── __init__.py
        └── resources.py

setup.py

$ cat bsNotify/setup.py
"""Base module setup."""
from setuptools import setup

setup(
    setup_requires=['pbr'],
    pbr=True
)

setup.cfg

$ cat bsNotify/setup.cfg
[metadata]
name = bsNotify
classifiers =
    License :: N/A :: N/A
    Programming Language :: Python :: 3.7

[options]
zip_safe = False
include_package_data = True
python_requires = >= 3.7
install_requires =
    mongoengine
    bson
package_dir=
    =src
packages=find:

[options.packages.find]
where=src

[tool:wheel]
universal = 1

[flake8]
exclude =
    venv,
    .tox,
    .git,
    __pycache__,
    *.pyc,
    *.egg-info,
    .cache,
    .eggs,
max-line-length = 80

[tox]
envlist = py37,unittest,lint

[testenv]
basepython=python3.7
deps =
    ipython
    pylint
    pytest
    pytest-cov
    pytest-xdist
    flake8
    flake8-docstrings

[testenv:unittest]
commands=
    pytest -v -s -n auto -l --cov=bsnotify --cov-report term-missing --cov-report xml --no-cov-on-fail tests/unit

[testenv:lint]
commands=
    flake8 src/bsnotify
    pylint src/bsnotify

src/bsnotify/__init__.py

$ cat bsNotify/src/bsnotify/__init__.py

src/bsnotify/resources.py

$ cat bsNotify/src/bsnotify/resources.py

Files in src are empty.

When I run the tox command, it create a virtualenv for testing.

$ tox -v -e unittest --notest
using tox.ini: /Users/myuser/bsNotify/setup.cfg (pid 30954)
using tox-3.14.6 from /Users/myuser/.pyenv/versions/3.7.4/lib/python3.7/site-packages/tox/__init__.py (pid 30954)
GLOB sdist-make: /Users/myuser/bsNotify/setup.py
[30956] /Users/myuser/bsNotify$ /Users/myuser/.pyenv/versions/3.7.4/bin/python3.7 setup.py sdist --formats=zip --dist-dir /Users/myuser/bsNotify/.tox/dist >.tox/log/GLOB-0.log
package .tmp/package/1/bsNotify-0.1.2.dev2.zip links to dist/bsNotify-0.1.2.dev2.zip (/Users/myuser/bsNotify/.tox)
unittest cannot reuse: no previous config /Users/myuser/bsNotify/.tox/unittest/.tox-config1
unittest create: /Users/myuser/bsNotify/.tox/unittest
[30992] /Users/myuser/bsNotify/.tox$ /Users/myuser/.pyenv/versions/3.7.4/bin/python3.7 -m virtualenv --no-download --python /Users/myuser/.pyenv/versions/3.7.4/bin/python3.7 unittest >unittest/log/unittest-0.log
unittest installdeps: ipython, pylint, pytest, pytest-cov, pytest-xdist, flake8, flake8-docstrings
[30993] /Users/myuser/bsNotify$ /Users/myuser/bsNotify/.tox/unittest/bin/python -m pip install ipython pylint pytest pytest-cov pytest-xdist flake8 flake8-docstrings >.tox/unittest/log/unittest-1.log
unittest inst: /Users/myuser/bsNotify/.tox/.tmp/package/1/bsNotify-0.1.2.dev2.zip
write config to /Users/myuser/bsNotify/.tox/unittest/.tox-config1 as '7186e9f46c94b6d9f7dde810ce83f8fe46740d9d42f2863cfd063c4c6f4e4a88 /Users/myuser/.pyenv/versions/3.7.4/bin/python3.7\n3.14.6 0 0 0\n00000000000000000000000000000000 ipython\n00000000000000000000000000000000 pylint\n00000000000000000000000000000000 pytest\n00000000000000000000000000000000 pytest-cov\n00000000000000000000000000000000 pytest-xdist\n00000000000000000000000000000000 flake8\n00000000000000000000000000000000 flake8-docstrings'
[31000] /Users/myuser/bsNotify$ /Users/myuser/bsNotify/.tox/unittest/bin/python -m pip install --exists-action w .tox/.tmp/package/1/bsNotify-0.1.2.dev2.zip >.tox/unittest/log/unittest-2.log
[31030] /Users/myuser/bsNotify$ /Users/myuser/bsNotify/.tox/unittest/bin/python -m pip freeze >.tox/unittest/log/unittest-3.log
unittest installed: apipkg==1.5,appnope==0.1.0,astroid==2.4.2,attrs==19.3.0,backcall==0.1.0,bsNotify==0.1.2.dev2,bson==0.5.10,coverage==5.1,decorator==4.4.2,execnet==1.7.1,flake8==3.8.3,flake8-docstrings==1.5.0,importlib-metadata==1.6.1,ipython==7.15.0,ipython-genutils==0.2.0,isort==4.3.21,jedi==0.17.0,lazy-object-proxy==1.4.3,mccabe==0.6.1,mongoengine==0.20.0,more-itertools==8.3.0,packaging==20.4,parso==0.7.0,pexpect==4.8.0,pickleshare==0.7.5,pluggy==0.13.1,prompt-toolkit==3.0.5,ptyprocess==0.6.0,py==1.8.1,pycodestyle==2.6.0,pydocstyle==5.0.2,pyflakes==2.2.0,Pygments==2.6.1,pylint==2.5.3,pymongo==3.10.1,pyparsing==2.4.7,pytest==5.4.3,pytest-cov==2.9.0,pytest-forked==1.1.3,pytest-xdist==1.32.0,python-dateutil==2.8.1,six==1.15.0,snowballstemmer==2.0.0,toml==0.10.1,traitlets==4.3.3,typed-ast==1.4.1,wcwidth==0.2.4,wrapt==1.12.1,zipp==3.1.0
_______________________________________________________ summary _______________________________________________________
  unittest: skipped tests
  congratulations :)

Then I try to access mongoengine in newly created virtualenv, it gives error.

$ .tox/unittest/bin/python -c "from mongoengine import connection"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/myuser/bsNotify/.tox/unittest/lib/python3.7/site-packages/mongoengine/__init__.py", line 2, in <module>
    from mongoengine import connection
  File "/Users/myuser/bsNotify/.tox/unittest/lib/python3.7/site-packages/mongoengine/connection.py", line 1, in <module>
    from pymongo import MongoClient, ReadPreference, uri_parser
  File "/Users/myuser/bsNotify/.tox/unittest/lib/python3.7/site-packages/pymongo/__init__.py", line 77, in <module>
    from pymongo.collection import ReturnDocument
  File "/Users/myuser/bsNotify/.tox/unittest/lib/python3.7/site-packages/pymongo/collection.py", line 20, in <module>
    from bson.code import Code
  File "/Users/myuser/bsNotify/.tox/unittest/lib/python3.7/site-packages/bson/code.py", line 18, in <module>
    from bson.py3compat import abc, string_type, PY3, text_type
ImportError: cannot import name 'abc' from 'bson.py3compat' (/Users/myuser/bsNotify/.tox/unittest/lib/python3.7/site-packages/bson/py3compat.py)

When I was using mongoengine alone with bson, it works fine.

$ python -m venv .venv
$ .venv/bin/pip install mongoengine
Collecting mongoengine
  Using cached https://files.pythonhosted.org/packages/7d/bd/9a7239b0032157f948c69febdf71dd82cb54fcd2499077300496a3f076c9/mongoengine-0.20.0-py3-none-any.whl
Collecting pymongo<4.0,>=3.4 (from mongoengine)
  Using cached https://files.pythonhosted.org/packages/23/cd/27fbc08f0bd835b4735504a758756e979b42c5bc9ebaac5ed3c2cbffd83f/pymongo-3.10.1-cp37-cp37m-macosx_10_9_x86_64.whl
Installing collected packages: pymongo, mongoengine
Successfully installed mongoengine-0.20.0 pymongo-3.10.1
$ .venv/bin/python -c "from mongoengine import connection"
$ .venv/bin/pip install bson
Collecting bson
  Using cached https://files.pythonhosted.org/packages/4d/53/7c534a38850f2252275d7f949aed2219095e90df1e2d180a9c8ed139e499/bson-0.5.10.tar.gz
Collecting python-dateutil>=2.4.0 (from bson)
  Using cached https://files.pythonhosted.org/packages/d4/70/d60450c3dd48ef87586924207ae8907090de0b306af2bce5d134d78615cb/python_dateutil-2.8.1-py2.py3-none-any.whl
Collecting six>=1.9.0 (from bson)
  Using cached https://files.pythonhosted.org/packages/ee/ff/48bde5c0f013094d729fe4b0316ba2a24774b3ff1c52d924a8a4cb04078a/six-1.15.0-py2.py3-none-any.whl
Installing collected packages: six, python-dateutil, bson
  Running setup.py install for bson ... done
Successfully installed bson-0.5.10 python-dateutil-2.8.1 six-1.15.0
You are using pip version 19.0.3, however version 20.2b1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
$ .venv/bin/python -c "from mongoengine import connection"

It check the pip freeze between both the virtualenv. Only difference I found is in .tox env has bsNotify==0.1.2.dev2 installed which is not exists in .venv env.

What make bson change the file py3compat.py?

Solution

  • change this 

    install_requires =
    mongoengine
    bson

    to this

    install_requires =
    mongoengine
    pymongo

    or remove altogether (mongoengine already requires pymongo

    install_requires =
    mongoengine

    requirements=bson is the same things as

    pip install bson

    which installs this 3rd party package which does not include all the goodies found in MongoDB's package

    https://pypi.org/project/bson/

    pymongo (official MongoDB driver) contains a bson package. I'll drop both a pymongo folder and a bson folder into your site-packages.

    bson (third party bson implementation) also wants to drop a bson folder into your site packages.

    Installing pymongo and installing bson will conflict folders inside your site-packages. PyMongo only knows how to use it's own implementation, which is why you're seeing pymongo looking for py3compat which is not part of the 3rd party bson package.

    Based on your install order, it would have installed mongo engine (and pymongo as a dependency) then installed bson (overwriting MongoDB's bson implementation packaged with pymongo). Pymongo called py3compat expecting it to be there, but 3td party did not implement that (as well as several other things).

    pip install pymongo

    https://pypi.org/project/pymongo/

    As noted on the pymongo pypi page

    Do not install the “bson” package from pypi. PyMongo comes with its own bson package; doing “easy_install bson” installs a third-party package that is incompatible with PyMongo.