Search code examples
pythonnumpypipnumpy-2.x

Python package dependecy upgrade from requirements.txt

I have a Python application with ~70 packages in requirements.txt file.

It was running fine, but suddenly snowfalke-connector-python==2.7.3 and schemachange==3.4.2 started installing numpy==2.0.0 while they were installing numpy==1.26.4 a day before, but from today they are installing numpy==2.0.0. This is resulting in a compatibility issue with Pandas as my app is throwing following errors.

    from pandas._libs.interval import Interval
  File "pandas/_libs/interval.pyx", line 1, in init pandas._libs.interval
ValueError: numpy.dtype size changed, may indicate binary incompatibility. Expected 96 from C header, got 88 from PyObject

  • If I add pandas== 2.2.2 or other latest version then other compatibility issues arise.

  • For now I am running my app by restricting numpy to 1.26.4

  • I want to understand why this might have happened, and is it normal in python?

  • What is the best way to deal with this type of situation.

Solution

  • Each package can specify its dependencies. What happened in your case is the following, chronologically:

    1. snowfalke-connector-python==2.7.3, and schemachange==3.4.2 are released, each specifying a minimum numpy version, so something like numpy>1.24, the latest numpy is 1.26.4 which is also compatible
    2. numpy releases 2.0, making breaking changes in their API (hence probably the major version increase)
    3. You try to install, pip will by default always try to pick the alatest compatible version of dependencies, so it picks up the latest numpy which is incompatible however, as the requirements formulated ins tep 1 are actually not compatible with the breaking changes in numpy which the devs of snowfalke-connector-python and schemachange were not aware of

    Is this normal? Depends on your definition of "normal". You could argue that someone who writes a package should make sure that all requirements are guaranteed to work, so also include an upper limit on the version, but this can be regarded as opinion based. So it is normal in the sense that it can happen and will happen

    What is the best way to deal with this type of situation report it to the relevant maintainers of the packages as they should fix it in their requirements. If you cannot wait for a fix, you can do what you did and try to find a mix of package versions that works yourself