Search code examples
python-3.xmacostensorflowapple-m1pipenv

How to use tensorflow-macos and tensorflow in the same Pipfile and keep the possibility to lock on all OSs and architectures?

The M1 architecture made it hard to use some packages but all of them have dealt with this by now except TensorFlow.

To be able to use tensorflow on an M1 (or M2) chip you need to use tensorflow-macos instead of tensorflow.

In my case I have a workflow where I use pipenv for dependency management. In this way no matter the system everything works. However for tensorflow this is a problem as it cannot be installed on M1 but tensorflow-macOS not on other architectures.

So when I lock my Pipfile and try to run a CI job it crashes because I use the mac version and this is now locked but does not work in CI.

Now this would not be that big of a problem if some packages requiring tensorflow would not also be a dependency for others meaning tensorflow in no longer in the Pipfile for those.

Anyone has any idea how to handle this?

Ideally you would want to make some sort of a alias on the m1 to use tensorflow-macos if tensorflow is used somewhere. But I don't see how this is possible.

Docker is a solution in some cases (but does not work terribly well on m1 with tensorflow) but again has problems if if the repo interacts much with other local processes like kubernetes, environment variables etc.

say you have the following Pipfile

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
tensorflow = "==2.7.*"


[requires]
python_version = "3.9"

Locking/installing this on M1 will not work so you have to change it to tensorflow-macos. However this will not work on other machines...

Solution

  • The following setup works for me. I keep two different Pipfiles

    Pipfile_x86 has

    ...
[packages]
tensorflow = "==2.11.0"
...

    Pipfile_Mac_ARM has

    ...
[packages]
tensorflow-macos = "==2.11.0"
tensorflow-metal = "==0.7.0"
...

    and likewise two Pipfile_x86.lock and Pipfile_Mac_ARM.lock.

    Then I create symbolic links to the x86 versions:

    ln -s Pipfile_x86 Pipfile
ln -s Pipfile_x86.lock Pipfile.lock

    and commit this to the repository, because most people (and CI) use x86.

    On a Mac, I will symlink to the ARM versions and ignore changes in Git so that the different symbolic links are not staged:

    git update-index --skip-worktree Pipfile Pipfile.lock

    Just remember to add additional packages to both Pipfiles, and to let the .lock files be generated upon changes through pipenv update for both versions.