Search code examples
pythonpython-typingmypy

How to share stubs for type hinting transitively?

I am currently using a Python library (x) that does not come with type hints. I've opted to add stub files for the x methods that I use in my project like this:

MyProject_1/
  stubs/
    x/ 
      __init__.pyi  # Contains type hints for any attributes of `x`.

  package_1/
    package_1_main.py  # Imports `x`.
    __init__.py   
    py.typed  # Indicates package 1 type hints are specified inline.

  setup.py  # Specifies `x` as a dependency under `install_requires` and `py.typed` under `package_data`.

Then in the MyProject_1 directory, I run MYPYPATH=stubs mypy ..

If package_1_main.py imports x, then mypy successfully type hints the x imports based on stubs/x. All is good here.

Now I've made another project MyProject_2 that depends on MyProject_1 My_Project_2 successfully uses the inline type hints specified in package_1_main.py because of the py.typed under package_1.

The problem is that MyProject_2 does not pick up on stubs/x. Since x is a transitive dependency of MyProject_2 through MyProject_1, I have the ability to use x in MyProject_2. However, this is without the type hints specified under stubs/x.

There are 3 ways I can imagine being able to use these stubs (and thus access the type hints of x):

  1. Add the stubs directory under MyProject_1 to the MYPYPATH when running mypy . in MyProject_2. Downside of this is it depends on how MyProject_1 was installed through pip, and stubs will have to be made as part of the distributed package data in setup.py.

  2. Copy the stubs directory under MyProject_1 to MyProject_2, keeping them manually in sync. Downside of this is overhead in maintenance.

  3. Make a third project called MyProject_Stubs, dedicated to keeping the stub files for any project depending on x, either transitively or directly. Downside of this is a dedicated project just for stubs.

Are there any other approaches I could consider? And for the slightly opinionated part: which one of these approaches seems to have the greatest net advantage?

Solution

  • There is, AFAIK, no way of sharing stubs transitively.

    Instead, you'll have to focus on finding ways of making your stubs available to both projects/making it a dependency of both.

    There are roughly six different ways to do so, which I'll list out in roughly increasing order of effort.

    1. Store your stubs somewhere on your local computer and have both your projects refer to it via the MYPYPATH environment variable or config file option.

      This is relatively straightforward to do, but the downside is that your projects will only work on your local computer. If you plan for others to use this project, they'll need to obtain your stubs and also set up their MYPYPATH.

    2. Copy your stubs to your second project. As you said, the main downside here you now need to keep your stubs in sync manually.

    3. Create a separate git repo for your stubs and add them as a git submodule into your two projects.

      This solves the code duplication problem option 2 has. The only real downside is you need to remember to bump your submodule pin in both projects every time you update your stub repo.

    4. Contribute type hints for your project to typeshed a repository of type hints for Python and either wait for the next mypy release or run mypy with the --custom-typeshed-dir flag/config option.

      Note that Typeshed accepts partial stubs and is hoping on switching to a more iterative and rapid release cadence some time in the future.

      This option would be the best if you want to open-source your work broadly, but don't necessarily want the burden of maintaining your stubs after you open-source them.

      The main downside is you would be unable to unblock yourself immediately.

    5. Create an X-stubs package with a py.typed file. This is kind of like your option 3, except your new project is explicitly for just project X's stubs and is open-sourced.

      You can then add your X-stubs package to your pip dependencies just like any pip package, and perhaps even eventually look into more broadly open-sourcing and distributing them.

      This is basically the same as your option 3, except that the name of your package is different and less specific to your project.

    6. Submit a pull request to project X adding a py.typed file along with some type hints and convince them to land it.

      This is easier said than done, of course -- not all projects are willing to accept type hints, of course, since it could lead to an increased maintenance burden for them.

    If you care just about unblocking yourself, I'd pick options 1-3. The other options are better fits if you also want to try open-sourcing your work and helping improve the Python typing ecosystem.

    I would avoid your option 1 -- it seems to involve roughly the same amount of work to set up as my options 3 and 5 while also requiring more work inside project 2 to use.