VSCode Python autocomplete for generated code in separate directory

Question

I use pants to manage a Python project that uses protocol buffers. Pants places the generated _pb2.py and _pb2.pyi files under a separate dist/codegen tree. Is it possible to get VS Code autocomplete to work when using the _pb2 modules?

The file tree looks like this:

.
|-- dist/
|   `-- codegen/
|       `-- src/
|           `-- project/
|               |-- data_pb2.py
|               `-- data_pb2.pyi
`-- src/
    `-- project/
        |-- __init__.py
        |-- code.py
        `-- data.proto

And in code.py I have import statements like this:

from project import data_pb2

I've tried setting python.analysis.extraPaths to ["dist/codegen/src"] in settings.json. This makes pylance stop complaining that data_pb2 is missing. But autocomplete still does not work, and pylance has no type information for members of data_pb2.

Answer 1

You can use Python implicit namespace packages (PEP 420) to make this work. Namespace packages are allowed to have modules within the same package reside in different directories. Which allows pylance and other tools to work correctly when code is split between src and dist/codegen/src.

To use implicit namespace packages, you just need to remove src/package/__init__.py, and leave "python.analysis.extraPaths" set to ["dist/codegen/src"].

See also the GitHub issue microsoft/pylance-release#2855, which describes using implicit namespace packages to make pylance work correctly in a similar situation.