How does pytest "know" where to import code from?
This thread is a follow-up to an unsuccessful attempt to resolve the issue on Reddit, which can be seen in its entirety here, though I will of course replicate the essential information here as well, starting with the OP:
I am following along with Python Testing with pytest and am utterly baffled as to how pytest finds a certain class in the example I am working with. All of the code is here in a .zip file:
https://pragprog.com/titles/bopytest2/python-testing-with-pytest-second-edition/
So code/ch2/test_card.py starts with from cards import Card. Class Card is defined in cards_proj/src/cards/api.py. I can confirm that removing code/cards_proj/pyproject.toml and code/pytest.init (which is just comments anyway), first severally, then jointly, did not prevent the successful run of the tests. pytest would not let me just print out sys.path in or out of any of the test functions, so I did something I am not terribly proud of instead:
from cards import Card
import sys
def test_field_access():
with open('path.txt', 'w') as f:
f.writelines(p + '\n' for p in sys.path)
c = Card("something", "brian", "todo", 123)
assert c.summary == "something"
assert c.owner == "brian"
assert c.state == "todo"
assert c.id == 123
# ...remainder omitted
The result in path.txt did nothing to clarify matters:
/home/readyready15728/programming/python/python-testing-with-pytest/code/ch2
/home/readyready15728/programming/python/python-testing-with-pytest/venv/bin
/usr/lib/python310.zip
/usr/lib/python3.10
/usr/lib/python3.10/lib-dynload
/home/readyready15728/programming/python/python-testing-with-pytest/venv/lib/python3.10/site-packages
There was only one serious attempt to help me, which did not go anywhere. Briefly:
I was asked about the contents of src/__init__.py, which did not exist, though I was able to divulge the contents of code/cards_proj/src/cards/__init__.py:
"""Top-level package for cards."""
__version__ = "1.0.0"
from .api import * # noqa
from .cli import app # noqa
I was encouraged to upload the code to GitHub. After being unsure whether I should due to copyright I figured it probably wouldn't be too bad; after all, I found the first edition code from a third party without a fuss being made about it. Because multiple files are obviously involved, I can't replicate everything easily in a Stack Overflow thread; however, here is a link to the initial commit which will not change even if I do more commits.
I ran two commands to shed some light on the directory structure. Here is the output of find -mindepth 1 -type d | sort executed in code/:
./cards_proj
./cards_proj/src
./cards_proj/src/cards
./ch1
./ch10
./ch11
./ch11/cards_proj
./ch11/cards_proj/.github
./ch11/cards_proj/.github/workflows
./ch11/cards_proj/src
./ch11/cards_proj/src/cards
./ch11/cards_proj/tests
./ch11/cards_proj/tests/api
./ch11/cards_proj/tests/cli
./ch12
./ch12/app
./ch12/app/src
./ch12/app/tests
./ch12/script
./ch12/script_funcs
./ch12/script_importable
./ch12/script_src
./ch12/script_src/src
./ch12/script_src/tests
./ch13
./ch13/cards_proj
./ch13/cards_proj/src
./ch13/cards_proj/src/cards
./ch13/cards_proj/tests
./ch13/cards_proj/tests/api
./ch13/cards_proj/tests/cli
./ch14
./ch14/random
./ch15
./ch15/just_markers
./ch15/local
./ch15/pytest_skip_slow
./ch15/pytest_skip_slow/examples
./ch15/pytest_skip_slow_final
./ch15/pytest_skip_slow_final/examples
./ch15/pytest_skip_slow_final/tests
./ch16
./ch1/.backup
./ch1/__pycache__
./ch2
./ch2/.backup
./ch2/__pycache__
./ch2/.pytest_cache
./ch2/.pytest_cache/v
./ch2/.pytest_cache/v/cache
./ch3
./ch3/a
./ch3/b
./ch3/c
./ch3/d
./ch4
./ch5
./ch6
./ch6/bad
./ch6/builtins
./ch6/combined
./ch6/multiple
./ch6/reg
./ch6/slow
./ch6/smoke
./ch6/strict
./ch6/tests
./ch7
./ch8
./ch8/alt
./ch8/dup
./ch8/dup/tests_no_init
./ch8/dup/tests_no_init/api
./ch8/dup/tests_no_init/cli
./ch8/dup/tests_with_init
./ch8/dup/tests_with_init/api
./ch8/dup/tests_with_init/cli
./ch8/project
./ch8/project/tests
./ch8/project/tests/api
./ch8/project/tests/cli
./ch9
./ch9/some_code
./exercises
./exercises/ch10
./exercises/ch11
./exercises/ch11/src
./exercises/ch12
./exercises/ch2
./exercises/ch5
./exercises/ch6
./exercises/ch8
./exercises/ch8/tests
./exercises/ch8/tests/a
./exercises/ch8/tests/b
./.pytest_cache
./.pytest_cache/v
./.pytest_cache/v/cache
I was also asked to carry out tree in the directory ch2, which I obliged specifically by running tree -a:
├── .backup
│ └── test_card.py~
├── path.txt
├── __pycache__
│ └── test_card.cpython-310-pytest-7.4.3.pyc
├── .pytest_cache
│ ├── CACHEDIR.TAG
│ ├── .gitignore
│ ├── README.md
│ └── v
│ └── cache
│ ├── nodeids
│ └── stepwise
├── test_alt_fail.py
├── test_card_fail.py
├── test_card.py
├── test_classes.py
├── test_exceptions.py
├── test_experiment.py
├── test_helper.py
└── test_structure.py
.backup/ and path.txt are the results of my involvement. It should be clear from the output of the two commands that pytest "knows" to climb to a higher level in the file system, then climb down into cards_proj/src/ to import the code. The other Reddit user suggested I wasn't just running pytest in the ch2/ subdirectory. I made certain:
(Text description to accompany image: the terminal shows the working directory as ch2/)
I also looked at some cached files but found no leads there.
It was then suggested: "Somehow I'm [sure] the root of the project is being added to your path and so you're able to import the modules that are exported at the root of the project". I then printed out the output of set -L (I'm using fish), with the sole exception of history, both for brevity and for privacy reasons:
CAML_LD_LIBRARY_PATH '/home/readyready15728/.opam/default/lib/stublibs' '/home/readyready15728/.opam/default/lib/ocaml/stublibs' '/home/readyready15728/.opam/default/lib/ocaml'
CLUTTER_BACKEND x11
CLUTTER_IM_MODULE ibus
CMD_DURATION 0
COLORTERM truecolor
COLUMNS 226
DBUS_SESSION_BUS_ADDRESS unix:path=/run/user/1000/bus
DESKTOP_SESSION xubuntu
DISPLAY :0.0
FISH_VERSION 3.3.1
GPG_AGENT_INFO /run/user/1000/gnupg/S.gpg-agent:0:1
GTK3_MODULES xapp-gtk3-module
GTK_IM_MODULE ibus
GTK_MODULES gail:atk-bridge
GTK_OVERLAY_SCROLLING 0
HOME /home/readyready15728
IFS \n\ \t
LANG en_US.UTF-8
LINES 53
LOGNAME readyready15728
LS_COLORS 'rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.webp=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:'
MANPATH '' '/home/readyready15728/.opam/default/man'
OCAML_TOPLEVEL_PATH /home/readyready15728/.opam/default/lib/toplevel
OMF_CONFIG /home/readyready15728/.config/omf
OMF_INVALID_ARG 3
OMF_MISSING_ARG 1
OMF_PATH /home/readyready15728/.local/share/omf
OMF_UNKNOWN_ERR 4
OMF_UNKNOWN_OPT 2
OPAM_SWITCH_PREFIX /home/readyready15728/.opam/default
PAM_KWALLET5_LOGIN /run/user/1000/kwallet5.socket
PANEL_GDK_CORE_DEVICE_EVENTS 0
PATH '/home/readyready15728/programming/python/python-testing-with-pytest/venv/bin' '/home/readyready15728/.opam/default/bin' '/home/readyready15728/altera/13.0sp1/quartus' '/home/readyready15728/altera/13.0sp1/nios2eds/bin' '/home/readyready15728/altera/13.0sp1/quartus/bin' '/usr/local/sbin' '/usr/local/bin' '/usr/sbin' '/usr/bin' '/sbin' '/bin' '/usr/games' '/usr/local/games' '/snap/bin'
PWD /home/readyready15728/programming/python/python-testing-with-pytest/code/ch2
QT_ACCESSIBILITY 1
QT_IM_MODULE ibus
QT_QPA_PLATFORMTHEME gtk2
SESSION_MANAGER local/stygies-viii:@/tmp/.ICE-unix/1334,unix/stygies-viii:/tmp/.ICE-unix/1334
SHELL /usr/bin/fish
SHLVL 2
SSH_AGENT_PID 1454
SSH_AUTH_SOCK /run/user/1000/keyring/ssh
TERM xterm-256color
TERM_PROGRAM tmux
TERM_PROGRAM_VERSION 3.2a
TMUX /tmp/tmux-1000/default,86921,0
TMUX_PANE '%7'
TMUX_PLUGIN_MANAGER_PATH /home/readyready15728/.tmux/plugins/
USER readyready15728
VIRTUAL_ENV /home/readyready15728/programming/python/python-testing-with-pytest/venv
VIRTUAL_ENV_PROMPT '(venv) '
VTE_VERSION 6800
WINDOWID 71303171
XAUTHORITY /home/readyready15728/.Xauthority
XDG_CONFIG_DIRS /etc/xdg/xdg-xubuntu:/etc/xdg
XDG_CURRENT_DESKTOP XFCE
XDG_DATA_DIRS /usr/share/xubuntu:/usr/share/xfce4:/home/readyready15728/.local/share/flatpak/exports/share:/var/lib/flatpak/exports/share:/usr/local/share:/usr/share
XDG_MENU_PREFIX xfce-
XDG_RUNTIME_DIR /run/user/1000
XDG_SEAT seat0
XDG_SEAT_PATH /org/freedesktop/DisplayManager/Seat0
XDG_SESSION_CLASS user
XDG_SESSION_DESKTOP XFCE
XDG_SESSION_ID 3
XDG_SESSION_PATH /org/freedesktop/DisplayManager/Session3
XDG_SESSION_TYPE x11
XDG_VTNR 1
XMODIFIERS @im=ibus
_ set
_OLD_FISH_PROMPT_OVERRIDE /home/readyready15728/programming/python/python-testing-with-pytest/venv
_OLD_VIRTUAL_PATH '/home/readyready15728/.opam/default/bin' '/home/readyready15728/altera/13.0sp1/quartus' '/home/readyready15728/altera/13.0sp1/nios2eds/bin' '/home/readyready15728/altera/13.0sp1/quartus/bin' '/usr/local/sbin' '/usr/local/bin' '/usr/sbin' '/usr/bin' '/sbin' '/bin' '/usr/games' '/usr/local/games' '/snap/bin'
__fish_active_key_bindings fish_default_key_bindings
__fish_added_user_paths
__fish_bin_dir /usr/bin
__fish_cd_direction prev
__fish_config_dir /home/readyready15728/.config/fish
__fish_config_interactive_done
__fish_data_dir /usr/share/fish
__fish_help_dir /usr/share/doc/fish
__fish_initialized 3100
__fish_last_bind_mode default
__fish_locale_vars 'LANG' 'LC_ALL' 'LC_COLLATE' 'LC_CTYPE' 'LC_MESSAGES' 'LC_MONETARY' 'LC_NUMERIC' 'LC_TIME'
__fish_ls_color_opt --color=auto
__fish_ls_command ls
__fish_sysconf_dir /etc/fish
__fish_user_data_dir /home/readyready15728/.local/share/fish
dirprev '/media/readyready15728/LIBERET NOS DEUS MACHINARIUS AB IGNORANTIA/Pictures' '/home/readyready15728' '/home/readyready15728/src/vim' '/home/readyready15728/programming/python/python-testing-with-pytest' '/home/readyready15728/programming/python/python-testing-with-pytest/code'
fish_bind_mode default
fish_color_autosuggestion '555' 'brblack'
fish_color_cancel -r
fish_color_command 005fd7
fish_color_comment 990000
fish_color_cwd green
fish_color_cwd_root red
fish_color_end 009900
fish_color_error ff0000
fish_color_escape 00a6b2
fish_color_history_current --bold
fish_color_host normal
fish_color_host_remote yellow
fish_color_match --background=brblue
fish_color_normal normal
fish_color_operator 00a6b2
fish_color_param 00afff
fish_color_quote 999900
fish_color_redirection 00afff
fish_color_search_match 'bryellow' '--background=brblack'
fish_color_selection 'white' '--bold' '--background=brblack'
fish_color_status red
fish_color_user brgreen
fish_color_valid_path --underline
fish_complete_path '/home/readyready15728/.config/fish/completions' '/home/readyready15728/.local/share/omf/pkg/omf/completions' '/etc/fish/completions' '/usr/share/xubuntu/fish/vendor_completions.d' '/usr/share/xfce4/fish/vendor_completions.d' '/home/readyready15728/.local/share/flatpak/exports/share/fish/vendor_completions.d' '/var/lib/flatpak/exports/share/fish/vendor_completions.d' '/usr/local/share/fish/vendor_completions.d' '/usr/share/fish/vendor_completions.d' '/usr/share/fish/completions' '/home/readyready15728/.local/share/fish/generated_completions'
fish_function_path '/home/readyready15728/.config/fish/functions' '/home/readyready15728/.local/share/omf/pkg/omf/functions/compat' '/home/readyready15728/.local/share/omf/pkg/omf/functions/core' '/home/readyready15728/.local/share/omf/pkg/omf/functions/index' '/home/readyready15728/.local/share/omf/pkg/omf/functions/packages' '/home/readyready15728/.local/share/omf/pkg/omf/functions/themes' '/home/readyready15728/.local/share/omf/pkg/omf/functions/bundle' '/home/readyready15728/.local/share/omf/pkg/omf/functions/util' '/home/readyready15728/.local/share/omf/pkg/omf/functions/repo' '/home/readyready15728/.local/share/omf/pkg/omf/functions/cli' '/home/readyready15728/.local/share/omf/pkg/fish-spec/functions' '/home/readyready15728/.local/share/omf/pkg/omf/functions' '/home/readyready15728/.local/share/omf/lib' '/home/readyready15728/.local/share/omf/lib/git' '/home/readyready15728/.local/share/omf/themes/bobthefish' '/home/readyready15728/.local/share/omf/themes/bobthefish/functions' '/etc/fish/functions' '/usr/share/xubuntu/fish/vendor_functions.d' '/usr/share/xfce4/fish/vendor_functions.d' '/home/readyready15728/.local/share/flatpak/exports/share/fish/vendor_functions.d' '/var/lib/flatpak/exports/share/fish/vendor_functions.d' '/usr/local/share/fish/vendor_functions.d' '/usr/share/fish/vendor_functions.d' '/usr/share/fish/functions'
fish_greeting Welcome\ to\ fish,\ the\ friendly\ interactive\ shell\nType\ `help`\ for\ instructions\ on\ how\ to\ use\ fish
fish_handle_reflow 0
fish_key_bindings fish_default_key_bindings
fish_kill_signal 0
fish_killring 'scrapers/' 'misc/docker-deep-dive-2023/' 's2' 'vim' 'git ' 'clean' 'pushd ~/'
fish_pager_color_completion
fish_pager_color_description 'B3A06D' 'yellow'
fish_pager_color_prefix 'white' '--bold' '--underline'
fish_pager_color_progress 'brwhite' '--background=cyan'
fish_pid 87527
fish_user_paths '/home/readyready15728/altera/13.0sp1/quartus' '/home/readyready15728/altera/13.0sp1/nios2eds/bin' '/home/readyready15728/altera/13.0sp1/quartus/bin'
hostname stygies-viii
last_pid 1358117
omf_init_path /home/readyready15728/.local/share/omf/pkg/omf
pipestatus 2
status 2
status_generation 32
umask 0002
version 3.3.1
You'll notice I made sure to reactivate my venv to ensure that the conditions are the same but, even so, PYTHONPATH is not set and nothing else I saw looks like a lead. Like I said, how does pytest "know" where to import the code being tested from?
On page 12, I did pip install ./cards_proj in the appropriate location and had forgotten. The sample Cards application truly was in the venv path.