TL;DR: How can I set up my GitLab test pipeline so that the tests also run locally on VS Code?
I'm very new to GitLab pipelines, so please forgive me if the question is amateurish. I have a GitLab repo set up online, and I'm using VS Code to develop locally. I've created a new pipeline, I want to make sure all my unit tests (written with PyTest) run anytime I make a commit.
The issue is, that even though I use the same setup.py file for both places (obviously), I can't get both VS Code testing and the GitLab pipeline test to work at the same time. The issue is, I'm doing an import for my tests, and if I import like
...
from external_workforce import misc_tools
# I want to test functions in this misc_tools module
...
Then it works on GitLab, but not on VS Code, as VS Code gives an error when I'm doing test discovery, namely: ModuleNotFoundError: No module named 'external_workforce'. However, this works on GitLab. But if I import (in my test_tools.py file, see location below) like this:
...
from hr_datapool.external_workforce import misc_tools
...
It works in VS Code, but now GitLab is doing a crazy on me saying ModuleNotFoundError: No module named 'hr_datapool'.
I think the relevant info might be the following, please ask for more if more info is needed!
My file structure is:
.
|__ requirements.txt
setup.py
hr_datapool
|__ external_workforce
|__ __init__.py
misc_tools.py
tests
|__ test_tools.py
|__ other_module
...
In my pipeline editor (the .gitlab-ci.yml file) I have:
image: python:3.9.7
variables:
PIP_CACHE_DIR: "$CI_PROJECT_DIR/.cache/pip"
cache:
paths:
- .cache/pip
- venv/
before_script:
- python --version # For debugging
- pip install virtualenv
- virtualenv venv
- source venv/bin/activate
- pip install -r requirements.txt
test:
script:
- pytest --pyargs hr_datapool #- python setup.py test
run:
script:
- python setup.py bdist_wheel
artifacts:
paths:
- dist/*.whl
And finally, my setup.py is:
import re
from unittest import removeResult
from setuptools import setup, find_packages
with open('requirements.txt') as f:
requirements = f.read().splitlines()
for req in ['wheel', 'bar']:
requirements.append(req)
setup(
name='hr-datapool',
version='0.1',
...
packages=find_packages(),
install_requires=requirements,
)
Basically, the question is: How can I set up my GitLab test pipeline so that the tests also run locally on VS Code? Thank you!
UPDATE:
Adding the full trace coming from VS Code:
> conda run -n base --no-capture-output --live-stream python ~/.vscode/extensions/ms-python.python-2022.2.1924087327/pythonFiles/get_output_via_markers.py ~/.vscode/extensions/ms-python.python-2022.2.1924087327/pythonFiles/testing_tools/run_adapter.py discover pytest -- --rootdir "." -s --cache-clear hr_datapool
cwd: .
[ERROR 2022-2-23 9:2:4.500]: Error discovering pytest tests:
[r [Error]: ============================= test session starts ==============================
platform darwin -- Python 3.9.7, pytest-6.2.4, py-1.10.0, pluggy-0.13.1
rootdir: /Users/myuser/Documents/myfolder
plugins: anyio-2.2.0
collected 0 items / 1 error
==================================== ERRORS ====================================
_____ ERROR collecting hr_datapool/external_workforce/tests/test_tools.py ______
ImportError while importing test module '/Users/myuser/Documents/myfolder/hr_datapool/external_workforce/tests/test_tools.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
../../opt/anaconda3/lib/python3.9/importlib/__init__.py:127: in import_module
return _bootstrap._gcd_import(name[level:], package, level)
hr_datapool/external_workforce/tests/test_tools.py:2: in <module>
from external_workforce import misc_tools
E ModuleNotFoundError: No module named 'external_workforce'
=========================== short test summary info ============================
ERROR hr_datapool/external_workforce/tests/test_tools.py
!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
===================== no tests collected, 1 error in 0.08s =====================
Traceback (most recent call last):
File "/Users/myuser/.vscode/extensions/ms-python.python-2022.2.1924087327/pythonFiles/get_output_via_markers.py", line 26, in <module>
runpy.run_path(module, run_name="__main__")
File "/Users/myuser/opt/anaconda3/lib/python3.9/runpy.py", line 268, in run_path
return _run_module_code(code, init_globals, run_name,
File "/Users/myuser/opt/anaconda3/lib/python3.9/runpy.py", line 97, in _run_module_code
_run_code(code, mod_globals, init_globals,
File "/Users/myuser/opt/anaconda3/lib/python3.9/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/Users/myuser/.vscode/extensions/ms-python.python-2022.2.1924087327/pythonFiles/testing_tools/run_adapter.py", line 22, in <module>
main(tool, cmd, subargs, toolargs)
File "/Users/myuser/.vscode/extensions/ms-python.python-2022.2.1924087327/pythonFiles/testing_tools/adapter/__main__.py", line 100, in main
parents, result = run(toolargs, **subargs)
File "/Users/myuser/.vscode/extensions/ms-python.python-2022.2.1924087327/pythonFiles/testing_tools/adapter/pytest/_discovery.py", line 44, in discover
raise Exception("pytest discovery failed (exit code {})".format(ec))
Exception: pytest discovery failed (exit code 2)
ERROR conda.cli.main_run:execute(33): Subprocess for 'conda run ['python', '/Users/myuser/.vscode/extensions/ms-python.python-2022.2.1924087327/pythonFiles/get_output_via_markers.py', '/Users/A111086670/.vscode/extensions/ms-python.python-2022.2.1924087327/pythonFiles/testing_tools/run_adapter.py', 'discover', 'pytest', '--', '--rootdir', '/Users/myuser/Documents/myfolder', '-s', '--cache-clear', 'hr_datapool']' command failed. (See above for error)
at ChildProcess.<anonymous> (/Users/myuser/.vscode/extensions/ms-python.python-2022.2.1924087327/out/client/extension.js:32:39235)
at Object.onceWrapper (events.js:422:26)
at ChildProcess.emit (events.js:315:20)
at maybeClose (internal/child_process.js:1048:16)
at Process.ChildProcess._handle.onexit (internal/child_process.js:288:5)]
The PYTHONPATH caused the problem.
As external_workforce parent folder path -> the path of hr_datapool in the PYTHONPATH when you are using GitLab. While hr_datapool parent folder path in the PYTHONPATH when you are using VSCode.
Are you running the test in the terminal on the VSCode? And have you added this in the settings.json file?
"terminal.integrated.env.windows": {
"PYTHONPATH": "${workspaceFolder};"
},
Then you can execute pytest in the terminal on the VSCode. But you have not configured this in GitLab instead of adding hr-datapool( - pytest --pyargs hr_datapool or setup( name='hr-datapool',), so you will get the error message.