I have a flask-restx folder with the following structure
.
├── app
│ ├── extensions.py
│ ├── __init__.py
│ └── pv_dimensioning
│ ├── controller.py
│ ├── __init__.py
│ ├── models
│ │ ├── dto.py
│ │ ├── __init__.py
│ │ ├── input_output_model.py
│ │ └── vendor_models.py
│ ├── services
│ │ ├── calculator.py
│ │ ├── database.py
│ │ ├── db_crud.py
│ │ ├── db_input_output.py
│ │ ├── __init__.py
│ │ └── processor.py
│ └── utils
│ ├── decode_verify_jwt.py
│ ├── decorator.py
│ └── __init__.py
├── config.py
├── main.py
├── package.json
├── package-lock.json
├── Pipfile
├── Pipfile.lock
├── README.md
├── serverless.yml
└── tests
└── test_processor.py
The app is connected to a database, that's why there are many scripts in the app that require VPN connection.
I am writing tests using pytest which I will then run on the gitlab CI/CD to get a proper coverage. I would like to avoid or omit scripts that can only run when VPN is connected and write tests only for scripts that don't require VPN. (I have tests for the scripts that require VPN, but I just don't want to run them in the CI/CD pipeline)
The only script that doesn't require VPN is the processor.py and the test for that is in test_processor.py.
The scripts that I would like to avoid are in the .coveragerc:
[run]
omit =
*/site-packages/*
*/distutils/*
tests/*
/usr/*
app/__init__.py
app/extensions.py
app/pv_dimensioning/models/*
app/pv_dimensioning/utils/*
app/pv_dimensioning/controller.py
app/pv_dimensioning/services/calculator.py
app/pv_dimensioning/services/database.py
app/pv_dimensioning/services/db_crud.py
app/pv_dimensioning/services/db_input_output.py
[html]
directory = htmlcov
the coverage part of .gitlab-ci.yml
stages:
- coverage
coverage:
image: python:3.7
stage: coverage
artifacts:
paths:
- htmlcov/
before_script:
- apt-get -y update
- apt-get install curl
- pip install pipenv
- pipenv install --dev
script:
- pipenv run python -m coverage run -m pytest
- pipenv run python -m coverage report -m
- pipenv run python -m coverage html
after_script:
- pipenv run bash <(curl -s https://codecov.io/bash)
When I run the test in the pipeline, I get the following error:
$ pipenv run python -m coverage run -m pytest
============================= test session starts ==============================
platform linux -- Python 3.7.10, pytest-6.2.1, py-1.10.0, pluggy-0.13.1
rootdir: /builds/EC/tool/dt-service
plugins: cov-2.11.0
collected 0 items / 1 error
==================================== ERRORS ====================================
___________________ ERROR collecting tests/test_processor.py ___________________
/usr/local/lib/python3.7/urllib/request.py:1350: in do_open
encode_chunked=req.has_header('Transfer-encoding'))
/usr/local/lib/python3.7/http/client.py:1277: in request
self._send_request(method, url, body, headers, encode_chunked)
/usr/local/lib/python3.7/http/client.py:1323: in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
/usr/local/lib/python3.7/http/client.py:1272: in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
/usr/local/lib/python3.7/http/client.py:1032: in _send_output
self.send(msg)
/usr/local/lib/python3.7/http/client.py:972: in send
self.connect()
/usr/local/lib/python3.7/http/client.py:1439: in connect
super().connect()
/usr/local/lib/python3.7/http/client.py:944: in connect
(self.host,self.port), self.timeout, self.source_address)
/usr/local/lib/python3.7/socket.py:707: in create_connection
for res in getaddrinfo(host, port, 0, SOCK_STREAM):
/usr/local/lib/python3.7/socket.py:752: in getaddrinfo
for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
E socket.gaierror: [Errno -2] Name or service not known
During handling of the above exception, another exception occurred:
tests/test_processor.py:2: in <module>
from app.pv_dimensioning.services.processor import PreProcessings
app/pv_dimensioning/__init__.py:4: in <module>
from .controller import admin_crud_ns as admin_crud_namespace
app/pv_dimensioning/controller.py:5: in <module>
from .services.calculator import DimensionCalculator
app/pv_dimensioning/services/calculator.py:3: in <module>
from .database import DatabaseService
app/pv_dimensioning/services/database.py:6: in <module>
from ..utils.decode_verify_jwt import verifier
app/pv_dimensioning/utils/decode_verify_jwt.py:12: in <module>
with urllib.request.urlopen(keys_url) as f:
/usr/local/lib/python3.7/urllib/request.py:222: in urlopen
return opener.open(url, data, timeout)
/usr/local/lib/python3.7/urllib/request.py:525: in open
response = self._open(req, data)
/usr/local/lib/python3.7/urllib/request.py:543: in _open
'_open', req)
/usr/local/lib/python3.7/urllib/request.py:503: in _call_chain
result = func(*args)
/usr/local/lib/python3.7/urllib/request.py:1393: in https_open
context=self._context, check_hostname=self._check_hostname)
/usr/local/lib/python3.7/urllib/request.py:1352: in do_open
raise URLError(err)
E urllib.error.URLError: <urlopen error [Errno -2] Name or service not known>
=========================== short test summary info ============================
ERROR tests/test_processor.py - urllib.error.URLError: <urlopen error [Errno ...
!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
=============================== 1 error in 1.62s ===============================
In the trace, it can be seen that the scripts that I am trying to ignore aren't being ignored. What is the mistake I am doing?
As I understand it, coverage is about reporting how much of your codebase is tested, not which tests to run. What you're doing is excluding things from a report, not stopping the data for the report being created.
What you should do is skip tests if you know they're going to fail (due to external configuration). Fortunately pytest provides for this with the skipif decorator.
I would create a function in tests/conftest.py which skips tests if the VPN is active. Something like:
import socket
import pytest
def _requires_vpn():
has_vpn = False
try:
socket.gethostbyname("<some address only accessible in the VPN>")
has_vpn = True
except socket.error:
pass
return pytest.mark.skipif(not has_vpn, reason="access to the vpn is required")
requires_vpn = _requires_vpn() # this is like the minversion example on the link
Then you can ignore entire test files adding this at the top:
pytestmark = requires_vpn
And you can skip specific tests by decorating with this.
@requires_vpn
def test_this_will_be_skipped():
pass