I am not a huge DevOps expert, but I am trying to configure a Jenkins pipeline to run my Python Flask application's Unittests, that rely on MongoDB.
Jenkins is running inside Docker.
I tried the following, but it keeps hanging without termination, I cannot understand what it's wrong:
pipeline {
agent {
docker {
image 'python:3.9'
args '--user 0:0'
}
}
stages {
stage('build') {
steps {
sh 'pip install -r requirements.txt'
sh 'pip install -e .'
}
}
stage('test') {
steps {
script {
node {
docker.image('mongo:4').withRun('-p 27017:27017') { c ->
sh 'coverage run web/test/main.py'
}
}
}
}
post {
always {
sh 'coverage report'
}
}
}
}
}
This is the full log (after about 5 minutes it is hanging, it outputs the process apparently never started log in the end):
Started by user admin
Running in Durability level: MAX_SURVIVABILITY
[Pipeline] Start of Pipeline
[Pipeline] node
Running on Jenkins in /var/jenkins_home/workspace/test-github-action-ci-pipeline
[Pipeline] {
[Pipeline] isUnix
[Pipeline] sh
+ docker inspect -f . python:3.9
.
[Pipeline] withDockerContainer
Jenkins seems to be running inside container 76b85911bd303d0e6220f124d6d36f0c387e529f65aa845486eefb6fcd6687d9
but /var/jenkins_home/workspace/test-github-action-ci-pipeline could not be found among []
but /var/jenkins_home/workspace/test-github-action-ci-pipeline@tmp could not be found among []
$ docker run -t -d -u 1000:1000 --user 0:0 -w /var/jenkins_home/workspace/test-github-action-ci-pipeline -v /var/jenkins_home/workspace/test-github-action-ci-pipeline:/var/jenkins_home/workspace/test-github-action-ci-pipeline:rw,z -v /var/jenkins_home/workspace/test-github-action-ci-pipeline@tmp:/var/jenkins_home/workspace/test-github-action-ci-pipeline@tmp:rw,z -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** python:3.9 cat
$ docker top cbdd5ceb2bb4b31c21b19bcc292a1de499e92ce04c12f815a302d62cdbdd423c -eo pid,comm
[Pipeline] {
[Pipeline] stage
[Pipeline] { (build)
[Pipeline] sh
+ pip install -r requirements.txt
WARNING: Value for scheme.headers does not match. Please report this to <https://github.com/pypa/pip/issues/9617>
distutils: /usr/local/include/python3.9/UNKNOWN
sysconfig: /usr/local/include/python3.9
WARNING: Additional context:
user = False
home = None
root = None
prefix = None
Collecting flask==1.1.2
Downloading Flask-1.1.2-py2.py3-none-any.whl (94 kB)
Collecting pymongo==3.11.3
Downloading pymongo-3.11.3-cp39-cp39-manylinux2014_x86_64.whl (518 kB)
Collecting coverage==5.5
Downloading coverage-5.5-cp39-cp39-manylinux2010_x86_64.whl (243 kB)
Collecting python-dotenv==0.17
Downloading python_dotenv-0.17.0-py2.py3-none-any.whl (18 kB)
Collecting click>=5.1
Downloading click-7.1.2-py2.py3-none-any.whl (82 kB)
Collecting Werkzeug>=0.15
Downloading Werkzeug-1.0.1-py2.py3-none-any.whl (298 kB)
Collecting Jinja2>=2.10.1
Downloading Jinja2-2.11.3-py2.py3-none-any.whl (125 kB)
Collecting itsdangerous>=0.24
Downloading itsdangerous-1.1.0-py2.py3-none-any.whl (16 kB)
Collecting MarkupSafe>=0.23
Downloading MarkupSafe-1.1.1-cp39-cp39-manylinux2010_x86_64.whl (32 kB)
Installing collected packages: MarkupSafe, Werkzeug, Jinja2, itsdangerous, click, python-dotenv, pymongo, flask, coverage
WARNING: Value for scheme.headers does not match. Please report this to <https://github.com/pypa/pip/issues/9617>
distutils: /usr/local/include/python3.9/UNKNOWN
sysconfig: /usr/local/include/python3.9
WARNING: Additional context:
user = False
home = None
root = None
prefix = None
Successfully installed Jinja2-2.11.3 MarkupSafe-1.1.1 Werkzeug-1.0.1 click-7.1.2 coverage-5.5 flask-1.1.2 itsdangerous-1.1.0 pymongo-3.11.3 python-dotenv-0.17.0
WARNING: Running pip as root will break packages and permissions. You should install packages reliably by using venv: https://pip.pypa.io/warnings/venv
[Pipeline] sh
+ pip install -e .
WARNING: Value for scheme.headers does not match. Please report this to <https://github.com/pypa/pip/issues/9617>
distutils: /usr/local/include/python3.9/UNKNOWN
sysconfig: /usr/local/include/python3.9
WARNING: Additional context:
user = False
home = None
root = None
prefix = None
Obtaining file:///var/jenkins_home/workspace/test-github-action-ci-pipeline
Requirement already satisfied: flask in /usr/local/lib/python3.9/site-packages (from web==0.0.0) (1.1.2)
Requirement already satisfied: pymongo in /usr/local/lib/python3.9/site-packages (from web==0.0.0) (3.11.3)
Requirement already satisfied: coverage in /usr/local/lib/python3.9/site-packages (from web==0.0.0) (5.5)
Requirement already satisfied: Werkzeug>=0.15 in /usr/local/lib/python3.9/site-packages (from flask->web==0.0.0) (1.0.1)
Requirement already satisfied: itsdangerous>=0.24 in /usr/local/lib/python3.9/site-packages (from flask->web==0.0.0) (1.1.0)
Requirement already satisfied: Jinja2>=2.10.1 in /usr/local/lib/python3.9/site-packages (from flask->web==0.0.0) (2.11.3)
Requirement already satisfied: click>=5.1 in /usr/local/lib/python3.9/site-packages (from flask->web==0.0.0) (7.1.2)
Requirement already satisfied: MarkupSafe>=0.23 in /usr/local/lib/python3.9/site-packages (from Jinja2>=2.10.1->flask->web==0.0.0) (1.1.1)
Installing collected packages: web
Running setup.py develop for web
WARNING: Value for scheme.headers does not match. Please report this to <https://github.com/pypa/pip/issues/9617>
distutils: /usr/local/include/python3.9/UNKNOWN
sysconfig: /usr/local/include/python3.9
WARNING: Additional context:
user = False
home = None
root = None
prefix = None
Successfully installed web-0.0.0
WARNING: Running pip as root will break packages and permissions. You should install packages reliably by using venv: https://pip.pypa.io/warnings/venv
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (test)
[Pipeline] script
[Pipeline] {
[Pipeline] node
Running on Jenkins in /var/jenkins_home/workspace/test-github-action-ci-pipeline@2
[Pipeline] {
[Pipeline] isUnix
[Pipeline] sh
process apparently never started in /var/jenkins_home/workspace/test-github-action-ci-pipeline@2@tmp/durable-d6462e0e
(running Jenkins temporarily with -Dorg.jenkinsci.plugins.durabletask.BourneShellScript.LAUNCH_DIAGNOSTICS=true might make the problem clearer)
Cannot contact : java.io.FileNotFoundException: File '/var/jenkins_home/workspace/test-github-action-ci-pipeline@2@tmp/durable-d6462e0e/output.txt' does not exist
Without the MongoDB container, it works as a charm. The following example is working fine:
pipeline {
agent {
docker {
image 'python:3.9'
args '--user 0:0'
}
}
stages {
stage('build') {
steps {
sh 'pip install -r requirements.txt'
sh 'pip install -e .'
}
}
stage('test') {
steps {
sh 'coverage run web/test/main.py'
}
post {
always {
sh 'coverage report'
}
}
}
}
}
Ok, I did it. This is working fine, even though that might not be the best solution (it looks a bit "hacky").
pipeline {
agent any
stages {
stage('MongoDB Setup') {
steps {
script {
try {
sh 'docker stop mongo_unittest'
sh 'docker rm mongo_unittest'
} catch (err) {
echo 'docker mongo_unittest is not already running' // err.getMessage()
}
}
sh 'docker pull mongo:4'
sh 'docker run -d --name mongo_unittest -p 27017:27017 mongo:4'
}
}
stage('Run Flask Unittests') {
agent {
docker {
image 'python:3.9'
args '--user 0:0 --net host'
}
}
steps {
checkout scm
sh 'pip install -r requirements.txt'
sh 'pip install -e .'
sh 'echo "MONGO_HOST=127.0.0.1" >> web/.env'
sh 'coverage run web/test/main.py'
}
post {
always {
sh 'coverage report'
}
}
}
}
post {
always {
sh 'docker stop mongo_unittest'
sh 'docker rm mongo_unittest'
}
}
}