I'm working on a Flutter project and need to run integration tests on a Gitlab CI pipeline.
The tests work fine locally, launching chromedriver -port=4444 and then flutter drive --driver=integration_test/test_driver/integration_test.dart --target=integration_test/integration/app_test.dart.
However, I cannot get them to run in CI. My yaml file looks like this :
...
test_ui_integration:
needs:
- build_deploy_user_service
- populate_db
tags:
- docker
stage: test_ui_integration
rules:
- if: $CI_PIPELINE_SOURCE == "merge_request_event"
when: always
before_script:
- chromedriver --port=4444 & # launching the chromedriver
- sleep 10
script:
- cd $CI_PROJECT_DIR/ui
- npm ${NODE_INSTALL_OPTIONS:-i}
- npm run build
- flutter drive --driver=integration_test/test_driver/integration_test.dart --target=integration_test/integration/app_test.dart -d chrome --web-run-headless --headless
# - flutter drive --driver=integration_test/test_driver/integration_test.dart --target=integration_test/integration/app_test.dart -d web-server --headless
# - flutter drive --driver=integration_test/test_driver/integration_test.dart --target=integration_test/integration/app_test.dart -d web-server --release --web-run-headless --headless
timeout: 40 minutes
(note that the pipeline is using a docker img with chromedriver and Flutter installed)
I have tried all kinds of different combination for the flags, as seen here too. However, none of it seems to work: Even locally, I cannot get the test to run headless-ly (without a browser).
The CI step usually gets stuck on something like this, until it times out :
=== running with flags -d web-server --web-run-headless --headless ===
Waiting for connection from debug service on Web Server... 37.1s
integration_test/integration/app_test.dart is being served at http://localhost:44415
The web-server device requires the Dart Debug Chrome extension for debugging. Consider using the Chrome or Edge devices for an improved development workflow.
...
=== running with flags -d chrome --web-run-headless --headless ===
Waiting for connection from debug service on Chrome... 37.5s
[CHROME]:[1391:1391:0609/090100.591700:ERROR:zygote_host_impl_linux.cc(90)] Running as root without --no-sandbox is not supported. See https://crbug.com/638180.
Failed to launch browser after 3 tries. Command used to launch it: google-chrome --user-data-dir=/tmp/flutter_tools.RDVJEL/flutter_tools_chrome_device.KRPHNF --remote-debugging-port=41647 --disable-background-timer-throttling --disable-extensions --disable-popup-blocking --bwsi --no-first-run --no-default-browser-check --disable-default-apps --disable-translate http://localhost:35691
I've also noticed that you can run integration tests locally with flutter run path/to/my/test/dart, which is something that the official docs never mention. Is there a difference between the two?
Should I be trying to run the CI tests with a mobile emulator instead?
Any help much appreciated!
EDIT: As requested by @MarkosTh09, here is the dockerfile and relevant CI workflow :
FROM fedora:35
# Add basic tools
RUN dnf -y install git make gcc gcc-c++ wget unzip zip jq diffutils \
&& dnf clean all
# Requirement of openapi generator
RUN dnf install -y java \
&& dnf clean all
# Install python
RUN dnf install -y python3 \
&& dnf clean all
# Install chromedriver
RUN wget https://chromedriver.storage.googleapis.com/99.0.4844.35/chromedriver_linux64.zip \
&& unzip chromedriver_linux64.zip \
&& cp chromedriver /usr/bin/chromedriver \
&& chown root /usr/bin/chromedriver \
&& chmod +x /usr/bin/chromedriver \
&& chmod 755 /usr/bin/chromedriver
# Install chrome
RUN wget https://dl.google.com/linux/chrome/rpm/stable/x86_64/google-chrome-stable-99.0.4844.84-1.x86_64.rpm \
&& yum install ./google-chrome-*.rpm -y
# Install AWS CLI V2
RUN curl -s https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip > awscliv2.zip \
&& unzip awscliv2.zip \
&& ./aws/install \
&& rm -rf aws awscliv2.zip
# Install and configure nvm for npm
# nvm environment variables
ENV NVM_DIR /opt/nvm
RUN mkdir -p $NVM_DIR
ENV NODE_VERSION 16.14.0
# install nvm
RUN curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.1/install.sh | bash
# install node and npm
RUN source $NVM_DIR/nvm.sh \
&& nvm install $NODE_VERSION \
&& nvm alias default $NODE_VERSION \
&& nvm use default
# add node and npm to path so the commands are available
ENV NODE_PATH $NVM_DIR/versions/node/v$NODE_VERSION/lib/node_modules
ENV NVM_BIN $NVM_DIR/versions/node/v$NODE_VERSION/bin
ENV NVM_INC $NVM_DIR/versions/node/v$NODE_VERSION/include/node
ENV PATH $NVM_BIN:$PATH
# Install serverless
RUN npm install -g serverless@3.2.0
# Install Flutter
RUN curl -s https://storage.googleapis.com/flutter_infra_release/releases/stable/linux/flutter_linux_3.0.0-stable.tar.xz > flutter.tar.xz \
&& tar xf flutter.tar.xz -C /opt/ \
&& rm flutter.tar.xz
ENV PATH $PATH:/opt/flutter/bin
RUN git config --global --add safe.directory /opt/flutter
CMD [ "/bin/bash" ]
ENTRYPOINT [ "/bin/bash", "-c" ]
include:
- '/ci/ci-build-deploy.yml'
- '/ci/ci-tests.yml'
- '/ci/ci-undeploy.yml'
image: registry.gitlab.com/project/project/build/project-build-env:4
variables:
PROJECT_API_URL: https://gitlab.com/api/v4/projects/12345678
NODE_INSTALL_OPTIONS: ci --cache $CI_PROJECT_DIR/.npm --prefer-offline
cache:
key: PROJECT_NODE_CACHE
paths:
- .npm/
workflow:
rules:
- if: $CI_PIPELINE_SOURCE == "merge_request_event"
variables:
AWS_STAGE: "ci${CI_PIPELINE_ID}"
STAGE_TYPE: "ci"
- if: $STAGE_TYPE == "staging" && $AWS_STAGE == "staging"
variables:
HOSTED_ZONE_NAME: "staging.project.io"
FRONT_DOMAIN: "staging.project.io"
- if: $STAGE_TYPE == "prod" && $AWS_STAGE == "prod"
variables:
HOSTED_ZONE_NAME: "project.io"
FRONT_DOMAIN: "project.io"
- when: always
stages:
- build_deploy_user_service
- build_deploy_shop_service
- build_deploy_brand_service
- build_deploy_product_service
- build_deploy_web_ui
- build_deploy_reverseproxy
- test_shop_ws
- test_brand_ws
- test_product_ws
- delete_test_objects
- populate_db
- test_ui
- populate_cognito_test_user
- test_ui_integration
- triggerundeploy
- undeploy
I'm not working anymore on that particular Flutter project, but I've been told the e2e testing works with a setup similar to this:
FROM centos:8
# Failed to download metadata for repo ‘AppStream’ [CentOS] : https://techglimpse.com/failed-metadata-repo-appstream-centos-8/
RUN cd /etc/yum.repos.d/ && sed -i 's/mirrorlist/#mirrorlist/g' /etc/yum.repos.d/CentOS-* && sed -i 's|#baseurl=http://mirror.centos.org|baseurl=http://vault.centos.org|g' /etc/yum.repos.d/CentOS-*
# Add basic tools
RUN dnf install -y git make gcc gcc-c++ wget unzip zip jq diffutils zlib.i686 ncurses-libs.i686 bzip2-libs.i686 curl which mesa-libGLU patch \
&& dnf clean all
# ............
# Install Flutter
RUN wget https://storage.googleapis.com/flutter_infra_release/releases/stable/linux/flutter_linux_3.13.9-stable.tar.xz \
&& tar xf flutter_linux_3.13.9-stable.tar.xz && rm flutter_linux_3.13.9-stable.tar.xz
ENV PATH "$PATH:/flutter/bin"
# Install firebase CLI
RUN npm install -g firebase-tools
# Install Android SDK
ENV VERSION_TOOLS "9477386"
ENV ANDROID_SDK_ROOT "/sdk"
# Keep alias for compatibility
ENV ANDROID_HOME "${ANDROID_SDK_ROOT}"
ENV PATH "$PATH:${ANDROID_SDK_ROOT}/cmdline-tools/latest/bin:${ANDROID_SDK_ROOT}/platform-tools"
RUN curl -s https://dl.google.com/android/repository/commandlinetools-linux-${VERSION_TOOLS}_latest.zip > /cmdline-tools.zip \
&& mkdir -p ${ANDROID_SDK_ROOT}/cmdline-tools \
&& unzip /cmdline-tools.zip -d ${ANDROID_SDK_ROOT}/cmdline-tools \
&& mv ${ANDROID_SDK_ROOT}/cmdline-tools/cmdline-tools ${ANDROID_SDK_ROOT}/cmdline-tools/latest \
&& chmod -R 777 $ANDROID_SDK_ROOT \
&& rm -v /cmdline-tools.zip
RUN mkdir -p $ANDROID_SDK_ROOT/licenses/ \
&& echo "8933bad161af4178b1185d1a37fbf41ea5269c55\nd56f5187479451eabf01fb78af6dfcb131a6481e\n24333f8a63b6825ea9c5514f83c2829b004d1fee" > $ANDROID_SDK_ROOT/licenses/android-sdk-license \
&& echo "84831b9409646a918e30573bab4c9c91346d8abd\n504667f4c0de7af1a06de9f4b1727b84351f2910" > $ANDROID_SDK_ROOT/licenses/android-sdk-preview-license \
&& yes | sdkmanager --licenses >/dev/null \
&& chmod -R 777 $ANDROID_SDK_ROOT/licenses/
RUN mkdir -p /root/.android \
&& touch /root/.android/repositories.cfg \
&& sdkmanager --update
ADD docker-env/build/packages.txt /sdk
RUN sdkmanager --package_file=/sdk/packages.txt && chmod -R 777 $ANDROID_SDK_ROOT
RUN sdkmanager "system-images;android-33;google_apis;x86_64"
RUN avdmanager create avd --force --name Pixel5 --abi google_apis/x86_64 --package "system-images;android-33;google_apis;x86_64" --device "pixel_5"
# Configure android emulator
RUN sed -i 's/disk.dataPartition.size=800M/disk.dataPartition.size=2000M/g' /root/.android/avd/Pixel5.avd/config.ini
RUN sed -i 's/hw.keyboard=no/hw.keyboard=yes/g' /root/.android/avd/Pixel5.avd/config.ini
CMD [ "/bin/bash" ]
ENTRYPOINT [ "/bin/bash", "-c" ]
Apparently we don't need to use flutter drive anymore on the CI step:
test_ui:
tags:
- docker
stage: test_ui
needs:
- ...
- ...
rules:
- if: $CI_PIPELINE_SOURCE == "merge_request_event"
when: always
before_script:
- !reference [.aws-prep, before_script]
script:
- git fetch origin
- export PUB_CACHE=$CI_PROJECT_DIR/.pub-cache
- export PATH="$PATH":"$PUB_CACHE/bin"
- echo "to see android emulator test in live, go to http://RUNNER_IP:6080"
- cd $CI_PROJECT_DIR/mobile/android
- bundle install
- bundle exec fastlane android bump_patch
- cd $CI_PROJECT_DIR/mobile
- flutter pub get
- flutter test
- adb connect <docker_host_ip>:5555
- flutter test integration_test/*.dart --dart-define=AWS_STAGE=${AWS_STAGE} --dart-define=AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID:-${GITLAB_TEST_AWS_ACCESS_KEY_ID}} --dart-define=AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY:-${GITLAB_TEST_AWS_SECRET_ACCESS_KEY}}
timeout: 60 minute
retry: 1