Search code examples
testingfluttercontinuous-integrationintegration-testingtravis-ci

Flutter Integration Tests with Travis CI

1. The Summary of the Problem

I would like a Travis CI setup that would let me run flutter driver tests inside an Android and an iOS environments. For this to happen, I expect I somehow have to install Flutter, Android and iOS in different environments.

2. What I Have So Far

Most of the posts I've been able to find on this topic are terribly outdated or feature incredibly complicated setups. Some of those that keep on appearing in my searches are:

  • Test Flutter apps on Travis, by Yegor Jbanov. This one covers unit and widget testing (flutter test), but not integration tests.

    • It's from early 2017 and Travis CI has maybe simplified its API, because I've managed to make it work with only this:

      language: dart

dart:
  - stable

dart_task:
  - dartfmt

install:
  - git clone https://github.com/flutter/flutter.git -b stable

script:
  - ./flutter/bin/flutter doctor
  - ./flutter/bin/flutter test
  • One resource that I've found very useful is the .travis.yml in the Flutter samples repo. The setup there seems very complicated to me though.
  • The closest I could get to what I wanted is similar to Maurice McCabe's Flutter unit, widget and integration testing with IOS and Android emulators on Travis-CI.
    • Again, this seems overcomplicated and outdated.

3. The Sketch of What I Have in Mind

The script and install steps in the example I mentioned previously could be replaced by jobs with stages. In this way, each stage would represent one sort of step. Unit and Widget stages in one, integration tests on Android and iOS in two others, which is similar to what Maurice McCabe and Flutter samples show. For example:

jobs:
  include:
    - stage: Flutter Test
      language: dart
      os: linux
      install: git clone $FLUTTER_GITHUB -b stable
      before_script:
        - ./flutter/bin/flutter doctor
      script: 
        - ./flutter/bin/flutter test
    - stage: Integration Test on Android
      os: linux
      dist: trusty
      language: android
      android: # the things here are what probably needs to be fixed
        components:
          - build-tools-28.0.3
          - android-28
      install: git clone $FLUTTER_GITHUB -b stable
      before_script:
        - ./flutter/bin/flutter doctor
      script: 
        - ./flutter/bin/flutter drive --target=test_driver/app.dart

If I could create a stage for the dartfmt task that would also be nice in terms of organization.

Solution

  • 1. The Overview

    Kudos to @MirceaMatei and Maurice McCabe for helping out in this.

    1. I haven't yet been able to make Android Integration Tests work (the last stage of the code below), but at least iOS is working.
      • Android is much harder to get right because of different versions and licenses, something Apple does much better.
    2. The code you will find below is part of a custom mono repo setup I'm working on.
      • The folder structure consists of an app and a packages folders.
    3. I haven't yet segmented code coverage by those two folders either, so this setup is currently overwriting code coverage from one folder to another.
    4. Using a top level install section makes Travis repeat that setup for every stage, so that avoids having to do that repeatedly for every stage.

    I'm sharing my current — albeit incomplete — setup below, but continuous improvement will happen through this Github Gist, which is a much better way of interacting with the community to improve code than StackOverflow — it's not its purpose.

    All in all, I think Travis CI is really dropping the ball when it comes to Flutter — and maybe Android in general. An easier alternative that has been hyped by many practitioners I've come into contact recently is Codemagic, which also offers code signing and automated deployment to the iOS and Android app stores.

    2. Useful Resources

    I didn't really like Travis documentation for the integration tests. People creating pure Android apps filed similar issues.

    Anyway, here are some useful resources I've found during my searches:

    1. Issue Comment on "Error: Target id is not valid. Use 'android list targets' to get the target ids."
    2. “Invalid --abi armeabi-v7a for the selected target” with Google APIs
    3. Is there a way to start android emulator in Travis CI build?
    4. Travis-CI Android 28 licenses have not been accepted
    5. Flutter Samples Travis Setup
    6. @MirceaMatei's Travis Setup Recommendation
    7. Travis Building Android Projects Documentation
    8. Maurice McCabe's Unit and Integration Tests with Travis Article on Medium

    3. The Code

    Please, do not post code improvement suggestions here, but in the the Github Gist. I'll be updating the yaml code below myself every time a useful iteration arrives.

    language: dart

env:
  global:
    - DARTSDK=./flutter/bin/cache/dart-sdk/bin
    - DARTFMT=$DARTSDK/dartfmt
    - FLUTTER=./flutter/bin/flutter
    - FLUTTER_UP=../flutter/bin/flutter
    - FLUTTER_GITHUB=https://github.com/flutter/flutter.git
    - CODECOV=https://codecov.io/bash
    - PACKAGES=packages
    - APP=app
    - ANDROID_API=28

install:
  - git clone $FLUTTER_GITHUB -b stable --depth 1
  - $FLUTTER doctor
  - $FLUTTER pub get

jobs:
  include:
    - stage: Formatting
      script:
        - $DARTFMT -n $PACKAGES --set-exit-if-changed

    - stage: Packages Flutter Test
      script:
        - $FLUTTER test --coverage $PACKAGES
      after_success:
        - bash <(curl -s $CODECOV)

    - stage: App Flutter Test
      script:
        - cd $APP
        - $FLUTTER_UP test --coverage
      after_success:
        - bash <(curl -s $CODECOV)

    - stage: iOS Integration Tests
      os: osx
      osx_image: xcode11
      before_script:
        - open /Applications/Xcode.app/Contents/Developer/Applications/Simulator.app
        - export HOMEBREW_NO_AUTO_UPDATE=1
        - brew install libimobiledevice
        - brew install ideviceinstaller
        - brew install ios-deploy
        - brew install cocoapods || echo 'ignore exit(1)'
        - brew link --overwrite cocoapods
      script:
        - cd $APP
        - $FLUTTER_UP driver --target=test_driver/app.dart77

    - stage: Android Integration Tests
      language: android
      dist: trusty
      android:
        components:
          - tools
          - platform-tools
          - build-tools-25.0.3
          - android-24
          - android-22
          - extra-google-google_play_services
          - extra-google-m2repository
          - extra-android-m2repository
          - sys-img-armeabi-v7a-android-22
      licenses:
        - 'android-sdk-preview-license-52d11cd2'
        - 'android-sdk-license-.+'
        - 'google-gdk-license-.+'
      before_install:
        - yes | sdkmanager "platforms;android-28"
        - echo no | android create avd --force -n test -t android-22 --abi armeabi-v7a -c 32M
        - emulator -avd test -no-audio -no-window &
        - android-wait-for-emulator
        - adb devices
        - adb shell input keyevent 82 &
      script:
        - $FLUTTER --version
        - $FLUTTER doctor
        - cd $APP
        - $FLUTTER_UP devices
        - $FLUTTER_UP driver --target=test_driver/app.dart