Search code examples
google-cloud-platformcontinuous-integrationgoogle-kubernetes-enginegoogle-cloud-buildcontinuous-testing

Parallelizing google cloudbuild steps without mangling the build logs

Following guidance here: https://cloud.google.com/cloud-build/docs/configuring-builds/configure-build-step-order

We have split up our build into a multilayer docker image where layers are

  1. Install OS and third party deps
  2. Install our sources and build (in debug or release depending on docker ARG)
  3. Run ci / code coverage

We have tagged the steps in cloudbuild.yaml with docker id's and are using waitfor to try to make it so that the debug and release versions can run in parallel.

However, when we do this, the build log is mixed up -- the build logs for release and debug are are jumbled together making it much harder to read. e.g.

Step #2 - "build-debug": �[0m�[91m  Downloaded colored v1.7.0
Step #5 - "build-release": �[0m�[91merror: couldn't read /tmp/mobilenode/src/attest/src/ias/../data/AttestationReportSigningCACert.pem: No such file or directory (os error 2)
Step #5 - "build-release":   --> /tmp/mobilenode/src/attest/src/ias/verify.rs:35:7
Step #5 - "build-release":    |
Step #5 - "build-release": 35 |     &[include_str!("../data/AttestationReportSigningCACert.pem")];
Step #5 - "build-release":    |       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Step #5 - "build-release": 
Step #2 - "build-debug": �[0m�[91m  Downloaded toml v0.4.10

[0m�[91m  Downloaded safemem v0.3.0
Step #2 - "build-debug": �[0m�[91m  Downloaded crunchy v0.1.6
Step #2 - "build-debug": �[0m�[91m  Downloaded grpcio-sys v0.4.4
Step #5 - "build-release": �[0m�[91merror: aborting due to previous error
Step #5 - "build-release": 
Step #5 - "build-release": �[0m�[91merror: Could not compile `attest`.
Step #5 - "build-release": warning: build failed, waiting for other jobs to finish...
Step #5 - "build-release": �[0m�[91merror: build failed
Step #5 - "build-release": �[0m�[91mmake: *** [src/enclave/target/release/libenclave.so] Error 101
Step #5 - "build-release": �[0mMakefile:90: recipe for target 'src/enclave/target/release/libenclave.so' failed
Step #2 - "build-debug": �[0m�[91m  Downloaded term v0.5.1
Step #2 - "build-debug": �[0m�[91m  Downloaded tiny_http v0.6.2
Step #2 - "build-debug": �[0m�[91m  Downloaded regex v0.1.80

Is there a way to configure cloudbuild so that there is a separate log file for each build step? Is the best answer to use a different cloudbuild.yaml for the parallel steps, and skip all this wait_for stuff?

Solution

  • I was having this same issue, but then I discovered that viewing a build via the History under Cloud Build in the Console will let you view each step individually, or view the entire job, like you're currently seeing (click Build Summary, the top entry in the sidebar). Click on the name of the step in sidebar, each step should be comprised of a number followed by a colon and then the name of the step. The number appears to be based upon the order in which the job appears in your cloudbuild.yaml

    What you're seeing is a result of the parallelization of your build, as each step that is marked as being able to run concurrently is at a different step in its execution.

    Example Build You can read more about viewing the Build History using the console here.