Search code examples
grpcbazel

Bazel runs very slowly


I have a relatively simple Bazel project which has the following:

  • 2 proto files (B.proto depends on A.proto)
  • Go/C++ libraries produced from these proto files with grpc support (used grpc/protobuf rules imported from pubref/rules_protobuf)
  • Server/client applications against these protos written in C++ and Go.

When I run bazel for the first time, it takes a LOT of time to execute. It compiles grpc, protobuf etc, which makes sense.

However, when I immediately run the compile again, my build takes ~80s even in the incremental case. For a project which is so simple, I would have expected much faster performance - especially since speed is said to be Bazel's primary advantage.

From what I can tell, the performance of my bazel builds was pretty fast until I incorporated grpc/protos.

Here's some information reported by bazel's profiler. I am unable to see any smoking guns in the profiler output.

One possible difference is that my builds run on an ubuntu docker container hosted on a macbook. The macos docker implementation runs on a lightweight hyperkit VM. So this is not a native build. But I would still not expect things to be this slow!

Phase Summary Information

  • Total launch phase time 101 ms 0.12%
  • Total init phase time 11.560 s 13.67%
  • Total loading phase time 282 ms 0.33%
  • Total analysis phase time 15.2 ms 0.02%
  • Total preparation phase time 1.002 s 1.19%
  • Total execution phase time 71.549 s 84.63%
  • Total finish phase time 30.9 ms 0.04%
  • Total run time 84.540 s 100.00%

Init Phase Information

  • Total init time 11.560 s
  • Total time (across all threads) spent on:
    • Type Total Count Average
    • VFS_STAT 88.18% 12223 166 ms
    • VFS_DIR 10.49% 785 307 ms
    • VFS_READLINK 0.81% 221 84.4 ms
    • VFS_OPEN 0.01% 2 109 ms
    • VFS_READ 0.01% 4 28.7 ms

Execution Phase Information

  • Total preparation time 1.002 s
  • Total execution time 71.549 s
  • Total finish time 30.9 ms
  • Action dependency map creation 0.00 ms
  • Actual execution time 71.549 s
  • Total time (across all threads) spent on:
    • Type Total Count Average
    • ACTION 0.00% 1 2.09 ms
    • ACTION_CHECK 0.00% 1 0.71 ms
    • ACTION_EXECUTE 0.00% 1 1.53 ms
    • INFO 0.00% 1 0.00 ms
    • VFS_STAT 39.71% 1803 26.3 ms
    • VFS_DIR 0.02% 2 14.0 ms
    • VFS_READLINK 0.36% 18 23.9 ms
    • VFS_MD5 0.00% 1 1.45 ms
    • VFS_DELETE 0.00% 1 1.44 ms
    • VFS_OPEN 0.02% 5 4.30 ms
    • VFS_READ 0.00% 4 0.48 ms
    • VFS_WRITE 0.00% 2 0.32 ms
    • SKYFRAME_EVAL 0.03% 1 31.0 ms
    • SKYFUNCTION 0.02% 5 5.83 ms
    • Critical path (25.7 ms):
    • Id Time Share Description
    • 15078 25.7 ms 100.00% action 'BazelWorkspaceStatusAction stable-status.txt'

Solution

  • I tried the same build on an AWS EC2 instance. There the incremental build is substantially faster. So I am assuming the slowness is because of some filesystem issue due to running inside VM.