GCP BigTable Metrics - what do 404 requests mean?

Question

We switched to BigTable some time ago and since then there is a number of "404 requests" and also a high number of errors in the GCP Metrics console.

We see no errors in our logs and even data storage/retrieval seems to work as expected. What is the cause for these errors and how is it possible to find out what is causing them?

Answer 1

As mentioned previously 404 means resource is not found. The relevant resource here is the Bigtable table (which could mean that either the instance id or table id are misconfigured in your application).

I'm guessing that you are looking at the metrics under APIs & Services > Cloud Bigtable API. These metrics show the response code from the Cloud Bigtable Service. You should be able to see this error rate under Monitoring > Metrics Explorer > metric:bigtable.googleapis.com/server/error_count and grouping by instance, method, error_code and app_profile. This will tell which instance and which RPC is causing the errors. Which let you grep your source code for incorrect usages.

A significantly more complex approach is that you can install an interceptor in Bigtable client that:

1. dumps the resource name of the RPC
2. once you identify the problematic table name, logs the stack trace of the caller

Something along these lines:

BigtableDataSettings.Builder builder = BigtableDataSettings.newBuilder()
        .setProjectId("...")
        .setInstanceId("...");

ConcurrentHashMap<String, Boolean> seenTables = new ConcurrentHashMap<>();

builder.stubSettings().setTransportChannelProvider(
        EnhancedBigtableStubSettings.defaultGrpcTransportProviderBuilder()
            .setInterceptorProvider(() -> ImmutableList.of(new ClientInterceptor() {
              @Override
              public <ReqT, RespT> ClientCall<ReqT, RespT> interceptCall(
                  MethodDescriptor<ReqT, RespT> methodDescriptor, CallOptions callOptions,
                  Channel channel) {
                return new ForwardingClientCall.SimpleForwardingClientCall<ReqT, RespT>(channel.newCall(methodDescriptor, callOptions)) {
                  @Override
                  public void sendMessage(ReqT message) {
                    Message protoMessage = (Message) message;
                    FieldDescriptor desc = protoMessage.getDescriptorForType()
                        .findFieldByName("table_name");
                    if (desc != null) {
                      String tableName = (String) protoMessage.getField(desc);
                      if (seenTables.putIfAbsent(tableName, true) == null) {
                        System.out.println("Found new tableName: " + tableName);
                      }
                      if ("projects/my-project/instances/my-instance/tables/my-mispelled-table".equals(
                          tableName)) {
                        new RuntimeException(
                            "Fake error to get caller location of mispelled table id").printStackTrace();
                      }
                    }
                    delegate().sendMessage(message);
                  }
                };
              }
            }))
            .build()
    );