App Engine stackdriver logging to Global log instead of service log

Question

I'm trying to set up logging for a django app hosted as an App Engine service on GAE.

I have set up the logging succesfully, except that the logging is showing up in the global log for that entire project instead of the log for that service. I would like for the logs to show up only in the specific service logs

this is my django logging config:

from google.cloud import logging as google_cloud_logging


log_client = google_cloud_logging.Client()
log_client.setup_logging()

LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    'handlers': {
        'stackdriver_logging': {
            'class': 'google.cloud.logging.handlers.CloudLoggingHandler',
            'client': log_client
        },
    },
    'loggers': {
        '': {
            'handlers': ['stackdriver_logging'],
            'level': 'INFO',
        }
    },
}

And I am able to succesfully log to the Global project log by calling like this:

def fetch_orders(request):
    logger.error('test error')
    logger.critical('test critical')
    logger.warning('test warning')
    logger.info('test info')
    return redirect('dashboard')

I'm wanting to figure out if I can configure the logger to always use the log for the service that it's running in.

EDIT:

I tried the suggestion below, however now it is returning the following error:

Traceback (most recent call last):
  File "/env/lib/python3.7/site-packages/google/cloud/logging/handlers/transports/background_thread.py", line 122, in _safely_commit_batch
    batch.commit()
  File "/env/lib/python3.7/site-packages/google/cloud/logging/logger.py", line 381, in commit
    entries = [entry.to_api_repr() for entry in self.entries]
  File "/env/lib/python3.7/site-packages/google/cloud/logging/logger.py", line 381, in <listcomp>
    entries = [entry.to_api_repr() for entry in self.entries]
  File "/env/lib/python3.7/site-packages/google/cloud/logging/entries.py", line 318, in to_api_repr
    info = super(StructEntry, self).to_api_repr()
  File "/env/lib/python3.7/site-packages/google/cloud/logging/entries.py", line 241, in to_api_repr
    info["resource"] = self.resource._to_dict()
AttributeError: 'ConvertingDict' object has no attribute '_to_dict'

I can over-ride this in the package source code to make it work, however the GAE environment requires that I use the package as supplied by google for the cloud logging. Is there any way to go from here?

Answer 1

To my understanding, it should be possible to accomplish what you want using the resource option of CloudLoggingHandler. In the Stackdriver Logging (and Stackdriver Monitoring) API, each object (log line, time-series point) is associated with a "resource" (some thing that exists in a project, that can be provisioned, and can be the source of logs or time-series or the thing that the logs or time-series are being written about). When the resource option is omitted, the CloudLoggingHandler defaults to global as you have observed.

There are a number of monitored resource types, including gae_app, which can be used to represent a specific version of a particular service that is deployed on GAE. Based on your code, this would look something like:

from google.cloud.logging import resource


def get_monitored_resource():
  project_id = get_project_id()
  gae_service = get_gae_service()
  gae_service_version = get_gae_service_version()
  resource_type = 'gae_app'
  resource_labels = {
    'project_id': project_id,
    'module_id': gae_service,
    'version_id': gae_service_version
  }
  return resource.Resource(resource_type, resource_labels)


GAE_APP_RESOURCE = get_monitored_resource() 
LOGGING = {
    # ...
    'handlers': {
        'stackdriver_logging': {
            'class': 'google.cloud.logging.handlers.CloudLoggingHandler',
            'client': log_client,
            'resource': GAE_APP_RESOURCE,
        },
    },
    # ...
}

In the code above, the functions get_project_id, get_gae_service, and get_gae_service_version can be implemented in terms of the environment variables GOOGLE_CLOUD_PROJECT, GAE_SERVICE, and GAE_VERSION in the Python flexible environment as documented by The Flexible Python Runtime as in:

def get_project_id():
  return os.getenv('GOOGLE_CLOUD_PROJECT')