Using YAML file to setup GCP Cloud Logging

Question

I have the following YAML file setup for my logger in my project:

---
version: 1
disable_existing_loggers: False
formatters:
  simple:
    format: "%(asctime)s - %(name)s - %(levelname)s - %(message)s"
  colored:
    (): "colorlog.ColoredFormatter"
    datefmt: "%Y-%m-%d %H:%M:%S"
    format: "%(white)s%(asctime)s.%(msecs)03d%(reset)s - %(cyan)s[%(module)s.%(funcName)s]%(reset)s - %(log_color)s[%(levelname)s] :=>%(reset)s %(message)s"
    log_colors:
      DEBUG: purple
      INFO: blue
      WARNING: yellow
      ERROR: red
      CRITICAL: red,bg_white
handlers:
  console:
    class: logging.StreamHandler
    level: DEBUG
    formatter: colored
    stream: ext://sys.stdout
root:
  level: INFO
  handlers: [console]

I would like to add the GCP Stackdriver handler to this file but am having an issue. The cloud log handler requires a client to be set and I am unsure of how to do this in the YAML.

I've tried added the following:

handlers:
    stackdriver:
      class: google.cloud.logging.handlers.CloudLoggingHandler
      client: ext://google.cloud.logging.Client
      name: my-project-log

However, I get the following error:

TypeError: logger() missing 1 required positional argument: 'name'

Any clues on how I can configure the GCP cloud log handler from a YAML file? I've had no luck even when changing class to () to define a user defined object for logging.

Answer 1

I was able to get this working by modifying/copying the CloudLoggingHandler class and changing the client to default client=google.cloud.logging.Client()

import logging
import google.cloud.logging

from google.cloud.logging.handlers.transports import BackgroundThreadTransport
from google.cloud.logging.logger import _GLOBAL_RESOURCE

DEFAULT_LOGGER_NAME = "python"
EXCLUDED_LOGGER_DEFAULTS = ("google.cloud", "google.auth", "google_auth_httplib2")


class Stackdriver(logging.StreamHandler):
    """Handler that directly makes Stackdriver logging API calls.

    This is a Python standard ``logging`` handler using that can be used to
    route Python standard logging messages directly to the Stackdriver
    Logging API.

    This handler is used when not in GAE or GKE environment.

    This handler supports both an asynchronous and synchronous transport.

    :type client: :class:`google.cloud.logging.client.Client`
    :param client: the authenticated Google Cloud Logging client for this
                   handler to use

    :type name: str
    :param name: the name of the custom log in Stackdriver Logging. Defaults
                 to 'python'. The name of the Python logger will be represented
                 in the ``python_logger`` field.

    :type transport: :class:`type`
    :param transport: Class for creating new transport objects. It should
                      extend from the base :class:`.Transport` type and
                      implement :meth`.Transport.send`. Defaults to
                      :class:`.BackgroundThreadTransport`. The other
                      option is :class:`.SyncTransport`.

    :type resource: :class:`~google.cloud.logging.resource.Resource`
    :param resource: (Optional) Monitored resource of the entry, defaults
                     to the global resource type.

    :type labels: dict
    :param labels: (Optional) Mapping of labels for the entry.

    :type stream: file-like object
    :param stream: (optional) stream to be used by the handler.

    Example:

    .. code-block:: python

        import logging
        import google.cloud.logging
        from google.cloud.logging.handlers import CloudLoggingHandler

        client = google.cloud.logging.Client()
        handler = CloudLoggingHandler(client)

        cloud_logger = logging.getLogger('cloudLogger')
        cloud_logger.setLevel(logging.INFO)
        cloud_logger.addHandler(handler)

        cloud_logger.error('bad news')  # API call
    """

    def __init__(self, client=google.cloud.logging.Client(), name=DEFAULT_LOGGER_NAME, transport=BackgroundThreadTransport, resource=_GLOBAL_RESOURCE, labels=None, stream=None):
        super(Stackdriver, self).__init__(stream)
        self.name = name
        self.client = client
        self.transport = transport(client, name)
        self.resource = resource
        self.labels = labels

    def emit(self, record):
        """Actually log the specified logging record.

        Overrides the default emit behavior of ``StreamHandler``.

        See https://docs.python.org/2/library/logging.html#handler-objects

        :type record: :class:`logging.LogRecord`
        :param record: The record to be logged.
        """
        message = super(Stackdriver, self).format(record)
        self.transport.send(record, message, resource=self.resource, labels=self.labels)

Now I am able to do the following in my logging YAML.

---
version: 1
disable_existing_loggers: False
formatters:
  simple:
    format: "%(asctime)s - %(name)s - %(levelname)s - %(message)s"
  colored:
    (): "colorlog.ColoredFormatter"
    datefmt: "%Y-%m-%d %H:%M:%S"
    format: "%(white)s%(asctime)s.%(msecs)03d%(reset)s - %(cyan)s[%(module)s.%(funcName)s]%(reset)s - %(log_color)s[%(levelname)s] :=>%(reset)s %(message)s"
    log_colors:
      DEBUG: purple
      INFO: blue
      WARNING: yellow
      ERROR: red
      CRITICAL: red,bg_white
handlers:
  console:
    class: logging.StreamHandler
    level: DEBUG
    formatter: colored
    stream: ext://sys.stdout
  stackdriver:
    (): myapp.handler.Stackdriver
    name: my-custom-stackdriver-log-name
root:
  level: INFO
  handlers: [console, stackdriver]