Doing the equivalent of log_struct in python logger

Question

In the google example, it gives the following:

logger.log_struct({
    'message': 'My second entry',
    'weather': 'partly cloudy',
})

How would I do the equivalent in python's logger. For example:

import logging
log.info(
    msg='My second entry', 
    extra = {'weather': "partly cloudy"}
)

When I view this in stackdriver, the extra fields aren't getting parsed properly:

2018-11-12 15:41:12.366 PST
My second entry

Expand all | Collapse all 

{
 insertId:  "1de1tqqft3x3ri"  
 jsonPayload: {
  message:  "My second entry"   
  python_logger:  "Xhdoo8x"   
 }
 logName:  "projects/Xhdoo8x/logs/python"  
 receiveTimestamp:  "2018-11-12T23:41:12.366883466Z"  
 resource: {…}  
 severity:  "INFO"  
 timestamp:  "2018-11-12T23:41:12.366883466Z"  
}

How would I do that?

The closest I'm able to do now is:

log.handlers[-1].client.logger('').log_struct("...")

But this still requires a second call...

Answer 1

Current solution:

Update 1 - user Seth Nickell improved my proposed solution, so I update this answer as his method is superior. The following is based on his response on GitHub:

https://github.com/snickell/google_structlog

pip install google-structlog

Used like so:

import google_structlog

google_structlog.setup(log_name="here-is-mylilapp")

# Now you can use structlog to get searchable json details in stackdriver...
import structlog
logger = structlog.get_logger()
logger.error("Uhoh, something bad did", moreinfo="it was bad", years_back_luck=5)

# Of course, you can still use plain ol' logging stdlib to get { "message": ... } objects
import logging
logger = logging.getLogger("yoyo")
logger.error("Regular logging calls will work happily too")

# Now you can search stackdriver with the query:
# logName: 'here-is-mylilapp'

---

Original answer:

Based on an answer from this GitHub thread, I use the following bodge to log custom objects as info payload. It derives more from the original _Worker.enqueue and supports passing custom fields.

from google.cloud.logging import _helpers
from google.cloud.logging.handlers.transports.background_thread import _Worker

def my_enqueue(self, record, message, resource=None, labels=None, trace=None, span_id=None):
    queue_entry = {
        "info": {"message": message, "python_logger": record.name},
        "severity": _helpers._normalize_severity(record.levelno),
        "resource": resource,
        "labels": labels,
        "trace": trace,
        "span_id": span_id,
        "timestamp": datetime.datetime.utcfromtimestamp(record.created),
    }

    if 'custom_fields' in record:
        entry['info']['custom_fields'] = record.custom_fields

    self._queue.put_nowait(queue_entry)

_Worker.enqueue = my_enqueue

Then

import logging
from google.cloud import logging as google_logging

logger = logging.getLogger('my_log_client')
logger.addHandler(CloudLoggingHandler(google_logging.Client(), 'my_log_client'))

logger.info('hello', extra={'custom_fields':{'foo': 1, 'bar':{'tzar':3}}})

Resulting in:

Which then makes it much easier to filter according to these custom_fields.

Let's admit this is not good programming, though until this functionality is officially supported there doesn't seem to be much else that can be done.