Logging on either GCP or local

Question

Suppose there is a system that is run on GCP, but as a backup, can be run locally.

When running on the cloud, stackdriver is pretty straightforward.

However, I need my system to push to stackdriver if on the cloud, and if not on the cloud, use the local python logger.

I also don't want to include any logic to do so, and this should be automatic.

1. When logging, log straight to Python/local logger.
2. If on GCP -> push these to stackdriver.

I can write logic that could implement this but that is bad practice. There surely is a direct way of getting this to work.

Example

import google.cloud.logging

client = google.cloud.logging.Client()
client.setup_logging()


import logging
cl = logging.getLogger()
file_handler = logging.FileHandler('file.log')
cl.addHandler(file_handler)
logging.info("INFO!")

This will basically log to python logger, and then 'always' upload to cloud logger. How can I have it so that I don't need to explicitly add import google.cloud.logging and basically if stackdriver is installed, it directly gets the logs? Is that even possible? If not can someone explain how this would be handled from a best practices perspective?

Attempt 1 [works]

Created /etc/google-fluentd/config.d/workflow_log.conf

<source>
    @type tail
    format none
    path /home/daudn/this_log.log
    pos_file /var/lib/google-fluentd/pos/this_log.pos
    read_from_head true
    tag workflow-log
</source>

Created /var/log/this_log.log

pos_file /var/lib/google-fluentd/pos/this_log.pos exists

import logging
cl = logging.getLogger()
file_handler = logging.FileHandler('/var/log/this_log.log')
file_handler.setFormatter(logging.Formatter("%(asctime)s;%(levelname)s;%(message)s"))
cl.addHandler(file_handler)

logging.info("info_log")
logging.error("error_log")

This works! Look for your logs for the specific VM and not global>python

Answer 1

You do not need to specifically enable or use Stackdriver in your program. You can use the Python logger and write to any file you want. However, Stackdriver only logs specific log files. This means that you would need to manually set up Stackdriver to log "your" log files.

In your example, you are writing to file.log. Modify /etc/google-fluentd/config.d/mylogfile.conf to include the following. You will need to specify the full path for file.log and not just the file name. In this example, I named it /var/log/mylogfile.log. This example also assumes that your logs start each line with a date.

<source>
  @type tail

  # Parse the timestamp, but still collect the entire line as 'message'
  format /^(?<message>(?<time>[^ ]*\s*[^ ]* [^ ]*) .*)$/

  path /var/log/mylogfile.log
  pos_file /var/lib/google-fluentd/pos/mylogfile.log.pos
  read_from_head true
  tag auth
</source>

For more information read the following document:

Stackdriver - Configuring the Agent

Now your program will run outside GCP and when running on a configured instance, log to Stackdriver.

Note: I would do the opposite of what you have asked. I would always use Stackdriver. When not running in GCP I would manually set up Stackdriver on my desktop, local server, etc and continue to log to Stackdriver.