Search code examples
google-cloud-platformgoogle-compute-enginewebhooksshutdowngoogle-cloud-pubsub

How to generate webhook when GCE instance shuts down?

I have a system that adds dynamically created instances with ephemeral IP addresses to GCP cloud DNS when they start. However, I need to remove them from the DNS when they shutdown, whether through GCP's own APIs, or through some other method.

The best I can figure now is:

  1. Poll the GCE API
    • Pros: Foolproof; pretty much guaranteed to work; doesn't require an endpoint for a webhook
    • Cons: Resource scaling issues; could easily hit API request quotas if not written intelligently
  2. Use API monitoring through Stackdriver/StackMonitor and create a webhook notification
    • Pros: Only run code when needed; simple webhook implementation; no polling; no false positives
    • Cons: Can't monitor for unexpected shutdowns (i.e. not an API call).
  3. Use uptime monitoring through Stackdriver/StackMonitor
    • Pros: Works for unexpected shutdowns
    • Cons: Stackdriver will track many other unused metrics; requires at least 5 minutes before uptime notification is sent; based on CPU/Memory stats, which could mean that a machine is simply frozen or the logging executable has crashed and not that the machine is truly shutdown.

Is there another way to do this?

Solution

  • There is two ways that I know of to do this. Both still require the Stackdriver Logging service, but it shouldn't require logging extraneous data like uptime monitoring would do and it should capture every GCE instance shutdown that happens in a given GCP project, no matter how it happens. I personally prefer the second method (Pub/Sub).

    The first method is through the Stackdriver service. The steps below assume you have already setup a Stackdriver account and connect it to the project you want to monitor.

    Stackdriver method

    1. Go to the Stackmonitor viewer for your desired project

    2. On the far right of the text field for the filter, select the advanced mode and enter the following (where my-project is the name of your project):

      resource.type="gce_instance" logName="projects/my- project/logs/compute.googleapis.com%2Factivity_log" (jsonPayload.event_subtype:"compute.instances.stop" OR jsonPayload.event_subtype:"compute.instances.guestTerminate") jsonPayload.event_type:"GCE_OPERATION_DONE"

    3. At the top of the page click Create Metric, then give it a name and a description, and submit your new metric

    4. Before proceeding, be sure to create your webhook in the Stackdriver notification settings. A good testbed is http://bin.mailgun.net/ (but keep in mind that it is publicly accessible, so you shouldn't post anything sensitive and/or you should delete the pastebin immediately after you are done).
    5. Now, head over to the Log-based Metrics page. Under User-defined Metrics you should now see your newly created metric
    6. Click on the dropdown menu on the far right, select Create alert from metric. This should take you to the Stackdriver alert panel with most of the proper information filled in
    7. Change the THRESHOLD value to 0, and the FOR value to most recent value. Then click Save Condition.
    8. Choose your notifications you want (which should at least include your webhook)
    9. Name and save your alert policy
    10. Start and stop a machine to test it out

    The downside for the above method is that you need to pay for a Stackdriver premium account to use features like webhooks. It is up to you whether the pricing on that is worth it. On the flip side, the method below (using GCP Pub/Sub) costs nothing from Stackdriver and works with both exempt and non-exempt logs. You do, of course, still need to pay any Pub/Sub charges you incur from using the service.

    Pub/Sub method

    The second method is similar, but doesn't necessarily require a premium Stackdriver account (you should be fine to do all this on the Free Stackdriver tier). In this method, you use Google Pub/Sub. Much of the relevant documentation can be found here.

    1. Go to the Stackmonitor viewer for your desired project

    2. On the far right of the text field for the filter, select the advanced mode and enter the following (where my-project is the name of your project):

      resource.type="gce_instance" logName="projects/my- project/logs/compute.googleapis.com%2Factivity_log" (jsonPayload.event_subtype:"compute.instances.stop" OR jsonPayload.event_subtype:"compute.instances.guestTerminate") jsonPayload.event_type:"GCE_OPERATION_DONE"

    3. At the top of the page click Create Export, give it a sink name, choose Cloud Pub/Sub as the sink service, then choose or create a Pub/Sub topic

    4. Look at the Google Pub/Sub documentation for push subscriptions. Pub/Sub alternatively allows for polling (i.e. pull subscriptions) and batching messages.

    I personally prefer the Pub/Sub method. It seems more fit for purpose and is (at least theoretically) less expensive than the Stackdriver method.

    There may be a way to do this all programmatically through the GCP APIs, but I haven't dug in that deeply. If I find any documentation for that, then I will update this answer with that information as well.