Search code examples
goprometheusmetrics

Golang prometheus: is there a way to measure metrics for outbound requests?

I have a service A where I am scrapping metrics using prometheus, A makes some outbound requests to another service B. Is there a way to get metrics for requets from A to B?

Like, I want a measure count and duration of requets from A -> B

Solution

  • The metrics can be record by using prometheus.NewCounterVec and prometheus.NewHistogramVec (or proemetheus.NewSummaryVec, depends on the use case).

    We could use custom labels to identify who is the client and who is the server. By this way, can filter out a client's request count and duration metrics to a particular server.

    Here is the sample snippet

    client.go

    package main

import (
    "net/http"
    "time"

    "github.com/labstack/echo/v4"
    "github.com/prometheus/client_golang/prometheus"
    "github.com/prometheus/client_golang/prometheus/promhttp"
)

var (
    requestCounter = prometheus.NewCounterVec(
        prometheus.CounterOpts{
            Name: "client_request_count",
            Help: "Total number of requests from client",
        },
        []string{"client", "server", "method", "route", "status"},
    )

    durationHistorgram = prometheus.NewHistogramVec(
        prometheus.HistogramOpts{
            Name:    "client_request_duration_secs",
            Help:    "Duration of requests from Client",
            Buckets: []float64{0.1, 0.5, 1, 2, 5},
        },
        []string{"client", "server", "method", "route", "status"},
    )
)

func init() {
    prometheus.MustRegister(requestCounter)
    prometheus.MustRegister(durationHistorgram)
}

func main() {

    e := echo.New()
    e.GET("/", func(c echo.Context) error {

        // the request start time
        start := time.Now()
        req, err := http.NewRequest("GET", "http://localhost:8002", nil)
        if err != nil {
            return c.JSON(http.StatusInternalServerError, err.Error())
        }

        resp, err := http.DefaultClient.Do(req)
        if err != nil {
            return c.JSON(http.StatusInternalServerError, err.Error())
        }
        defer resp.Body.Close()

        labels := prometheus.Labels{
            "client": "Server A", // defines the client server
            "server": "Server B", // defines the outbound request server
            "method": "GET",// HTTP method
            "route":  "/", // Request route
            "status": resp.Status,// Response status
        }
        duration := time.Since(start).Seconds()

        // the duration
        durationHistorgram.With(labels).Observe(duration)

        // request api count
        requestCounter.With(labels).Inc()

        return c.String(http.StatusOK, "[Client -> Server] Request send and received")
    })

    e.GET("/metrics", echo.WrapHandler(promhttp.Handler()))

    if err := e.Start(":8001"); err != nil {
        panic(err)
    }
}

    server.go

    package main

import (
    "net/http"

    "github.com/labstack/echo/v4"
)

func main() {
    e := echo.New()

    e.GET("/", func(c echo.Context) error {
        return c.JSON(http.StatusOK, map[string]interface{}{
            "message": "response sending from server",
        })
    })

    if err := e.Start(":8002"); err != nil {
        panic(err)
    }
}

    As you can see in the code, these are our custom labels.

     []string{"client", "server", "method", "route", "status"},

    and these are the label values

    labels := prometheus.Labels{
            "client": "Server A", // defines the client server
            "server": "Server B", // defines the outbound request server
            "method": "GET", // HTTP method
            "route":  "/", // Request route
            "status": resp.Status,// Response status
        }

    and finally we can query

    client_request_count {client="Server A", server="Server B"}

    The request Server A send to the Server B

    image

    References