Custom metric is not exposed in prometheus

Question

I'm writing a Go application and I need to log some custom metrics using Prometheus. I have a local instance of Prometheus and this is my prometheus.yml file:

scrape_configs:
- job_name: myapp
  scrape_interval: 10s
  static_configs:
  - targets:
    - localhost:2112

And this is my Go code:

package main

import (
    "github.com/prometheus/client_golang/prometheus"
    "github.com/prometheus/client_golang/prometheus/promauto"
    "github.com/prometheus/client_golang/prometheus/promhttp"

    "net/http"
    "time"
)

func recordMetrics() {
    go func() {
        for {
            opsProcessed.Inc()
            time.Sleep(2 * time.Second)

        }
    }()
}

var (
    opsProcessed = promauto.NewCounter(prometheus.CounterOpts{
        Name: "myapp_processed_ops_total",
        Help: "The total number of processed events",
    })
)

func main() {
    recordMetrics()

    http.Handle("/metrics", promhttp.Handler())
    http.ListenAndServe(":2112", nil)
}

As you see, I have defined a custom metric called opsProcessed which its name is myapp_processed_ops_total. I can see myapp_processed_ops_total at http://localhost:2112/metrics. However, I can not see this metric at my Prometheus instance.

What is the problem? I think my server is scraped since I can see other metrics like scrape_duration_seconds in Prometheus:

Maybe The problem is from my docker-compose file for prometheus. This is the target page in prometheus UI:

And this is my docker-compose file:

version: '2.1'

networks:
  monitor-net:
    driver: bridge

volumes:
    prometheus_data: {}
    grafana_data: {}

services:

  prometheus:
    image: prom/prometheus:v2.15.2
    container_name: prometheus1
    volumes:
      - ./prometheus:/etc/prometheus
      - prometheus_data:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
      - '--web.console.libraries=/etc/prometheus/console_libraries'
      - '--web.console.templates=/etc/prometheus/consoles'
      - '--storage.tsdb.retention.time=200h'
      - '--web.enable-lifecycle'
    restart: unless-stopped
    expose:
      - 9090
    ports:
      - "9090:9090"
    networks:
      - monitor-net
    labels:
      org.label-schema.group: "monitoring"

  grafana:
    image: grafana/grafana:6.5.3
    container_name: grafana1
    volumes:
      - grafana_data:/var/lib/grafana
      - ./grafana/provisioning:/etc/grafana/provisioning
    environment:
      - GF_SECURITY_ADMIN_USER=${ADMIN_USER:-admin}
      - GF_SECURITY_ADMIN_PASSWORD=${ADMIN_PASSWORD:-admin}
      - GF_USERS_ALLOW_SIGN_UP=false
    restart: unless-stopped
    ports:
      - "3000:3000"
    networks:
      - monitor-net
    labels:
      org.label-schema.group: "monitoring"

Answer 1

Based on @Peter and @Henry helpful comments, The answer is this:

The connection from the Prometheus docker container to my locally running application is refused. I found out about it from the Prometheus UI at /targets. The reason is the prometheus.yml config file. If you have set up Prometheus and Grafana using docker(my case), prometheus.yml should be like this:

1- If the running application is also in a container, its service name from the docker-compose file should be used:

scrape_configs:
- job_name: myapp
  scrape_interval: 10s
  static_configs:
  - targets:
    - <service>:2112

2- If the application is running locally on your machine but not in a container:

scrape_configs:
- job_name: myapp
  scrape_interval: 10s
  static_configs:
  - targets:
    - <External_IP>:2112

And find your machine's External IP address from ifconfig.