Prometheus Alert rule not showing up

I created alert rule file.yml and copy it under "/" of prometheus contianer and I added it to the values.yml file but I can't see the rule in the UI

    prometheus.yml:
    rule_files:
      - /etc/config/recording_rules.yml
      - /etc/config/alerting_rules.yml
      - /custom_alerting_rules.yml
    ## Below two files are DEPRECATED will be removed from this default values file
      - /etc/config/rules
      - /etc/config/alerts

    alerting:
      alertmanagers:
        - static_configs:
            - targets: ['alertmanager:9093'] here i tried the @IP of alert manager service

here is the alert file

groups:
  - name: my-custom-alerts
    rules:
      - alert: HighPodCount
        expr: count(kube_pod_info{pod=~"consumer.*"}) > 2
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: High pod count
          description: The number of pods is above the threshold.

k get svc shows

prometheus-alertmanager               ClusterIP      10.10x.21x.x8    <none>        9093/TCP                       68m

what am doing wrong ?

Solution

From the perspective of Prometheus server, its configuration and rule files are not auto-reloaded. You need to call the Reload management api manually.

There are 3 ways for this purpose:

If you're in the prometheus contianer:
```
curl -XPOST http://:::9090/-/reload
```

If you're in another pod container of the same namespace:

# prometheus pod name: prometheus-0
curl -XPOST http://prometheus-0:9090/-/reload

If you're out of K8s cluster(i.e. where you run kubectl command):

# prometheus pod name: prometheus-0, namespace: monitoring
kubectl port-forward prometheus-0 -nmonitoring 9090:9090
# in another shell
curl -XPOST http://127.0.0.1:9090/-/reload