Prometheus Alertmanager: Advanced Routing and Silencing

Introduction

Alertmanager is the component in the Prometheus ecosystem that handles alert routing, grouping, silencing, and notification delivery. Without proper Alertmanager configuration, you'll either get alert storms (thousands of redundant notifications) or miss critical alerts entirely. This guide teaches you how to design robust alert routing rules for real-world infrastructure.

Understanding the Alert Pipeline

When Prometheus evaluates an alerting rule and finds it firing, it sends the alert to Alertmanager. Alertmanager then:

Groups similar alerts (prevent alert storms)
Routes the alert to the right receiver (who gets notified?)
Deduplicates alerts that arrive multiple times
Silences alerts during maintenance windows
Inhibits alerts when higher-priority alerts are firing

Basic Alertmanager Configuration

YAML

# /etc/alertmanager/alertmanager.yml
global:
  resolve_timeout: 5m
  smtp_smarthost: 'smtp.gmail.com:587'
  smtp_from: 'alerts@company.com'
  smtp_auth_username: 'alerts@company.com'
  smtp_auth_password: 'app-password-here'
  slack_api_url: 'https://hooks.slack.com/services/YOUR/SLACK/WEBHOOK'

route:
  # Default receiver if no route matches
  receiver: 'team-ops-email'

  # Group alerts by these labels
  group_by: ['alertname', 'cluster', 'service']

  # Wait this long before sending first notification (collect related alerts)
  group_wait: 30s

  # Wait this long before sending next notification for same group
  group_interval: 5m

  # Wait this long before re-sending a resolved/continuing alert
  repeat_interval: 4h

  routes:
    # Critical alerts go to PagerDuty immediately
    - match:
        severity: critical
      receiver: 'pagerduty-critical'
      group_wait: 10s
      repeat_interval: 1h
      continue: true   # Also send to other matching routes

    # Database alerts go to DBA team
    - match_re:
        alertname: '(MySQL|Postgres|Redis).*'
      receiver: 'team-dba-slack'

    # Security alerts route to security team
    - match:
        team: security
      receiver: 'team-security-email'
      group_by: ['alertname']

receivers:
  - name: 'team-ops-email'
    email_configs:
      - to: 'ops-team@company.com'
        subject: '[{{ .Status | toUpper }}] {{ .GroupLabels.alertname }}'
        html: '{{ template "email.default.html" . }}'

  - name: 'pagerduty-critical'
    pagerduty_configs:
      - routing_key: 'YOUR_PAGERDUTY_INTEGRATION_KEY'
        description: '{{ .GroupLabels.alertname }}: {{ .CommonAnnotations.summary }}'

  - name: 'team-dba-slack'
    slack_configs:
      - channel: '#database-alerts'
        title: '{{ .Status | toUpper }}: {{ .GroupLabels.alertname }}'
        text: '{{ range .Alerts }}{{ .Annotations.description }}{{ end }}'
        color: '{{ if eq .Status "firing" }}danger{{ else }}good{{ end }}'

inhibit_rules:
  # If a node is down, suppress all other alerts from that node
  - source_match:
      alertname: NodeDown
    target_match_re:
      alertname: '.*'
    equal: ['instance']

Advanced Routing with Multiple Conditions

YAML

route:
  receiver: 'blackhole'  # Default: drop (alerts must match a route)
  routes:
    # Business hours vs after-hours routing
    - match:
        severity: warning
      routes:
        - match:
            # Only during business hours (complex: use time_intervals)
            business_hours: 'true'
          receiver: 'slack-warnings'
        - receiver: 'pagerduty-warnings'  # After hours: page someone

    # Multi-condition matching
    - match:
        severity: critical
        environment: production
      receiver: 'pagerduty-prod-critical'

    - match:
        severity: critical
        environment: staging
      receiver: 'slack-staging-critical'

    # Namespace-based routing in Kubernetes
    - match_re:
        namespace: 'kube-system|monitoring'
      receiver: 'team-platform-slack'

    - match_re:
        namespace: 'app-.*'
      receiver: 'team-dev-slack'

Time-Based Routing (Business Hours)

YAML

# alertmanager.yml with time intervals
time_intervals:
  - name: business-hours
    time_intervals:
      - times:
          - start_time: '08:00'
            end_time: '18:00'
        weekdays: ['monday:friday']
        location: 'America/New_York'

  - name: weekends
    time_intervals:
      - weekdays: ['saturday', 'sunday']

route:
  routes:
    - match:
        severity: warning
      routes:
        - active_time_intervals:
            - business-hours
          receiver: 'slack-channel'
        - mute_time_intervals:
            - business-hours
          receiver: 'pagerduty'  # Only page outside business hours

Creating Effective Alert Templates

YAML

# templates/slack.tmpl
{{ define "slack.title" }}
[{{ .Status | toUpper }}{{ if eq .Status "firing" }} :fire:{{ else }} :white_check_mark:{{ end }}]
{{ .GroupLabels.alertname }} ({{ .Alerts | len }} alert{{ if gt (len .Alerts) 1 }}s{{ end }})
{{ end }}

{{ define "slack.text" }}
{{ range .Alerts }}
*Alert:* {{ .Annotations.summary }}
*Severity:* {{ .Labels.severity }}
*Instance:* {{ .Labels.instance }}
*Description:* {{ .Annotations.description }}
*Started:* {{ .StartsAt | since }}
---
{{ end }}
{{ end }}

YAML

# Reference template in receiver config
receivers:
  - name: 'slack-ops'
    slack_configs:
      - channel: '#ops-alerts'
        title: '{{ template "slack.title" . }}'
        text: '{{ template "slack.text" . }}'

Silencing Alerts During Maintenance

BASH

# Using amtool CLI to create silences
amtool --alertmanager.url=http://localhost:9093 silence add   --author="ops-engineer"   --comment="Planned maintenance window"   --duration=2h   alertname="NodeDown" instance="web-01.company.com"

# Silence by regex
amtool silence add   --author="jenkins-deploy"   --comment="Deployment in progress"   --duration=30m   'namespace=~"production"' 'severity=~"warning|critical"'

# List active silences
amtool silence query

# Expire a silence early
amtool silence expire <silence-id>

Inhibition Rules Deep Dive

YAML

inhibit_rules:
  # If the whole cluster is down, suppress individual node alerts
  - source_match:
      alertname: ClusterDown
    target_match_re:
      alertname: '(NodeDown|PodCrashLooping|ServiceUnavailable)'
    equal: ['cluster']

  # If disk is full, suppress high disk usage warnings (redundant)
  - source_match:
      alertname: DiskFull
    target_match:
      alertname: DiskUsageHigh
    equal: ['instance', 'mountpoint']

  # If a node's NIC is down, suppress high latency alerts
  - source_match:
      alertname: NetworkInterfaceDown
    target_match_re:
      alertname: 'HighLatency.*'
    equal: ['instance']

Testing Your Configuration

BASH

# Validate config file syntax
alertmanager --config.file=/etc/alertmanager/alertmanager.yml --check-config

# Test with amtool
amtool --alertmanager.url=http://localhost:9093 config routes test   --verify.receivers=pagerduty-critical   severity=critical environment=production

# Send a test alert via API
curl -X POST http://localhost:9093/api/v2/alerts   -H 'Content-Type: application/json'   -d '[{
    "labels": {
      "alertname": "TestAlert",
      "severity": "critical",
      "environment": "staging"
    },
    "annotations": {
      "summary": "Test alert for routing verification",
      "description": "This is a test"
    },
    "generatorURL": "http://localhost:9090"
  }]'

High Availability Alertmanager Setup

YAML

# docker-compose for HA Alertmanager cluster
version: '3.8'
services:
  alertmanager-1:
    image: prom/alertmanager:latest
    command:
      - '--config.file=/etc/alertmanager/alertmanager.yml'
      - '--cluster.listen-address=0.0.0.0:9094'
      - '--cluster.peer=alertmanager-2:9094'
      - '--cluster.peer=alertmanager-3:9094'
    ports: ["9093:9093", "9094:9094"]
    volumes:
      - ./alertmanager.yml:/etc/alertmanager/alertmanager.yml

  alertmanager-2:
    image: prom/alertmanager:latest
    command:
      - '--config.file=/etc/alertmanager/alertmanager.yml'
      - '--cluster.listen-address=0.0.0.0:9094'
      - '--cluster.peer=alertmanager-1:9094'
      - '--cluster.peer=alertmanager-3:9094'
    ports: ["9095:9093", "9096:9094"]
    volumes:
      - ./alertmanager.yml:/etc/alertmanager/alertmanager.yml

YAML

# Prometheus should point to all Alertmanager instances
# prometheus.yml
alerting:
  alertmanagers:
    - static_configs:
        - targets:
          - 'alertmanager-1:9093'
          - 'alertmanager-2:9093'
          - 'alertmanager-3:9093'

Proper Alertmanager configuration is the difference between an oncall team that responds to real incidents and one that is drowning in noise. Start with simple routing, add inhibition to reduce redundancy, and use time-based routing to respect your team's working hours.