Linux Monitoring with Prometheus Node Exporter

Introduction

Prometheus is a pull-based monitoring system that scrapes metrics from targets at defined intervals. Node Exporter exposes Linux system metrics (CPU, memory, disk, network) in Prometheus format. Together they form the foundation of modern infrastructure monitoring.

Architecture

TEXT

[Linux Servers] 
   └── [Node Exporter :9100] ← Prometheus scrapes every 15s
   
[Applications]
   └── [Custom Metrics :8080/metrics]
   
[Prometheus :9090] ← Stores time series data
   └── [Grafana :3000] ← Visualizes data
   └── [Alertmanager :9093] ← Sends alerts

Installing Node Exporter

BASH

# Download latest release
wget https://github.com/prometheus/node_exporter/releases/download/v1.7.0/node_exporter-1.7.0.linux-amd64.tar.gz
tar xzf node_exporter-1.7.0.linux-amd64.tar.gz
cp node_exporter-1.7.0.linux-amd64/node_exporter /usr/local/bin/

# Create system user
useradd --no-create-home --shell /bin/false node_exporter

# Create systemd service
cat > /etc/systemd/system/node_exporter.service << EOF
[Unit]
Description=Node Exporter
After=network.target

[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter   --collector.filesystem.mount-points-exclude="^/(sys|proc|dev|run)($|/)"   --collector.systemd   --collector.processes

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl enable node_exporter
systemctl start node_exporter

# Verify
curl -s http://localhost:9100/metrics | head -30

Installing Prometheus

BASH

# Create user
useradd --no-create-home --shell /bin/false prometheus

# Create directories
mkdir -p /etc/prometheus /var/lib/prometheus

# Download
wget https://github.com/prometheus/prometheus/releases/download/v2.49.0/prometheus-2.49.0.linux-amd64.tar.gz
tar xzf prometheus-2.49.0.linux-amd64.tar.gz
cp prometheus-2.49.0.linux-amd64/prometheus /usr/local/bin/
cp prometheus-2.49.0.linux-amd64/promtool /usr/local/bin/

chown prometheus:prometheus /etc/prometheus /var/lib/prometheus

Prometheus Configuration

Create /etc/prometheus/prometheus.yml:

YAML

global:
  scrape_interval: 15s
  evaluation_interval: 15s

alerting:
  alertmanagers:
    - static_configs:
        - targets: ['localhost:9093']

rule_files:
  - "/etc/prometheus/rules/*.yml"

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'node'
    static_configs:
      - targets:
          - 'web-server-1:9100'
          - 'web-server-2:9100'
          - 'db-server-1:9100'
        labels:
          env: production
    
  - job_name: 'node-dev'
    static_configs:
      - targets: ['dev-server-1:9100']
        labels:
          env: development

BASH

# Create systemd service
cat > /etc/systemd/system/prometheus.service << EOF
[Unit]
Description=Prometheus
After=network.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus   --config.file=/etc/prometheus/prometheus.yml   --storage.tsdb.path=/var/lib/prometheus   --storage.tsdb.retention.time=30d   --web.listen-address=:9090

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl enable prometheus
systemctl start prometheus

Alert Rules

Create /etc/prometheus/rules/infrastructure.yml:

YAML

groups:
  - name: infrastructure
    rules:
      - alert: HighCPULoad
        expr: 100 - (avg by(instance) (irate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 85
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High CPU on {{ $labels.instance }}"
          description: "CPU usage is {{ $value | printf '%.2f' }}%"

      - alert: LowDiskSpace
        expr: (node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"}) * 100 < 15
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Low disk on {{ $labels.instance }}"
          description: "Only {{ $value | printf '%.2f' }}% disk space remaining"

      - alert: HostDown
        expr: up == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Host {{ $labels.instance }} is down"

      - alert: HighMemoryUsage
        expr: (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100 > 90
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High memory on {{ $labels.instance }}: {{ $value | printf '%.2f' }}%"

Useful PromQL Queries

PROMQL

# CPU usage per host (percentage)
100 - (avg by(instance) (irate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

# Memory usage percentage
(1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100

# Disk usage percentage
(1 - node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"}) * 100

# Network in/out (bytes per second)
irate(node_network_receive_bytes_total{device="eth0"}[5m])
irate(node_network_transmit_bytes_total{device="eth0"}[5m])

# Disk I/O
irate(node_disk_read_bytes_total{device="sda"}[5m])
irate(node_disk_written_bytes_total{device="sda"}[5m])

# Load average vs CPU count
node_load1 / count by(instance) (node_cpu_seconds_total{mode="idle"})

Service Discovery (for Dynamic Environments)

Instead of static targets, use file-based service discovery:

YAML

# prometheus.yml
- job_name: 'dynamic-nodes'
  file_sd_configs:
    - files: ['/etc/prometheus/targets/*.json']
      refresh_interval: 30s

Create target files dynamically:

BASH

cat > /etc/prometheus/targets/web-servers.json << EOF
[
  {
    "targets": ["web-1:9100", "web-2:9100", "web-3:9100"],
    "labels": {"env": "production", "role": "webserver"}
  }
]
EOF