SaltStack for Infrastructure Automation and Configuration

Introduction

SaltStack (Salt) is a powerful infrastructure automation and configuration management tool that excels at managing thousands of servers simultaneously. Unlike Ansible's push model, Salt uses a master-minion architecture with a persistent, encrypted ZeroMQ message bus that enables real-time remote execution. This guide teaches you Salt from the basics to production-ready state management.

Salt Architecture

TEXT

┌──────────────────────────────────────────────────────┐
│                   Salt Master                         │
│  ┌─────────────┐  ┌──────────────┐  ┌─────────────┐  │
│  │  State Tree │  │  Pillar Data │  │  Event Bus  │  │
│  │  (/srv/salt)│  │ (secrets/cfg)│  │  (ZeroMQ)   │  │
│  └─────────────┘  └──────────────┘  └─────────────┘  │
└───────────────────────┬──────────────────────────────┘
                        │ Port 4505 (publish)
                        │ Port 4506 (return)
          ┌─────────────┼─────────────┐
          │             │             │
   ┌──────▼─────┐ ┌─────▼──────┐ ┌──▼──────────┐
   │ Salt Minion│ │ Salt Minion│ │ Salt Minion │
   │ web-01     │ │ db-01      │ │ app-01      │
   └────────────┘ └────────────┘ └─────────────┘

Installation

BASH

# Install Salt Master (on the control node)
curl -fsSL https://bootstrap.saltproject.io -o install_salt.sh
sudo sh install_salt.sh -M stable  # -M = install master

# Install Salt Minion (on managed nodes)
# Run on each server being managed:
curl -fsSL https://bootstrap.saltproject.io | sudo sh -s -- stable

# Configure minion to point to master
sudo nano /etc/salt/minion

YAML

# /etc/salt/minion
master: 192.168.1.10    # Salt master IP or hostname
id: web-01              # Unique minion ID (defaults to hostname)

BASH

# Start services
sudo systemctl enable salt-master salt-minion
sudo systemctl start salt-master salt-minion

# On master: accept minion keys
salt-key -L              # List pending/accepted/rejected keys
salt-key -A              # Accept all pending keys
salt-key -a web-01       # Accept specific minion

Running Remote Execution Commands

BASH

# Test connectivity to all minions
salt '*' test.ping

# Run on specific minion
salt 'web-01' test.ping

# Target by grain (OS, role, etc.)
salt -G 'os:Ubuntu' cmd.run 'uname -a'

# Target by regex
salt -E 'web-0[1-3]' cmd.run 'nginx -t'

# Target multiple minions with list
salt -L 'web-01,web-02,db-01' cmd.run 'systemctl status nginx'

# Parallel execution (default is parallel)
salt '*' cmd.run 'apt-get update' --timeout=120

# Get disk space from all servers
salt '*' disk.usage

# Collect CPU info
salt '*' grains.item cpu_model num_cpus

# Install a package on all web servers
salt -G 'role:webserver' pkg.install nginx

Salt Grains: Node Metadata

Grains are metadata about each minion — OS, CPU, memory, roles, etc.

BASH

# List all grains for a minion
salt 'web-01' grains.ls

# Get specific grain
salt 'web-01' grains.get os
# Returns: Ubuntu

salt 'web-01' grains.get mem_total
# Returns: 16384 (MB)

# Set custom grain
salt 'web-01' grains.setval role webserver
salt 'web-01' grains.setval environment production

# Now target by custom grain
salt -G 'role:webserver' cmd.run 'nginx -s reload'

Custom grains file on minion:

YAML

# /etc/salt/grains (on minion)
role: webserver
environment: production
datacenter: us-east-1
team: platform

Writing Salt States (Configuration Management)

States describe the desired configuration of your systems:

YAML

# /srv/salt/nginx/init.sls
# Install and configure nginx

nginx_package:
  pkg.installed:
    - name: nginx
    - version: latest

nginx_service:
  service.running:
    - name: nginx
    - enable: True
    - reload: True
    - require:
        - pkg: nginx_package
    - watch:
        - file: nginx_config

nginx_config:
  file.managed:
    - name: /etc/nginx/nginx.conf
    - source: salt://nginx/files/nginx.conf
    - template: jinja
    - user: root
    - group: root
    - mode: 644

# Apply the state
# salt 'web-01' state.apply nginx

Jinja templating in state files:

YAML

# /srv/salt/nginx/files/nginx.conf
worker_processes {{ grains['num_cpus'] }};

events {
    worker_connections {{ pillar.get('nginx:worker_connections', 1024) }};
}

server {
    listen 80;
    server_name {{ grains['fqdn'] }};

    {% if pillar.get('ssl_enabled', False) %}
    listen 443 ssl;
    ssl_certificate /etc/ssl/certs/{{ grains['id'] }}.crt;
    {% endif %}
}

Pillar: Secure Configuration Data

Pillar stores sensitive data (passwords, API keys) that is only sent to targeted minions:

YAML

# /srv/pillar/top.sls
base:
  '*':
    - common
  'role:webserver':
    - match: grain
    - webserver
  'role:database':
    - match: grain
    - database

# /srv/pillar/database.sls
postgres:
  password: "super-secret-password"
  max_connections: 200
  shared_buffers: "4GB"

backup:
  s3_bucket: "company-db-backups"
  aws_key: "AKIAIOSFODNN7EXAMPLE"
  aws_secret: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"

BASH

# Verify pillar data for a minion
salt 'db-01' pillar.get postgres

# Use in state file:
# {{ pillar['postgres']['password'] }}

Top File: Targeting States to Minions

YAML

# /srv/salt/top.sls
base:
  # Apply to all minions
  '*':
    - common.users
    - common.security
    - common.monitoring

  # Apply to web servers (by grain)
  'role:webserver':
    - match: grain
    - nginx
    - ssl

  # Apply to database servers
  'role:database':
    - match: grain
    - postgres
    - backup

  # Apply to production only
  'environment:production':
    - match: grain
    - monitoring.pagerduty

# Apply all states from top.sls
# salt '*' state.highstate

Salt Reactor: Event-Driven Automation

YAML

# /etc/salt/master.d/reactor.conf
reactor:
  - 'salt/minion/*/start':
    - /srv/reactor/new_minion.sls

  - 'salt/cloud/*/created':
    - /srv/reactor/cloud_provision.sls

# /srv/reactor/new_minion.sls
# When a new minion connects, apply base states
{% if data['id'].startswith('web-') %}
apply_web_states:
  local.state.apply:
    - tgt: {{ data['id'] }}
    - arg:
      - nginx
      - ssl
{% endif %}

Production Deployment Pattern

BASH

#!/bin/bash
# deploy.sh - Deploy application with Salt

# 1. Target specific minions
TARGETS="web-0[1-3]"

# 2. Pull latest code
salt "$TARGETS" git.pull /opt/app origin master

# 3. Install dependencies
salt "$TARGETS" cmd.run "cd /opt/app && pip install -r requirements.txt"

# 4. Run database migrations (only on one server)
salt "web-01" cmd.run "cd /opt/app && python manage.py migrate"

# 5. Restart application
salt "$TARGETS" service.restart gunicorn

# 6. Verify deployment
salt "$TARGETS" cmd.run "curl -s http://localhost/health | python3 -m json.tool"

# 7. Check all services are running
salt "$TARGETS" service.status nginx gunicorn

Troubleshooting Salt

BASH

# Check minion connectivity
salt-run manage.status    # Shows connected vs disconnected minions
salt-run manage.down      # Shows only disconnected minions

# Debug minion connection
salt-minion -l debug      # Run minion in foreground with debug logging

# Test state without applying
salt 'web-01' state.apply nginx test=True

# Check master event bus
salt-run state.event tagmatch='salt/*' full=True

# Sync custom modules/grains
salt '*' saltutil.sync_all

# Clear minion cache
salt 'web-01' saltutil.clear_cache

SaltStack's real power emerges when you combine remote execution with state management and the reactor system — you get a platform that can self-heal your infrastructure by reacting to events and enforcing desired state automatically.