Self-Host Grafana + Prometheus: Full Metrics Stack 2026

TL;DR

Prometheus (Apache 2.0, ~56K stars, Go) is the industry-standard metrics collection and alerting toolkit. Grafana (AGPL 3.0, ~64K stars, Go + TypeScript) is the visualization layer — beautiful dashboards, alerting, and exploration. Together they form the most deployed self-hosted observability stack. Add Loki for log aggregation and you have a complete replace for DataDog ($15-23/host/month) or New Relic.

Key Takeaways

• Prometheus: Time-series metrics database with a powerful query language (PromQL)
• Grafana: Dashboards, alerting, and data exploration for Prometheus, Loki, and 50+ other sources
• Node Exporter: Collects OS metrics (CPU, RAM, disk, network) from Linux hosts
• cAdvisor: Collects Docker container metrics automatically
• Loki: Log aggregation with the same label-based model as Prometheus (cheaper than Elasticsearch for logs)
• Alertmanager: Routes alerts to Slack, PagerDuty, email with deduplication and silencing

---

The Full Stack

Your Services → Prometheus Exporters → Prometheus (scrape + store)
                                             ↓
                                        Alertmanager → Slack/PagerDuty/Email
                                             ↓
                                          Grafana (visualize + alert)

Your Apps → Promtail → Loki → Grafana (logs)

---

Part 1: Docker Compose — Full Stack

# docker-compose.yml
services:
  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus
    restart: unless-stopped
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - ./prometheus/rules:/etc/prometheus/rules:ro
      - prometheus_data:/prometheus
    command:
      - "--config.file=/etc/prometheus/prometheus.yml"
      - "--storage.tsdb.path=/prometheus"
      - "--storage.tsdb.retention.time=30d"
      - "--web.console.libraries=/usr/share/prometheus/console_libraries"
      - "--web.console.templates=/usr/share/prometheus/consoles"
      - "--web.enable-lifecycle"    # Enable config hot-reload

  grafana:
    image: grafana/grafana:latest
    container_name: grafana
    restart: unless-stopped
    ports:
      - "3000:3000"
    volumes:
      - grafana_data:/var/lib/grafana
      - ./grafana/provisioning:/etc/grafana/provisioning
    environment:
      GF_SECURITY_ADMIN_PASSWORD: "${GRAFANA_PASSWORD}"
      GF_USERS_ALLOW_SIGN_UP: "false"
      GF_SERVER_DOMAIN: "grafana.yourdomain.com"
      GF_SERVER_ROOT_URL: "https://grafana.yourdomain.com"

  node-exporter:
    image: prom/node-exporter:latest
    container_name: node-exporter
    restart: unless-stopped
    pid: host
    network_mode: host
    volumes:
      - /proc:/host/proc:ro
      - /sys:/host/sys:ro
      - /:/rootfs:ro
    command:
      - "--path.procfs=/host/proc"
      - "--path.rootfs=/rootfs"
      - "--path.sysfs=/host/sys"
      - "--collector.filesystem.mount-points-exclude=^/(sys|proc|dev|host|etc)($$|/)"

  cadvisor:
    image: gcr.io/cadvisor/cadvisor:latest
    container_name: cadvisor
    restart: unless-stopped
    ports:
      - "8080:8080"
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:ro
      - /sys:/sys:ro
      - /var/lib/docker/:/var/lib/docker:ro
      - /dev/disk/:/dev/disk:ro
    privileged: true
    devices:
      - /dev/kmsg

  alertmanager:
    image: prom/alertmanager:latest
    container_name: alertmanager
    restart: unless-stopped
    ports:
      - "9093:9093"
    volumes:
      - ./alertmanager/alertmanager.yml:/etc/alertmanager/alertmanager.yml:ro
      - alertmanager_data:/alertmanager

  loki:
    image: grafana/loki:latest
    container_name: loki
    restart: unless-stopped
    ports:
      - "3100:3100"
    volumes:
      - ./loki/loki-config.yml:/etc/loki/local-config.yaml:ro
      - loki_data:/loki

  promtail:
    image: grafana/promtail:latest
    container_name: promtail
    restart: unless-stopped
    volumes:
      - ./promtail/promtail-config.yml:/etc/promtail/config.yml:ro
      - /var/log:/var/log:ro
      - /var/lib/docker/containers:/var/lib/docker/containers:ro

volumes:
  prometheus_data:
  grafana_data:
  alertmanager_data:
  loki_data:

---

Part 2: Prometheus Configuration

# prometheus/prometheus.yml
global:
  scrape_interval: 15s
  evaluation_interval: 15s

alerting:
  alertmanagers:
    - static_configs:
        - targets: ["alertmanager:9093"]

rule_files:
  - "rules/*.yml"

scrape_configs:
  # Prometheus self-monitoring:
  - job_name: "prometheus"
    static_configs:
      - targets: ["localhost:9090"]

  # Linux host metrics:
  - job_name: "node-exporter"
    static_configs:
      - targets: ["node-exporter:9100"]

  # Docker container metrics:
  - job_name: "cadvisor"
    static_configs:
      - targets: ["cadvisor:8080"]

  # Additional hosts (add more servers):
  - job_name: "remote-servers"
    static_configs:
      - targets:
          - "server2.yourdomain.com:9100"
          - "server3.yourdomain.com:9100"
    labels:
      env: production

---

Part 3: Alert Rules

# prometheus/rules/host-alerts.yml
groups:
  - name: host-alerts
    rules:
      - alert: HighCPUUsage
        expr: 100 - (avg by(instance) (irate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 90
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High CPU usage on {{ $labels.instance }}"
          description: "CPU usage is {{ $value | printf \"%.1f\" }}%"

      - alert: LowDiskSpace
        expr: (node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"}) * 100 < 15
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Low disk space on {{ $labels.instance }}"
          description: "Disk is {{ $value | printf \"%.1f\" }}% free"

      - alert: HighMemoryUsage
        expr: (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100 > 90
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High memory usage on {{ $labels.instance }}"

      - alert: ContainerDown
        expr: absent(container_last_seen{name!=""})
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Container {{ $labels.name }} is down"

---

Part 4: Alertmanager Configuration

# alertmanager/alertmanager.yml
global:
  slack_api_url: "https://hooks.slack.com/services/YOUR/WEBHOOK"

route:
  group_by: ["alertname", "instance"]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  receiver: "slack-notifications"
  routes:
    - match:
        severity: critical
      receiver: "pagerduty"

receivers:
  - name: "slack-notifications"
    slack_configs:
      - channel: "#alerts"
        title: "{{ range .Alerts }}{{ .Annotations.summary }}{{ end }}"
        text: "{{ range .Alerts }}{{ .Annotations.description }}{{ end }}"

  - name: "pagerduty"
    pagerduty_configs:
      - service_key: "your-pagerduty-key"

inhibit_rules:
  - source_match:
      severity: critical
    target_match:
      severity: warning
    equal: ["alertname", "instance"]

---

Part 5: Loki Configuration

# loki/loki-config.yml
auth_enabled: false

server:
  http_listen_port: 3100

ingester:
  lifecycler:
    ring:
      kvstore:
        store: inmemory
      replication_factor: 1

schema_config:
  configs:
    - from: "2024-01-01"
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 24h

storage_config:
  boltdb_shipper:
    active_index_directory: /loki/index
    cache_location: /loki/index_cache
  filesystem:
    directory: /loki/chunks

limits_config:
  retention_period: 30d

# promtail/promtail-config.yml
server:
  http_listen_port: 9080

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:
  - job_name: "docker"
    docker_sd_configs:
      - host: unix:///var/run/docker.sock
        refresh_interval: 5s
    relabel_configs:
      - source_labels: ["__meta_docker_container_name"]
        target_label: "container"
      - source_labels: ["__meta_docker_compose_service"]
        target_label: "service"

---

Part 6: Grafana Setup

1. Visit http://your-server:3000
2. Log in with admin / your GRAFANA_PASSWORD
3. Configuration → Data Sources → Add data source:
   • Prometheus: URL http://prometheus:9090
   • Loki: URL http://loki:3100

Import Pre-built Dashboards

In Grafana → + → Import:

Dashboard	ID	What it shows
Node Exporter Full	1860	CPU, RAM, disk, network for Linux hosts
Docker / cAdvisor	14282	Container CPU, RAM, network
Nginx	9614	Nginx requests, errors, latency
PostgreSQL	9628	Queries, connections, cache hit rate
Redis	11835	Memory, ops/s, hit rate
Alertmanager	9578	Active alerts, notification history

---

Part 7: HTTPS with Caddy

grafana.yourdomain.com {
    reverse_proxy grafana:3000
}

prometheus.yourdomain.com {
    basicauth {
        admin $2a$14$...
    }
    reverse_proxy prometheus:9090
}

---

PromQL Quick Reference

# CPU usage % per host:
100 - (avg by(instance) (irate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

# RAM available GB:
node_memory_MemAvailable_bytes / 1024 / 1024 / 1024

# Disk usage %:
100 - ((node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"}) * 100)

# Container CPU usage:
rate(container_cpu_usage_seconds_total{name!=""}[5m]) * 100

# HTTP requests/s by service:
rate(http_requests_total[5m])

# Error rate %:
rate(http_requests_total{status=~"5.."}[5m]) / rate(http_requests_total[5m]) * 100

---

Maintenance

# Update all services:
docker compose pull
docker compose up -d

# Reload Prometheus config (no restart):
curl -X POST http://localhost:9090/-/reload

# Backup Grafana:
tar -czf grafana-backup-$(date +%Y%m%d).tar.gz \
  $(docker volume inspect grafana_data --format '{{.Mountpoint}}')

# Check Prometheus targets:
curl http://localhost:9090/api/v1/targets | jq '.data.activeTargets[].health'

Why Self-Host Grafana + Prometheus?

The case for self-hosting Grafana + Prometheus comes down to three practical factors: data ownership, cost at scale, and operational control.

Data ownership is the fundamental argument. When you use a SaaS version of any tool, your data lives on someone else's infrastructure subject to their terms of service, their security practices, and their business continuity. If the vendor raises prices, gets acquired, changes API limits, or shuts down, you're left scrambling. Self-hosting Grafana + Prometheus means your data and configuration stay on infrastructure you control — whether that's a VPS, a bare metal server, or a home lab.

Cost at scale matters once you move beyond individual use. Most SaaS equivalents charge per user or per data volume. A self-hosted instance on a $10-20/month VPS typically costs less than per-user SaaS pricing for teams of five or more — and the cost doesn't scale linearly with usage. One well-configured server handles dozens of users for a flat monthly fee.

Operational control is the third factor. The Docker Compose configuration above exposes every setting that commercial equivalents often hide behind enterprise plans: custom networking, environment variables, storage backends, and authentication integrations. You decide when to update, how to configure backups, and what access controls to apply.

The honest tradeoff: you're responsible for updates, backups, and availability. For teams running any production workloads, this is familiar territory. For individuals, the learning curve is real but the tooling (Docker, Caddy, automated backups) is well-documented and widely supported.

Server Requirements and Sizing

Before deploying Grafana + Prometheus, assess your server capacity against expected workload.

Minimum viable setup: A 1 vCPU, 1GB RAM VPS with 20GB SSD is sufficient for personal use or small teams. Most consumer VPS providers — Hetzner, DigitalOcean, Linode, Vultr — offer machines in this range for $5-10/month. Hetzner offers excellent price-to-performance for European and US regions.

Recommended production setup: 2 vCPUs with 4GB RAM and 40GB SSD handles most medium deployments without resource contention. This gives Grafana + Prometheus headroom for background tasks, caching, and concurrent users while leaving capacity for other services on the same host.

Storage planning: The Docker volumes in this docker-compose.yml store all persistent Grafana + Prometheus data. Estimate your storage growth rate early — for data-intensive tools, budget for 3-5x your initial estimate. Hetzner Cloud and Vultr both support online volume resizing without stopping your instance.

Operating system: Any modern 64-bit Linux distribution works. Ubuntu 22.04 LTS and Debian 12 are the most commonly tested configurations. Ensure Docker Engine 24.0+ and Docker Compose v2 are installed — verify with docker --version and docker compose version. Avoid Docker Desktop on production Linux servers; it adds virtualization overhead and behaves differently from Docker Engine in ways that cause subtle networking issues.

Network: Only ports 80 and 443 need to be publicly accessible when running behind a reverse proxy. Internal service ports should be bound to localhost only. A minimal UFW firewall that blocks all inbound traffic except SSH, HTTP, and HTTPS is the single most effective security measure for a self-hosted server.

Backup and Disaster Recovery

Running Grafana + Prometheus without a tested backup strategy is an unacceptable availability risk. Docker volumes are not automatically backed up — if you delete a volume or the host fails, data is gone with no recovery path.

What to back up: The named Docker volumes containing Grafana + Prometheus's data (database files, user uploads, application state), your docker-compose.yml and any customized configuration files, and .env files containing secrets.

Backup approach: For simple setups, stop the container, archive the volume contents, then restart. For production environments where stopping causes disruption, use filesystem snapshots or database dump commands (PostgreSQL pg_dump, SQLite .backup, MySQL mysqldump) that produce consistent backups without downtime.

For a complete automated backup workflow that ships snapshots to S3-compatible object storage, see the Restic + Rclone backup guide. Restic handles deduplication and encryption; Rclone handles multi-destination uploads. The same setup works for any Docker volume.

Backup cadence: Daily backups to remote storage are a reasonable baseline for actively used tools. Use a 30-day retention window minimum — long enough to recover from mistakes discovered weeks later. For critical data, extend to 90 days and use a secondary destination.

Restore testing: A backup that has never been restored is a backup you cannot trust. Once a month, restore your Grafana + Prometheus backup to a separate Docker Compose stack on different ports and verify the data is intact. This catches silent backup failures, script errors, and volume permission issues before they matter in a real recovery.

Security Hardening

Self-hosting means you are responsible for Grafana + Prometheus's security posture. The Docker Compose setup provides a functional base; production deployments need additional hardening.

Always use a reverse proxy: Never expose Grafana + Prometheus's internal port directly to the internet. The docker-compose.yml binds to localhost; Caddy or Nginx provides HTTPS termination. Direct HTTP access transmits credentials in plaintext. A reverse proxy also centralizes TLS management, rate limiting, and access logging.

Strong credentials: Change default passwords immediately after first login. For secrets in docker-compose environment variables, generate random values with openssl rand -base64 32 rather than reusing existing passwords.

Firewall configuration:

ufw default deny incoming
ufw allow 22/tcp
ufw allow 80/tcp
ufw allow 443/tcp
ufw enable

Internal service ports (databases, admin panels, internal APIs) should only be reachable from localhost or the Docker network, never directly from the internet.

Network isolation: Docker Compose named networks keep Grafana + Prometheus's services isolated from other containers on the same host. Database containers should not share networks with containers that don't need direct database access.

VPN access for sensitive services: For internal-only tools, restricting access to a VPN adds a strong second layer. Headscale is an open source Tailscale control server that puts your self-hosted stack behind a WireGuard mesh, eliminating public internet exposure for internal tools.

Update discipline: Subscribe to Grafana + Prometheus's GitHub releases page to receive security advisory notifications. Schedule a monthly maintenance window to pull updated images. Running outdated container images is the most common cause of self-hosted service compromises.

Troubleshooting Common Issues

Container exits immediately or won't start

Check logs first — they almost always explain the failure:

docker compose logs -f grafana

Common causes: a missing required environment variable, a port already in use, or a volume permission error. Port conflicts appear as bind: address already in use. Find the conflicting process with ss -tlpn | grep PORT and either stop it or change Grafana + Prometheus's port mapping in docker-compose.yml.

Cannot reach the web interface

Work through this checklist:

1. Confirm the container is running: docker compose ps
2. Test locally on the server: curl -I http://localhost:PORT
3. If local access works but external doesn't, check your firewall: ufw status
4. If using a reverse proxy, verify it's running and the config is valid: caddy validate --config /etc/caddy/Caddyfile

Permission errors on volume mounts

Some containers run as a non-root user. If the Docker volume is owned by root, the container process cannot write to it. Find the volume's host path with docker volume inspect VOLUME_NAME, check the tool's documentation for its expected UID, and apply correct ownership:

chown -R 1000:1000 /var/lib/docker/volumes/your_volume/_data

High resource usage over time

Memory or CPU growing continuously usually indicates unconfigured log rotation, an unbound cache, or accumulated data needing pruning. Check current usage with docker stats grafana. Add resource limits in docker-compose.yml to prevent one container from starving others. For ongoing visibility into resource trends, deploy Netdata.

Data disappears after container restart

Data stored in the container's writable layer — rather than a named volume — is lost when the container is removed or recreated. This happens when the volume mount path in docker-compose.yml doesn't match where the application writes data. Verify mount paths against the tool's documentation and correct the mapping. Named volumes persist across container removal; only docker compose down -v deletes them.

Keeping Grafana + Prometheus Updated

Grafana + Prometheus follows a regular release cadence. Staying current matters for security patches and compatibility. The update process with Docker Compose is straightforward:

docker compose pull          # Download updated images
docker compose up -d         # Restart with new images
docker image prune -f        # Remove old image layers (optional)

Read the changelog before major version updates. Some releases include database migrations or breaking configuration changes. For major version bumps, test in a staging environment first — run a copy of the service on different ports with the same volume data to validate the migration before touching production.

Version pinning: For stability, pin to a specific image tag in docker-compose.yml instead of latest. Update deliberately after reviewing the changelog. This trades automatic patch delivery for predictable behavior — the right call for business-critical services.

Post-update verification: After updating, confirm Grafana + Prometheus is functioning correctly. Most services expose a /health endpoint that returns HTTP 200 — curl it from the server or monitor it with your uptime tool.

Frequently Asked Questions

How much does it cost to self-host Grafana + Prometheus?

The primary cost is your server. A Hetzner CAX11 (2 vCPU ARM, 4GB RAM) runs about $5/month — enough for Grafana + Prometheus plus a few companion services. Add a domain ($12/year) and you're under $75/year for a complete self-hosted deployment. Compare that to SaaS pricing that typically starts at $5-15/user/month.

Can I run Grafana + Prometheus on a VPS with other services?

Yes. The docker-compose.yml above isolates Grafana + Prometheus on its own named Docker network. As long as your server has sufficient RAM and disk — 4GB RAM and 20GB disk handles most combinations — running multiple self-hosted services on one VPS is both practical and common. Tools like Dozzle and Portainer make monitoring multi-container setups manageable.

How do I migrate data if I switch servers?

Stop the Grafana + Prometheus container, export the Docker volumes (using docker run --rm -v VOLUME:/data -v $(pwd):/backup alpine tar czf /backup/backup.tar.gz /data), transfer to the new server, restore the volumes, and update your DNS. Most migrations complete in under an hour. Test the restoration on the new server before decommissioning the old one.

What happens if Grafana + Prometheus releases a breaking update?

Pin your docker-compose.yml to a specific image tag (e.g., image: grafana/+/prometheus:1.2.3 instead of latest). Subscribe to the GitHub releases page for advance notice. When you're ready to upgrade, read the release notes, back up first, test on a staging instance, then update production.

Is Grafana + Prometheus suitable for production use?

Yes, with the hardening described above: reverse proxy for HTTPS, firewall rules, regular backups, and a pinned image tag. Many teams run Grafana + Prometheus in production successfully. The main requirement is treating your self-hosted instance with the same operational discipline you'd apply to any business-critical service.

---

See all open source monitoring and observability tools at OSSAlt.com/categories/monitoring.

See open source alternatives to Grafana on OSSAlt.