duyne

    EC2 Node Exporter

    • kubernetes
    • exporter
    • ec2
    • victoriametrics
    • prometheus

    Stay tuned for more Kubernetes updates and best practices!

    Overview

    This document covers the procedure for installing Prometheus Node Exporter on a standalone EC2 instance, not on a worker node managed by a Kubernetes cluster.

    It targets general EC2 servers (not EKS worker nodes or Kubernetes environments) and describes how to install Node Exporter as a binary. Node Exporter is a lightweight agent composed of a single executable file that collects system metrics without complex dependencies.

    Background

    Best Practices for Installing Node Exporter

    The recommended installation method for Node Exporter varies by environment:

    • Standalone servers (EC2, VM): Binary installation ✅

      • No dependencies (single binary)
      • Direct access to system resources
      • Native systemd service management
      • No Docker overhead

    • Kubernetes (EKS, GKE): DaemonSet (containerized)

      • Automatically deployed on each node
      • Declarative configuration management
      • Supports rolling updates
      • Native Kubernetes integration

    Why is binary installation best for standalone EC2?

    1. Official recommendation: Prometheus official docs suggest binary install for non-containerized environments
    2. Performance: Direct access to host metrics without container overhead
    3. Simplicity: Managed via systemd (auto-restart, logging)
    4. Efficiency: No Docker daemon, minimal CPU/memory usage

    Environment

    flowchart TB
    subgraph Architecture[Environment Architecture]
        direction TB
        subgraph EC2[EC2 Instance]
            direction TB
            NodeExporter[node_exporter v1.9.1<br/>systemd service<br/>Port: 9100]
        end
        
        subgraph K8s[Kubernetes Cluster]
            direction TB
            VM[VictoriaMetrics<br/>VMStaticScrape<br/>Scrapes every 30s]
        end
        
        NodeExporter -->|HTTP GET /metrics<br/>tcp/9100| VM
    end
    
    style Architecture fill:#fff,stroke:#000,stroke-width:2px
    style EC2 fill:#fff,stroke:#000,stroke-width:2px
    style K8s fill:#fff,stroke:#000,stroke-width:2px
    style NodeExporter fill:#fff,stroke:#000,stroke-width:2px
    style VM fill:#fff,stroke:#000,stroke-width:2px

    Installation Guide

    Prometheus Node Exporter

    Node Exporter is distributed as a single binary, making it extremely easy to install. No package dependencies or libraries required. Just download from GitHub and copy to a system path.

    export NODE_EXPORTER_VERSION="1.9.1"

cd /tmp
curl -L "https://github.com/prometheus/node_exporter/releases/download/v${NODE_EXPORTER_VERSION}/node_exporter-${NODE_EXPORTER_VERSION}.linux-amd64.tar.gz" | tar xzf -

sudo cp node_exporter-*/node_exporter /usr/local/bin/
sudo chown root:root /usr/local/bin/node_exporter
sudo chmod +x /usr/local/bin/node_exporter

# Cleanup downloaded files
rm -rf node_exporter-*

    Create a Dedicated Linux User

    Run Node Exporter under a dedicated system user for security. This user will have no home directory or shell access.

    sudo useradd --no-create-home --shell /bin/false node_exporter

    Create systemd Service

    Register Node Exporter as a system service for auto-start and recovery.

    sudo tee /etc/systemd/system/node_exporter.service > /dev/null <<EOF
[Unit]
Description=Node Exporter v1.9.1
Documentation=https://prometheus.io/docs/guides/node-exporter/
Wants=network-online.target
After=network-online.target

[Service]
User=node_exporter
Group=node_exporter
Type=simple
Restart=on-failure
RestartSec=5s
ExecStart=/usr/local/bin/node_exporter
SyslogIdentifier=node_exporter

[Install]
WantedBy=multi-user.target
EOF

    Start the Service

    Reload systemd, enable the service on boot, and start it.

    sudo systemctl daemon-reload
sudo systemctl enable node_exporter
sudo systemctl start node_exporter
sudo systemctl status node_exporter

    Verify Metrics Endpoint

    Node Exporter listens on TCP port 9100 and exposes metrics via the /metrics endpoint. Use curl to verify locally:

    curl http://localhost:9100/metrics

    You should see output like:

    # HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 0
go_gc_duration_seconds{quantile="0.25"} 0
...
# HELP node_cpu_seconds_total Seconds the CPUs spent in each mode.
# TYPE node_cpu_seconds_total counter
node_cpu_seconds_total{cpu="0",mode="idle"} 1234.56
...

    Configure Security Group

    Ensure TCP port 9100 is open in the EC2 Security Group. For security best practices:

    • Source: Limit access to your monitoring infrastructure (specific security group or CIDR block)
    • Protocol: TCP
    • Port: 9100
    • Description: Node Exporter metrics endpoint

    This prevents unauthorized access to system metrics while allowing your monitoring system to collect data.

    VictoriaMetrics Integration

    About VictoriaMetrics

    VictoriaMetrics is a high-performance, cost-effective monitoring solution and time series database. It's designed as a drop-in replacement for Prometheus with better performance, lower resource consumption, and horizontal scalability.

    Key advantages over Prometheus:

    • Better performance: Up to 10x faster ingestion and queries
    • Lower resource usage: Requires less RAM and CPU
    • Long-term storage: Built-in support for long-term metric retention
    • Horizontal scaling: Native clustering support
    • Prometheus compatibility: 100% compatible with PromQL and Prometheus ecosystem

    Configure VictoriaMetrics Scraping

    To scrape Node Exporter metrics from your EC2 instances, create a VMStaticScrape resource in your Kubernetes cluster:

    ---
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMStaticScrape
metadata:
  name: ec2-node-exporter
  namespace: monitoring
spec:
  jobName: ec2-node-exporter
  targetEndpoints:
    - targets:
        - "10.20.1.6:9100"
      relabelConfigs:
        # bastion-host
        - sourceLabels: [__address__]
          regex: "10.20.1.6:9100"
          targetLabel: instance
          replacement: "bastion-host"

    This configuration:

    • Targets: Defines the EC2 instances to scrape
    • Relabel configs: Maps IP addresses to friendly instance names
    • Job name: Groups these targets under ec2-node-exporter job

    Apply the Configuration

    kubectl apply -f ec2-node-exporter-scrape.yaml

    Verify the scraping targets are discovered:

    # Check VMStaticScrape resource
kubectl get vmstaticscrape -n monitoring

# Check VictoriaMetrics targets (if vmselect is exposed)
curl "http://vmselect:8481/select/0/prometheus/api/v1/targets"

    References

    GitHub:

    Official Documentation: