Stay tuned for more Kubernetes updates and best practices!

Overview

This document covers the procedure for installing Prometheus Node Exporter on a standalone EC2 instance, not on a worker node managed by a Kubernetes cluster.

It targets general EC2 servers (not EKS worker nodes or Kubernetes environments) and describes how to install Node Exporter as a binary. Node Exporter is a lightweight agent composed of a single executable file that collects system metrics without complex dependencies.

Background

Best Practices for Installing Node Exporter

The recommended installation method for Node Exporter varies by environment:

  • Standalone servers (EC2, VM): Binary installation ✅

    • No dependencies (single binary)
    • Direct access to system resources
    • Native systemd service management
    • No Docker overhead

  • Kubernetes (EKS, GKE): DaemonSet (containerized)

    • Automatically deployed on each node
    • Declarative configuration management
    • Supports rolling updates
    • Native Kubernetes integration

Why is binary installation best for standalone EC2?

  1. Official recommendation: Prometheus official docs suggest binary install for non-containerized environments
  2. Performance: Direct access to host metrics without container overhead
  3. Simplicity: Managed via systemd (auto-restart, logging)
  4. Efficiency: No Docker daemon, minimal CPU/memory usage

Environment

┌─────────────────────────────────────┐
│         EC2 Instance                │
│  ┌─────────────────────────────────┐ │
│  │     node_exporter v1.9.1       │ │
│  │     (systemd service)           │ │
│  │     Port: 9100                  │ │
│  └─────────────────────────────────┘ │
└─────────────────────────────────────┘
                   │
                   │ HTTP GET /metrics
                   │ (tcp/9100)
                   ▼
┌─────────────────────────────────────┐
│      Kubernetes Cluster            │
│  ┌─────────────────────────────────┐ │
│  │     VictoriaMetrics             │ │
│  │     (VMStaticScrape)            │ │
│  │     Scrapes every 30s           │ │
│  └─────────────────────────────────┘ │
└─────────────────────────────────────┘

Installation Guide

Prometheus Node Exporter

Node Exporter is distributed as a single binary, making it extremely easy to install. No package dependencies or libraries required. Just download from GitHub and copy to a system path.

export NODE_EXPORTER_VERSION="1.9.1"

cd /tmp
curl -L "https://github.com/prometheus/node_exporter/releases/download/v${NODE_EXPORTER_VERSION}/node_exporter-${NODE_EXPORTER_VERSION}.linux-amd64.tar.gz" | tar xzf -

sudo cp node_exporter-*/node_exporter /usr/local/bin/
sudo chown root:root /usr/local/bin/node_exporter
sudo chmod +x /usr/local/bin/node_exporter

# Cleanup downloaded files
rm -rf node_exporter-*

Create a Dedicated Linux User

Run Node Exporter under a dedicated system user for security. This user will have no home directory or shell access.

sudo useradd --no-create-home --shell /bin/false node_exporter

Create systemd Service

Register Node Exporter as a system service for auto-start and recovery.

sudo tee /etc/systemd/system/node_exporter.service > /dev/null <<EOF
[Unit]
Description=Node Exporter v1.9.1
Documentation=https://prometheus.io/docs/guides/node-exporter/
Wants=network-online.target
After=network-online.target

[Service]
User=node_exporter
Group=node_exporter
Type=simple
Restart=on-failure
RestartSec=5s
ExecStart=/usr/local/bin/node_exporter
SyslogIdentifier=node_exporter

[Install]
WantedBy=multi-user.target
EOF

Start the Service

Reload systemd, enable the service on boot, and start it.

sudo systemctl daemon-reload
sudo systemctl enable node_exporter
sudo systemctl start node_exporter
sudo systemctl status node_exporter

Verify Metrics Endpoint

Node Exporter listens on TCP port 9100 and exposes metrics via the /metrics endpoint. Use curl to verify locally:

curl http://localhost:9100/metrics

You should see output like:

# HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 0
go_gc_duration_seconds{quantile="0.25"} 0
...
# HELP node_cpu_seconds_total Seconds the CPUs spent in each mode.
# TYPE node_cpu_seconds_total counter
node_cpu_seconds_total{cpu="0",mode="idle"} 1234.56
...

Configure Security Group

Ensure TCP port 9100 is open in the EC2 Security Group. For security best practices:

  • Source: Limit access to your monitoring infrastructure (specific security group or CIDR block)
  • Protocol: TCP
  • Port: 9100
  • Description: Node Exporter metrics endpoint

This prevents unauthorized access to system metrics while allowing your monitoring system to collect data.

VictoriaMetrics Integration

About VictoriaMetrics

VictoriaMetrics is a high-performance, cost-effective monitoring solution and time series database. It’s designed as a drop-in replacement for Prometheus with better performance, lower resource consumption, and horizontal scalability.

Key advantages over Prometheus:

  • Better performance: Up to 10x faster ingestion and queries
  • Lower resource usage: Requires less RAM and CPU
  • Long-term storage: Built-in support for long-term metric retention
  • Horizontal scaling: Native clustering support
  • Prometheus compatibility: 100% compatible with PromQL and Prometheus ecosystem

Configure VictoriaMetrics Scraping

To scrape Node Exporter metrics from your EC2 instances, create a VMStaticScrape resource in your Kubernetes cluster:

---
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMStaticScrape
metadata:
  name: ec2-node-exporter
  namespace: monitoring
spec:
  jobName: ec2-node-exporter
  targetEndpoints:
    - targets:
        - "10.20.1.6:9100"
      relabelConfigs:
        # bastion-host
        - sourceLabels: [__address__]
          regex: "10.20.1.6:9100"
          targetLabel: instance
          replacement: "bastion-host"

This configuration:

  • Targets: Defines the EC2 instances to scrape
  • Relabel configs: Maps IP addresses to friendly instance names
  • Job name: Groups these targets under ec2-node-exporter job

Apply the Configuration

kubectl apply -f ec2-node-exporter-scrape.yaml

Verify the scraping targets are discovered:

# Check VMStaticScrape resource
kubectl get vmstaticscrape -n monitoring

# Check VictoriaMetrics targets (if vmselect is exposed)
curl "http://vmselect:8481/select/0/prometheus/api/v1/targets"

References

GitHub: