duyne

    NodeLocal DNSCache on EKS

    • aws
    • eks
    • nodelocal
    • dnscache

    Overview

    Node Local DNS Cache improves DNS performance in Kubernetes by caching DNS queries locally on each node. Below are the steps to deploy Node Local DNS in an Amazon EKS cluster.

    DNS query workflow with NodeLocal DNSCache.

    flowchart LR
  A["Pod DNS Query"] --> B["NodeLocal DNSCache"]
  B --> C{"Cache Hit?"}
  C -->|Yes| D["Return Cached Result"]
  C -->|No| E{"Query Type?"}
  E -->|Local Domain| F["CoreDNS"]
  E -->|Reverse DNS| F
  E -->|External| G["VPC DNS Resolver"]
  F --> H["Response to Pod"]
  G --> H

    Installation

    Prepare a manifest similar to the sample nodelocaldns.yaml and save it as nodelocaldns.yaml.

    wget https://raw.githubusercontent.com/kubernetes/kubernetes/master/cluster/addons/dns/nodelocaldns/nodelocaldns.yaml

    Modify the default parameters in the template:

    PILLAR__DNS__SERVER=$(kubectl get svc kube-dns -n kube-system -o jsonpath={.spec.clusterIP})
    sed -i "s/__PILLAR__LOCAL__DNS__/169.254.20.10/g; s/__PILLAR__DNS__DOMAIN__/cluster.local/g; s/__PILLAR__DNS__SERVER__/$PILLAR__DNS__SERVER/g" nodelocaldns.yaml

    The __PILLAR__CLUSTER__DNS__ and __PILLAR__UPSTREAM__SERVERS__ parameters do not need to be replaced in the template.

    Apply the template:

    kubectl apply -f nodelocaldns.yaml

    Check if the NodeLocal DNSCache pods are running:

    kubectl get pods -n kube-system | grep node-local-dns
node-local-dns-2mq4b                                         1/1     Running   0          24m
node-local-dns-44tbj                                         1/1     Running   0          24m
node-local-dns-8h9mc                                         1/1     Running   0          24m
node-local-dns-9fr7r                                         1/1     Running   0          24m
node-local-dns-9ltws                                         1/1     Running   0          24m
node-local-dns-j65bw                                         1/1     Running   0          24m
node-local-dns-mdxcc                                         1/1     Running   0          24m
node-local-dns-qssqs                                         1/1     Running   0          23m
node-local-dns-xc89s                                         1/1     Running   0          24m
node-local-dns-z6jqj                                         1/1     Running   0          24m
node-local-dns-zww2l                                         1/1     Running   0          24m

    Key Improvements

    Latency: DNS queries now resolve in ~0-1ms (local cache) vs 5-10ms (upstream).

    Resource Usage: CoreDNS CPU/Memory significantly reduced.

    Stability: Fewer conntrack entries and UDP connections.

    Scalability: CoreDNS can handle cluster growth without being overwhelmed.

    The drop at 08:55 is when NodeLocal DNSCache went live — upstream CoreDNS traffic falls off a cliff because the node-local cache is answering instead:

    Grafana: CoreDNS total forward requests and cache hits/misses both collapse at 08:55 once NodeLocal DNSCache is serving queries

    Grafana: CoreDNS requests by type drop from roughly 32/s to 10/s at 08:55, almost entirely A and AAAA queries

    References