Problem!
I have deployed GKE cluster and run our workloads in there. As a monitoring stack I deployed Prometheus and Grafana using stable helm charts. Problem here is, how we can monitor these kube-dns pods? I can get cpu and memory usage of these kube-dns pods, but other metrics like skydns_skydns_dns_error_count_total etc won’t scrape by prometheus, by default. The reason for this is those metrics endpoint are not exposed yet.
If you check the deployment config you can see metrics ports are there in the container.
Prometheus is not able to discover these endpoint unless we crate a service for them, Therefore we create a service like bellow. and apply.
kube-dns-metrics.yaml
apiVersion: v1
kind: Service
metadata:
name: metrics-sidecar-kube-dns
labels:
app: metrics-sidecar-kube-dns
namespace: kube-system
annotations:
prometheus.io/port: "10054"
prometheus.io/scrape: "true"
spec:
clusterIP: None
ports:
- name: http-metrics-sidecar
port: 10054
protocol: TCP
targetPort: 10054
selector:
k8s-app: kube-dns
---
apiVersion: v1
kind: Service
metadata:
name: metrics-kube-dns
labels:
app: metrics-kube-dns
namespace: kube-system
annotations:
prometheus.io/port: "10055"
prometheus.io/scrape: "true"
spec:
clusterIP: None
ports:
- name: http-metrics-kube-dns
port: 10055
protocol: TCP
targetPort: 10055
selector:
k8s-app: kube-dns
# kubectl apply -f kube-dns-metrics.yaml
After creating that service you should be able to see following metrics
Dashboard
You can get above dashboard from following url