Skip to content

Cluster health checks

Manual

kubectl cluster-info
kubectl get nodes

API checks

curl -k https://localhost:6443/healthz

or

kubectl get --raw='/healthz?verbose'
kubectl get --raw='/readyz?verbose'
kubectl get --raw='/livez?verbose'

Monitoring kubernetes clusters with Prometheus

  • Helm chart
  • Uses prometheus-operator
    • prometheus-operator doesn't honor pod annotation for sevice discovery like prometheus does. It's configured by CRDs like podMonitor or serviceMonitor

Exporter

  • eventrouter
    • A simple introspective kubernetes service that forwards events to a specified sink
    • Comes with an integrated Prometheus exporter
      • Exposes eventrouter_normal_total and eventrouter_warnings_total metrics
  • KubeStateMetrics Documentation

Show events when eventrouter is configured to log to /logs/eventrouter.log:

alias k8s_events="kubectl -n monitoring exec deployments/eventrouter -- cat /logs/eventrouter.log | jq '. | select(.event.type != \"Normal\")'"
k8s_events

Filter out Normal events:

alias k8s_events_warn="k8s_events | jq -r '. | select(.event.type != \"Normal\")'"
k8s_events_warn

Pretty-print table of non-Normal events (glow wraps long lines in table):

k8s_events_warn | jq -r '. |  [ .event.metadata.creationTimestamp, .event.type, .event.involvedObject.kind, .event.involvedObject.namespace, .event.involvedObject.name, .event.reason, .event.message ] | @csv' | csv2md -H | glow