When a Kubernetes incident starts, speed matters, but random debugging creates noise.
This is the compact checklist I use to stay methodical under pressure.
#1. Confirm impact before touching anything
- Which user-facing feature is failing?
- Is the issue global or limited to one region/cluster?
- What is the exact start time?
A clear impact statement keeps everyone aligned and helps avoid unnecessary fixes.
#2. Validate cluster health in 60 seconds
kubectl get nodes
kubectl get pods -A --field-selector=status.phase!=Running
kubectl get events -A --sort-by=.lastTimestamp | tail -n 30
Look for broad failure signals first: node pressure, scheduling failures, image pull failures, or crash loops.
#3. Identify the failing workload path
Start from the affected service and walk down:
- Ingress/Gateway
- Service
- Endpoints
- Deployment/StatefulSet
- Pods
This path avoids jumping to random pods before checking routing and service discovery.
#4. Check recent changes
Most production incidents correlate with a change.
- New release?
- ConfigMap/Secret update?
- HPA or resource limit tweak?
- Cluster upgrade or node rotation?
If a change lines up with incident start, test rollback quickly.
#5. Capture a short timeline while debugging
Log key events with timestamps in a shared channel/doc.
Example:
- 10:03 UTC: error rate spike detected
- 10:07 UTC: image pull back-off confirmed in
payments-api - 10:12 UTC: rollback started
- 10:16 UTC: success rate recovered
This timeline becomes the backbone of the postmortem.
#Closing note
Incidents feel chaotic, but a repeatable checklist helps teams move faster with less stress.
If you are building your own runbook culture, start with one-page checklists like this and evolve them over time.