Root Cause Analysis
Going beyond symptoms to find systemic causes.
1Why Root Causes Matter
Most problem-solving addresses symptoms, not causes.
Deploy failed? Roll back and redeploy. Bug in production? Hotfix it. Build is slow? Add more build servers.
These fixes address the immediate pain. The problem goes away... until it comes back. Because you didn't fix what caused it.
Root cause analysis means asking "why?" until you find the systemic cause—the thing that, if fixed, would prevent the problem from recurring.
The deploy didn't fail because of a random glitch. It failed because the test suite is flaky. The test suite is flaky because tests share state. Tests share state because the testing framework wasn't understood when it was set up.
Fix the immediate symptom (retry the deploy) and you'll retry again tomorrow. Fix the root cause (fix the test framework understanding) and deploys stop failing.
Root cause analysis takes more time upfront. But it saves exponentially more time by preventing recurrence.
The Danger of Quick Fixes
Every quick fix that doesn't address root causes creates a pattern: the problem recurs, you fix it again, it becomes 'normal.' Eventually you have a system held together by workarounds, where the actual problems are invisible beneath layers of patches.
2Five Whys
The Five Whys is the simplest root cause technique: keep asking "why?" until you reach a root cause, usually around five iterations.
Example:
Problem: Production went down for 2 hours.
- Why? A bad configuration was deployed.
- Why? The config change wasn't tested.
- Why? We don't have config change tests.
- Why? Config is managed separately from code.
- Why? The infrastructure team set it up before the dev team adopted GitOps.
Root cause: Historical separation of config from code in the deployment pipeline.
Countermeasure: Move config into the same repository and CI/CD pipeline as application code.
Pitfalls of Five Whys:
- Stopping too early: "Human error" is never a root cause. Why did the error happen? What system allowed it?
- Single-threaded: Real problems often have multiple causes. Five Whys can miss parallel causal chains.
- Opinion-driven: Without data, you might ask "why?" and get speculation instead of fact.
- Blame-seeking: If Five Whys degenerates into "who messed up?", it's being misused.
Five Whys works best as a starting point, supplemented with data and multiple perspectives.
3Ishikawa (Fishbone) Diagrams
The Ishikawa diagram, also called a fishbone or cause-and-effect diagram, helps explore multiple cause categories simultaneously.
The problem is the "head" of the fish. Major cause categories are the "bones." Sub-causes branch off each bone.
Common categories (the 6 M's):
- Manpower: People, skills, training
- Method: Processes, procedures
- Machine: Tools, equipment, systems
- Material: Inputs, data, dependencies
- Measurement: Metrics, monitoring, feedback
- Mother Nature (Environment): External factors, context
For software, you might adapt:
- People: Skills, communication, team structure
- Process: Workflow, handoffs, policies
- Technology: Tools, infrastructure, architecture
- Data: Inputs, state, dependencies
- Measurement: Visibility, monitoring, alerting
- Environment: External services, load, context
The diagram helps teams brainstorm causes systematically rather than anchoring on the first idea. It reveals that problems usually have multiple contributing causes.
Combine Techniques
Use Five Whys to drill deep on each branch of the fishbone. The fishbone ensures you explore broadly; Five Whys ensures you explore deeply. Together they're powerful.
Incident: Deploy failure caused 2-hour outage. Five Whys led to: config not tested → config managed separately → historical architecture decision. Fishbone revealed: also no deploy runbook (Process), no canary process (Method). Addressed all three.
Incident: Deploy failure caused 2-hour outage. Post-mortem conclusion: 'Developer should have been more careful.' Action item: 'Be more careful next time.' Problem recurs two weeks later with a different developer.
- Fixing symptoms without root causes leads to recurring problems
- Five Whys drills down by asking 'why?' repeatedly
- Human error is never a root cause—ask what system allowed it
- Fishbone diagrams help explore multiple cause categories
- Use data, not speculation, to validate causal chains
- •Stopping at 'human error' or 'lack of attention'
- •Using Five Whys without data to validate assumptions
- •Finding one cause and stopping (problems often have multiple causes)
- •Turning root cause analysis into blame assignment