Workshop Exercises: Understand
Competing Hypotheses Matrix
Weigh the evidence against each explanation instead of collecting support for a favorite
Why: Find the explanation with the least evidence against it, see which evidence actually discriminates between explanations, and decide what to check next
When: Use when several explanations for a problem are still plausible and the team needs to decide which to pursue and what evidence would separate them
A competing hypotheses matrix lists possible explanations in columns and evidence in rows, and marks each cell as consistent, inconsistent or not applicable, so a team can see which explanation has the least evidence against it. It is adapted from Analysis of Competing Hypotheses (ACH), which Richards J. Heuer, Jr. describes in Psychology of Intelligence Analysis.
Heuer’s starting point is that people tend to pick the explanation that seems most likely and then look for evidence that supports it. Most evidence is consistent with several explanations, so confirming evidence proves little. Evidence that is inconsistent with an explanation is harder to find and tells you more.
This addresses a gap in tools such as the fishbone diagram and the Five Whys. They organize and trace possible causes, but as the root cause analysis guide points out, they do not verify them. The matrix makes the team weigh what it actually knows against every explanation at once.
Worked example: incomplete product setup
This scenario, its evidence and its conclusions are illustrative, not findings from a real investigation. It continues the example from the root cause analysis guide, the incident timeline and Is / Is Not. The team has four hypotheses for why fewer new accounts finish the company-details step:
- H1: The next step is unclear.
- H2: The new form component does not capture autofilled values.
- H3: The completion event is missing, although setup succeeds.
- H4: The campaign shifted the mix of new accounts toward mobile visitors, who complete less often.
| Evidence | H1 | H2 | H3 | H4 |
|---|---|---|---|---|
| Support tickets report an error after pressing Continue | I | C | I | N/A |
| The saved setup state is incomplete for the blocked attempts | C | C | I | C |
| Observed participants find and press Continue | I | C | C | C |
| Completion falls from the day the new component shipped | I | C | C | C |
| The failure reproduces with autofill on the affected browser, not with manual entry | I | C | I | N/A |
| The share of new accounts on mobile rose after the campaign launch | N/A | N/A | N/A | C |
Reading down the columns:
- H1 has four inconsistencies, including the reproduction and the observation that participants find Continue. It is the weakest explanation for this failure.
- H3 is contradicted by the error reports, the saved setup state and the reproduction.
- H2 has no evidence against it, and the reproduction is the most decisive row.
- H4 also has no evidence against it. That does not make it equally strong: none of the rows is inconsistent with it partly because H4 makes few specific predictions about errors.
Two rows do not discriminate between H2, H3 and H4: the timing, because the release and the campaign happened in the same week, and the observation that participants find Continue. Both count against H1, but they do not help choose among the others.
H2 and H4 are not mutually exclusive. A defect could cause the errors while a traffic shift also lowers the overall rate. The missing evidence that would separate them is completion within comparable cohorts: if completion fell for mobile accounts using autofill but not for mobile accounts entering details manually, the defect accounts for the change within the cohort, and the campaign explains, at most, a change in its size.
The team records:
- Leading hypothesis: H2, supported by a reproduced failure path.
- Still open: how much H4 contributes to the overall metric; whether other browsers are affected.
- Would change the conclusion: a manual-entry cohort with the same drop, or a failed reproduction on a second device.
It then corrects the value capture and verifies the change, as the root cause analysis guide describes.
Limitations
The matrix only weighs the hypotheses you list. If the true explanation is not one of the columns, the matrix will still produce a leader. Heuer stresses generating hypotheses with people who bring different perspectives.
Marks are judgments. Deciding whether a piece of evidence is consistent with an explanation is itself an interpretation, and the group can share the same blind spot. Write down the reason for each inconsistency so others can challenge it.
Not all evidence weighs the same. A count of inconsistencies is a rough ranking. A reproduced failure and a recalled anecdote should not count equally; discuss the reliability of the decisive rows.
A vague hypothesis looks strong. An explanation that predicts little attracts few inconsistencies. Make each hypothesis specific enough that some evidence could contradict it.
The leading hypothesis is not verified. The matrix tells you which explanation is least contradicted by the evidence so far and what to check next. A test, reproduction or follow-up measurement still has to establish the mechanism.
Instructions for running this play
- Prepare a problem statement and a list of candidate explanations, for example from a Fishbone Diagram or Is / Is Not. Discuss the problem statement first thing.
- Write each hypothesis at the top of a column. Make each one specific enough that some evidence could count against it, and include at least one that nobody favors. Note which hypotheses could be true at the same time.
- Write each piece of evidence at the start of a row: one observation per row, with its source. Include test results, logs, tickets, analytics, interview observations and expected things that did not happen. Label assumptions and arguments as such.
- Work across one row at a time. For each piece of evidence, mark every hypothesis C (consistent), I (inconsistent) or N/A (does not apply). Discuss disagreements and note the reason for each I.
- Flag rows that are consistent with every hypothesis. They do not help choose between explanations, however much they confirm the problem. Keep them on the record but set them aside.
- Work down the columns. Look at the evidence against each hypothesis and how reliable it is. The leading hypothesis is the one with the least evidence against it, not the one with the most evidence for it.
- Ask which few rows decide the result, and what would change if one of them were wrong or misread. Check those items at the source where you can.
- For the hypotheses still standing, identify the missing evidence that would separate them: an observation that one hypothesis predicts and another does not. Prioritize the checks that are cheap and decisive, and assign an owner to each.
- Record the leading hypothesis and your confidence in it, the other hypotheses still standing, what remains unresolved and what new evidence would change the conclusion. Consider a Who / What / When Matrix to commit to the next checks.
Tips to perfect this play
Master and adapt the play to fit your context and needs.
Tip: Work across rows, not down columns
Take one piece of evidence at a time and judge it against every hypothesis. Going hypothesis by hypothesis invites the group to argue for its favorite.
Tip: Don't count the ticks
A long list of consistent evidence is easy to build for almost any reasonable explanation. Look at the marks against each hypothesis, and weigh them: one reliable inconsistency can matter more than several weak ones.
Tip: Include an unpopular hypothesis
Add at least one explanation that nobody in the room favors. If it survives the matrix, the team has learned something.
Tip: Use a spreadsheet for large matrices
Sticky notes work for four or five hypotheses. Beyond that, a shared spreadsheet keeps the matrix readable and leaves a record of the reasoning.
Every card deck includes workshop patterns, so your team can put the methods to work in your next session.
Running behavioral design workshops? The Persuasive Patterns Workshop Playbook (PDF) shows you how to design and run them.
Browse the card decks- Psychology of Intelligence Analysis, Chapter 8: Analysis of Competing Hypotheses by Richards J. Heuer, Jr.
- Quality tool: Cause and effect diagram by Richard Green