Most PM tools measure what you shipped. This one measures your predictions.

Every product manager has shipped a feature, predicted a number, and forgotten the prediction before the data came in. It is the central, unmeasured failure mode of the role.

I built Hypothesis Lab to fix it for myself. This case study walks through the product, the one design decision the whole tool turns on, the three things shipped wrong and had to correct, and what I improve my PM skills because of it.

For those who want to try: https://hypothesis-lab.vercel.app/

The reframe

Roadmap tools answer what is being built. Experimentation platforms answer did the experiment win. Discovery tools answer should we build this.

The gap lives between we decided to ship this and did it do what we thought it would do. Some PMs close it informally, six months later, in a sentence to a peer. The team learns nothing. The PM learns less.

So I built a tool that does exactly one thing well. It lets you write down what you think will happen, before it happens, in a structure that can be marked against reality later. The structure is a causal chain of four claims: what you changed, then behavior change, then metric movement, then business outcome. A predicted magnitude per claim. A pre-mortem. A confidence rating. A review date. Then you ship.

When the review date arrives, you mark each claim:
Held/Wrong-Magnitude/Broke/Inconclusive. The system computes your calibration. Over many hypotheses, your calibration is the evidence of whether you are good at the predictive part of the job.

What it looks like

A PM starts by picking a template. There are fourteen of them, each pre-structured for a specific kind of bet, from pricing changes to platform investments to retention interventions. The point of templates is not to constrain thinking. It is to make sure the causal chain you write actually matches the shape of the decision you are making. There is also a custom template for those special cases.

Template picker. Each option pre-fills the causal chain structure for that bet type so a PM never starts from a blank page.

From the template, the PM fills in four claim descriptions, four predicted magnitudes, a pre-mortem with five stress-test questions (weakest link, base rate, concurrent confounds, earliest failure signal, kill criteria), a visibility setting (public or private), a review date, and finally a confidence rating. The confidence number is collected last on purpose. More on that below.

Active and resolved hypotheses live in the PM’s workspace, sorted by lifecycle status. Draft, Active, Awaiting Resolution, Resolved.

The PM workspace. Hypotheses become visible to the whole team. Awaiting Resolution surfaces overdue review dates with persistent badging.

When a PM opens a resolved hypothesis, they see the causal chain laid out horizontally, left to right, with the predicted value and the actual value stacked vertically within each column. Each column is colored by outcome: green for Held, amber for Wrong Magnitude, red for Broke. The layout is identical in creation, active, and resolved views, so the eye reads the chain the same way at every stage of the lifecycle.

A resolved hypothesis. Four claims, each marked independently. The overall outcome chip in the top right reflects the mechanical score.

That is the loop: Make prediction, ship the bet, learn from the result.

The one decision the product turns on

Mechanical scoring, no override.

Every other design choice in Hypothesis Lab is a consequence of that one. If the PM can move their own score, the score is opinion. If the score is opinion, the team dashboard is a collection of self-evaluations. If the team dashboard is a collection of self-evaluations, the Head of Product is back where they started, trying to figure out who on the team is well-calibrated and who is not, by feel, in 1:1s.

The whole product is an attempt to replace one specific HoP intuition (I think X is sharper at retention bets than Y) with one specific number, computed the same way for everyone, that the HoP and the PM can both read.

So: The PM can flag a contested resolution in their learning note. The flag rolls up into a “contested rate” metric the HoP sees alongside calibration. The PM’s view is recorded. The score does not move.

This is the most unpopular constraint in the product. Every PM I have shown it to has, within five minutes, proposed a feature that softens it. I have not shipped any of them.

A few related refusals were followed:

• Confidence is rated after the pre-mortem, never before, because a confidence number collected before the work of thinking-through is an anchor, not an assessment.
• Private hypotheses are permanently excluded from the team dashboard and learning library, with no toggle anywhere that exposes them later, because the value of “private” depends on the user trusting that private means private forever.
• The Head of Product role is enforced as a singleton at the database layer through a unique partial index, not in application logic, because a team with two HoPs is a broken org chart regardless of which bug produced it.

What it costs

The cost of mechanical scoring with no override is real.

The number is visible, repeatable, and not negotiable. For a PM who is well-calibrated, this is a career asset. For a PM who is not, it is a quarterly conversation they would rather avoid.

Adoption is slower than it would be otherwise. A more permissive tool with a self-assessed view would onboard PMs faster. They would also use it less seriously, and the data would be worth less.

Calibration accrues slowly. A PM ships maybe 4 to 12 meaningful hypotheses a quarter. The headline number is meaningful around year one of use.

I made every one of those trades knowingly. The alternative was a tool PMs use comfortably and produces no signal worth acting on.

What it buys

Three things.

Data the team can trust. A HoP looking at the team dashboard is looking at the same metric, computed the same way, across every PM. Nobody is graded on a curve they set themselves.

The HoP team dashboard. Each PM is shown individually rather than averaged. The high-confidence broke panel surfaces the C4 and C5 bets that recently failed, because those are the highest-learning-opportunity moments on the team.

A 1:1 conversation that did not exist before. The HoP can walk into a 1:1 with a number and a contested-rate signal and ask, specifically, your C5 hypotheses break twice as often as the team median, walk me through the last one. Coaching becomes evidence-driven.

A learning library that is honest. Every resolved public hypothesis goes into a searchable library with the original prediction, the actual outcome, and the PM’s learning note. A PM about to ship a pricing change can find every past pricing hypothesis the team has ever resolved. Not the cleaned-up retrospectives. The raw record. The library supports search and filters by template category, PM, outcome, and date range.

The learning library. Every resolved public hypothesis becomes a searchable artifact with the full causal chain, the original prediction, the outcome marks, and the PM’s learning note.

The personal side mirrors the team view at smaller scale. A PM’s calibration page shows their headline calibration percentage, the magnitude accuracy stat alongside it, accuracy broken down by claim position (so the PM can see whether their predictions break at the behavior step, the metric step, or the business-outcome step), and the inconclusive rate.

What I shipped wrong and corrected

I want to name three things I shipped with confidence and had to delete or rewrite in this section.

My headline visualization did not work at real-world data volumes. The original PM dashboard featured a calibration curve, confidence buckets 1 through 5 on one axis and hit rate on the other, with the ideal diagonal as the reference. Every PM-calibration article in existence features this chart. Of course it had to be the hero. Then I ran the math against a realistic team’s data volume and realized a meaningful curve needs roughly 20 to 30 resolved hypotheses per bucket. That is 100 to 150 resolved hypotheses minimum. PMs ship 4 to 12 meaningful bets a quarter, so the curve I had made the hero of the dashboard reaches usefulness around quarter six. I deleted it and replaced it with a confidence-by-outcome grid that is useful from the first resolved hypothesis. Rows are confidence 1 through 5. Columns are Held/Wrong Magnitude/Broke/Inconclusive. Cells are integer counts. Reads like a small balance sheet. Scales with data rather than assuming it.

I shipped two metrics on the same screen that contradicted each other. My accuracy-by-position panel counted only HELD claims in the numerator. The headline calibration percentage counted HELD plus WRONG_MAGNITUDE. A PM looking at the dashboard saw 50 percent in one panel and 75 percent in another, both labeled accuracy, with no way to reconcile them. The math read correctly in isolation. It read wrong in composition. I had not reviewed the dashboard as a whole the way a user would read it.

I collected redundant data. The pre-mortem’s kill criteria question asked for metric, threshold, and deadline. The deadline duplicated the review date. I had been collecting wasted data into the schema for the entire build. The fix took a short time to fix. The lesson, every field in the form must be distinct from every other field, was free, and I should have applied it on day one.

None of these were edge cases. They were the central numbers on the central screens. I had shipped them with confidence because the mockups were beautiful and the seeded fixtures looked populated. I had not run the math against production-realistic data volume. That is exactly the failure mode the product is supposed to fight against.

What this taught me, in one sentence: shipping is the first draft. The dashboards a PM builds before there is data are themselves hypotheses, and a lot of those are wrong.

What I changed my mind about

I started this project thinking the unit of measurement should be the hypothesis. Did the bet hold or break? I changed my mind to the claim. Each link in the causal chain marked separately.

Here is why: “The feature did not move the metric” is not the same insight as “the behavior did not change at all.” The first means your theory of what drives the metric is wrong. The second means your feature did not land. A PM who shipped a feature that changed behavior but did not move the metric is a PM whose mental model needs work. A PM who shipped a feature that did not change behavior is a PM whose feature needs work. Conflating them throws away the most useful information the tool can produce.

What I believe about the PM role

The PM role have a forecasting part that needs to be marked in order for a team to learn from. We make consequential predictions every week (this onboarding change will lift activation 8 points, this pricing tier will not increase churn under 20 seats) and the predictions vanish into Slack threads and roadmap tickets. The PM moves on. The prediction has no scoreboard.

Tracking our own calibration changes how it feels to make a prediction. You can no longer say I think this will lift retention without remembering that the system will check you. Over time the looseness drops out of your reasoning, because the looseness will be the part that breaks.

I now think the PM hiring conversation should look very different. Tell me about a feature you shipped should be replaced by tell me about a prediction you made and how it resolved. A PM who can describe their last three honest miscalibrations is a different candidate than a PM who can describe their last three launches.

A note on the build

I built Hypothesis Lab using Claude Code as a pair programmer. Every line is AI-assisted. The economics of that build are real: 14 templates, two dashboards, OAuth for Google and Microsoft, an in-app notification system, a learning library, multi-team support, and three rounds of UI polish, shipped in a window that would not have been possible solo without AI.

The product decisions are a different story. AI ships features. AI does not decide which features should not exist. The one decision the whole product turns on (mechanical scoring, no override, anywhere) is a decision I had to make six times against six different temptations to soften it. AI did not help me hold the line. It helped me build the product the line produced.

If anything, AI-assisted building makes product discipline more important, not less. You can ship things faster than you can think them through. The bottleneck moves to judgment. Every Phase 3 correction in this case study was a judgment failure I would have caught earlier with a teammate to argue with.