The Experiment You Shouldn't Run

4 MINS

Working in retail and pricing, I've sat through a lot of experiment reviews. Over time I've started to notice a pattern that nobody warns new PMs about: the most expensive experiments are the ones you should never have run in the first place. Not the ones that lost. The ones that, in hindsight, asked a question that wasn't worth asking.

This is a short note on how to spot them before you run them.

Three signs the experiment is wrong

I now ask myself three questions before approving an experiment design. If any of them returns a vague answer, the experiment goes back into the freezer.

What decision will this change? If the answer is "we'll see" or "it depends", we're not running an experiment. We're running a curiosity. Curiosities are fine — but call them what they are, and don't spend a sprint of engineering on them.

What's the smallest version of this we'd ship if it works? Teams that can't answer this often don't have a feature; they have a hope. Hopes don't survive launch reviews.

What does losing teach us? A good experiment is informative either way. If "we lose" leaves you no smarter, design something else.

The seductive bad experiment

There's a specific kind of bad experiment that retail PMs love and shouldn't: the "will this lift conversion?" experiment with no theory of why. It's seductive because the metric is unambiguous and the engineering cost feels low.

But you almost always end up in one of two bad places:

It "wins" by 0.2%, statistically significant, and now you've committed your platform to a small, unexplained behavior that is genuinely hard to reason about later.

It "loses" by 0.3%, and you can't tell if the idea is wrong, the implementation is wrong, or the segment is wrong — so you re-run it three more times. The fix is not to run more experiments. The fix is to start every experiment with a one-paragraph theory, in plain English, that someone outside your team could read and immediately argue with.

When to skip experiments entirely

Sometimes the right answer is to not run an experiment at all. I've found two situations where shipping straight is better:

The decision is reversible and small. If you can roll back in an afternoon, ship the change, watch the dashboards, and skip the ceremony.

The cost of the experiment exceeds the cost of the wrong decision. This sounds obvious, but PMs ignore it constantly. If a 3-week experiment guards a 1-day rollback, the experiment is the expensive option. The opposite is also true: there are situations where an experiment is a moral requirement, not a methodological one. Anything that touches price visibly, anything regulated, anything that affects merchant trust — those need experiments, even if the team is impatient.

A small closing note

I love experiments. I run a lot of them. But the most underrated PM skill in retail right now is the ability to look at a proposed experiment and say, calmly: "this isn't worth running. Here's the question we should be asking instead." That conversation is hard. It is also the most expensive piece of advice I will ever give a junior PM.

The cheapest experiment is the one you confidently decide not to run.

From Analyst to Staff PM, in Slow Motion

Nabendu skipped presentations and built real AI products.

Nabendu Goswami was part of the March 2026 cohort at Curious PM, alongside 17 other talented participants.