Relay was shipping every week but nobody knew which of the shipped things worked. I set up the experimentation program on the side of the app work.

The setup

• One test a week at first, run by whoever shipped the feature
• A one-page template: hypothesis, metric, sample, decision
• A Friday reading where results were read out, including the losers

The number

Forty tests a quarter by the end of the year, and a written decision for every one of them.

A test you do not write down is a story you tell later.

What it taught me

Half of what we were sure about was wrong. The other half we could now prove.

How it ran

The program had no budget and no headcount, so it had to be cheap. The flag system was a config file, the sample-size calculator was a spreadsheet, and the reading was twenty minutes on a Friday with coffee. The engineers ran their own tests because the template was short enough to fill in while the build was running. By the second quarter people were asking to be on the Friday list, and by the third the product decisions in the planning meeting came with a test number attached.