When I joined Relay we shipped every week and tested nothing. When I left we ran forty experiments a quarter. This is how a team of nine got there without a growth team.

Start with one

The first test was a notification copy change. It was small, it was boring, and it lost. We read the result out on Friday with the loser's name on it, which was mine. That Friday reading became the most important meeting in the company.

The template

One page. A hypothesis in one sentence, the metric, the sample size we needed, the date we would decide. Anyone could fill it in; most weeks the engineer who built the feature did.

The rules

• Every test gets a written decision, even a boring one
• A test with no sample size is not a test
• Losers are read out with the same voice as winners

Forty a quarter

By the end of the year the template was a habit and the Friday reading had a waiting list. Forty tests a quarter sounds like a lot; it is three a week, most of them small.

A test you do not write down is a story you tell later, and the story is always better than the test.

What changed

Half of what we were sure about turned out to be wrong. That is not a sad number. It is the number of arguments we stopped having.