Define Experiment Success Metrics

A framework for running successful business experiments by clearly defining success metrics and win conditions upfront, based on Shaan Puri's experiences.

Key Problems with Poorly Defined Experiments

• Using the word "experiment" incorrectly

  • Running tests without clear hypotheses
  • No defined success or failure conditions
  • Just trying random things without structure

• Not knowing what winning looks like

  • Can't properly assess if experiment is working
  • No benchmark metrics to compare against
  • Unable to make informed decisions to continue or stop

Real Example: Crystal Store Experiment

• Initial metrics looked promising

  • 1.7-2x return on ad spend (ROAS)
  • Sales coming in quickly after launch
  • Good customer response

• Failed due to poor win conditions

  • Thought 3-4x ROAS was required based on hearsay
  • No industry research on actual benchmarks
  • Shut down prematurely due to incorrect success metrics
  • Later realized 1.7-2x ROAS was actually good

Framework for Better Experiments

• Start with clear hypothesis

  • What do you believe will happen?
  • Why do you believe it will work?

• Define specific success metrics

  • Research industry benchmarks
  • Set realistic targets
  • Choose key metrics that matter

• Set timeframes and milestones

  • How long to run the test
  • What results needed at each stage
  • When to assess and make decisions

• Document learnings

  • Track what worked and didn't work
  • Use insights for future experiments
  • Build institutional knowledge

The key lesson is: Don't confuse a clear view for a short distance. Have clear metrics but be realistic about timelines and benchmarks.