Ben Yoskovitz – Measure What Matters – MicroConf 2013

The MicroConf Hub Page with links to all the notes for all the talks can be found here. 

Speaker: Ben Yoskovitz (@byosko)

Watch the video (sorry, but I’m not allowed to embed it here)

What I have learned

• It is easy to get zombified (you get used to your life and don’t improve anymore)
• Recruitment sucks
• Startup accelerators are fun (managing one that is)
• I am not an analytics expert; but I wrote a book on analytics anyway

The basics of Lean Startup

• Ideas (Everyone’s idea is the best – right?)
• Build (People love this part – especially developers)
• Product
• Measure (This is where things fall apart)

What I hope you get from it

• The importance of intellectual honesty (be honest to yourself about the success of your product)
• Using your gut properly
• Better decision making abilities
  • Everyone has data, the key is figuring out what pieces will improve your learning and decision making
• Focus (Don’t chase shiny objects. Success without focus is just an accident)

Measure What Matters

• What makes a good metric?
  • Measurement of movement towards your business goals
  • comparative
  • understandable
  • ratio or rate (absolute values often worthless)
  • changes your behaviour (If a metric doesn’t change your behaviour, it is a bad metric)

​​Types of Metrics

• Qualitative (warm and fuzzy) vs. Quantitative (cold and hard)
  • Discover qualitatively
  • Prove quantitatively
  • AirBNB experiment: Professional photography will result in more bookings
    • Test by sending 20 photographers in the field
    • 2-3x number of bookings
    • now build that into your product
• Exploratory (gets you new insights) vs. Reporting (neccessary to know)
• Lagging (historical metric; churn) vs. Leading (prediction of the future; customer complains – which lead to churn)
• Correlated vs. Causal (Ice Cream Consumption and Drownings are correlated – both caused by season!)
• Don’t just follow the leader – monthly subscriptions don’t work for everyone!

Lean Analytics Stages

• Empathy (I’ve found a real, poorly-met need)
• Stickiness
• Virality
• Revenue
• Scale
• Skip steps at your own risk

How it All Comes Together

• Choose only one metric and draw a line in the sand
• Example: Paid Churn = Paid Cancellations over Paid Signups
• Some interesting benchmarks
  • Growth (5% / week – revenue or active users)
  • Engaged visitors
  • Time on site (may be vanity, if user spends 20 minutes in help files)
  • page load time ( < 5 seconds – else you are loosing people)

Lean Analytics Cycle

1. Pick the KPI (Key Performance Indicator)
2. Draw a line in the sand
3. Find a potential improvement
   1. Without data: make a good guess
   2. With data: Find a commonality
4. Hypothesis
5. Design a test OR make changes in production
6. Measure the results
7. Did we move the needle?
   1. Yes: Bring out the Champagne!
   2. No: Pivot or give up!
8. Start over at 1.

-Christoph