“Startups that succeed are those that manage to iterate enough times before running out of resources”
Even though an iteration (in Lean Startup) is defined as a Build/Measure/Learn experiment, the iteration loop can be applied equally well to optimization experiments as well as pivot experiments which has often led to Lean Startups being mischaracterized as “optimization driven”. Eric wrote a post not too long ago to address this: “Learning is better than optimization” but I feel he didn’t go far enough to draw the distinction between when you should optimize versus pivot.
Unlike Eric, I do feel there is a place for optimization experiments (like the infamous Google 41 shades of blue test) – just not before Product/Market fit. Before Product/Market fit, the goal of a startup is finding a repeatable and scalable business model. Unless you are pointed in the right general direction, stepping on the gas here only gets you lost faster. You don’t “really know” if you are pointed in the right direction until Product/Market fit. Once you are, optimization is the right action.
How are Pivot Experiments Different from Optimization Experiments?
Yes, technically you learn something in both cases but the goal of pivot experiments is very different from optimization experiments. In a pivot experiment, you attempt to validate parts of a business model hypothesis towards finding a scalable and repeatable business model. In an optimization experiment, you attempt to refine parts of a business model hypothesis towards building a scalable and repeatable business model. The goal of the first is course correction (or a pivot). The goal of the second is efficiency (or scale).
Pivots are about changing direction, while Optimizations are about going faster.
The 3 Facets of Pivots
A pivot experiment is designed to validate parts of a business model hypothesis. I like to formulate my business model hypothesis along 3 questions: WHAT, WHO, and HOW:
1. WHAT: Problem
What is the problem you are trying to solve?
2. WHO: Customer
Who has the problem?
3. HOW: Product/Market
How do you solve this problem?
How big is this market?
How are you going to reach this customer?
How do you drive demand?
How will you make money?
The first pivot experiments you run are during Customer Discovery when you attempt to validate problem/solution fit with potential customers in the form of problem and product presentations. You use these results to course-correct your initial hypothesis. Beyond Problem/Solution fit, you test various aspects of HOW starting with how well you solve the problem.
4 Rules for Running a Pivot Experiment
Eric recommends focussing on the macro effect of an experiment (such as sign-ups versus button clicks) but this is required for both optimization and pivot experiments. The following additional rules help differentiate pivot experiments:
Rule 1: Maximize Learning
One of the key distinction between pivot and optimization lies in the expected outcome.
The Principle of Optimum Failure Rate:
Fifty percent failure rate is usually optimum for generating information.
Donald Reinersten – The Principles of Product Development Flow
This means you stand to learn the most when you don’t really know what to expect. If the goal is to maximize learning, you have to pick bold outcomes versus chase incremental improvements. So rather than changing the color of your call to action button, change the unique value proposition. Rather than experimenting with different prices, experiment with different pricing models.
Rule 2: Focus on Retention
While I agree that focussing on the macro is a good rule of thumb for running any experiment, unless you focus on the right macro metric, it is just as easy to lead yourself astray.
A startup’s life before Product/Market fit is very different from life after Product/Market fit and so are the key metrics. Of the 5 AARRR startup metrics, the only metric that most closely tracks Product/Market Fit is Retention. Yes, I rank it above Revenue because it is possible for a customer to buy a subscription, not use it, and simply forget to cancel. But it’s hard (if not impossible) to fake retention.
While getting paid is the first validation, getting customers to keep coming back is the ultimate validation towards building a repeatable and scalable business model.
Rule 3: Minimize Waste
Where things get a little gray, is that you need some initial and steady traffic to feed into your conversion funnel since Retention is towards the bottom. You can initially get by with customers you found during Customer Discovery but you eventually need to start testing customers through other channels like your website.
I like to start experimenting with Unique Value Proposition statements and signup flows but it’s important to watch out for the point when pivot experiments start looking like optimization experiments. Some judgement is required to know when you have “just enough” traffic to support learning. You eventually reach a point of diminishing returns which is usually a good sign to stop and focus further downstream. While squeezing out additional 10-20% improvements on landing page conversions or a new signup flow might sound like progress, it’s fairly easy to get sucked into the “optimization” trap which is highly addicting and a sure way to burn both time and money.
If you don’t yet have a product customers want, leading 100 versus 10 horses to the water won’t make them drink any more.
Rule 4: Validate Qualitatively, Verify Quantitatively
Before Product/Market fit, you typically do not have enough traffic to afford waiting for statistically significant results so I prefer to validate all my experiments in 2 phases. The first pass is qualitative which requires talking to customers. Picking bold outcomes work quite well with qualitative testing as they typically don’t require large sample sizes for a strong initial signal. The trick is building a validated learning loop into your experiments. I routinely use a combination of customer feature validation, usability tests, and product presentations to do this.
If I don’t get a strong initial signal, I kill the variant immediately. Otherwise, I setup a cohort to analyze quantitative data over time. Again the key metric I pay attention to is Retention. My products use a trial period so retention naturally converts to revenue.
Pivot Before Product/Market fit, Optimize After
I wrote this post because I’ve experienced first hand just how easy it is to unknowingly get caught up in the optimization trap even when you are supposedly practicing “Lean Startup” techniques. Until recently, I used to view pivots as big course corrections you make at some point X in the future when things don’t work or you stumble into an accidental discovery – like how PayPal, flickr, or Starbucks came to be. That’s because those were usually the best examples of pivots cited.
I now view a pivot as any iteration (big or small) towards validating a business model hypothesis. The size is usually relative to when the pivot is applied. For instance, pivoting on customer segment is easier during Customer Discovery than Customer Validation. I’ll share a case study on how (and why) I’ve been pivoting on 2 customer segments in parallel next time.
Words matter. The word iterate is too general, and the word optimize too dangerous (before Product/Market fit). So as a first step, I’d recommend removing it from your vocabulary till after Product/Market fit.