
p < 0.05 is (probably) stupid
Ya boy is now back on the job market! Mostly I’m taking Data Science (DS) interviews, though I’m not 100% certain that that is what I want to go back to. Regardless, because I am doing these interviews, I need to refresh my statistical/experimental toolkits because I haven’t thought about any of that for over a year.
And since the only way I (re)learn is by doing, I am studying by building simulations. If I can generate simulations that illustrate concepts like Alpha, Beta, Confidence Intervals, etc. from the ground up, then I understand them. As I was doing this, I kept getting stuck on 2 simple questions:
- Why do we experiment at all? Is it even a good idea?
- Why do we use p < 0.05 as the success criteria for experimentation? Is it optimal?
The average DS may read those questions and scoff, because it’s obvious how to answer them. But the average DS is an idiot (me included), and can’t reason from first principles! In addition, as AI accelerates feature development at all technology companies, some of the fundamental beliefs underlying current experimentation philosophy is worth reexamination. So let’s start from the beginning and develop the fundamentals needed to evaluate these questions.
By the end, I hope to convince you of 2 things:
- Running experiments is not always the right decision, but it USUALLY is - shipping without experimenting works if you believe almost all your features are good, and pre/post comparisons are sufficient if your metric is not noisy and unaffected by other influences. However, those are big (and probably wrong) assumptions to be making
- Analytics/Experimentation teams may be making the wrong tradeoffs - If you experiment at your company, your ship criteria may undervalue Beta error and overvalue Alpha error! Simulations in this article suggest that optimizing for Beta while keeping experiment length the same could improve your ability to move metrics by ~15%!
Let’s get started!
Back to Basics: why experiment?
Experimentation should be used at companies because companies care about winning - so experimentation must somehow help with winning. Implicitly, the decision to use p < 0.05 (we’ll cover what this means later) as a cutoff for experiments implies that this is the best strategy for winning.
But what does it mean to win, and how can experiments help do that?
In my mind, winning is about long term profit potential. And companies make decisions and ship changes that are in service of this long term profit potential. The strategy of winning is different for every company, but every company cares about different short term metrics and goals that they think are proxies for the long term win and builds features that are aligned with these short term proxies.
So in reality, experimentation should be in service of optimizing critical metrics and goals. A common criticism of experimentation (that, frankly, I agree with more often than you might expect) is that it’s simply a time drag. And a mistake that proponents of experimentation often make is that they religiously claim that experimentation is required to know what’s right and wrong.
In reality, we must view all ship decisions for a product/business as a tradeoff between 2 things:
- Velocity - how quickly is a feature built, and how long does it take to get it out to the world?
- Knowledge - how accurate is our understanding of the impact of that feature?
The value of velocity is obvious - if you have good ideas you should get them out the door to improve metrics ASAP. But the ‘good’ in that sentence is doing some heavy lifting!
Knowledge is our understanding of the goodness of an idea. When it comes to knowledge, teams can often think they know the impact of ideas before they’re launched. Or even more simply, they know which ideas are good, so everything they ship will be good.
Unfortunately, when companies actually measure this empirically, this belief turns out to be misguided. Many teams have much lower success rates than they’d expect and/or do not predict unforeseen negative impacts on other metrics they care about. Section 5 of Kohavi et al.’s 2009 paper from Microsoft illustrates this and gives us a baseline: ⅓ of ‘good’ ideas have positive impacts, ⅓ have no impact, and ⅓ have negative impacts.
So experimentation, when done correctly, is a tradeoff that a company makes by sacrificing some velocity, in order to try and only ship positive features. And there are many different tradeoffs that could be made! You could have:
- A pure velocity CEO
- A pure knowledge CEO
And everything in between! Since this is a spectrum you can go from a no experimentation CEO, to a CEO who experiments but with loose guardrails, to a very strict CEO that takes extra time to maximize knowledge when making decisions.
But when chatting with Analytics professionals vs. non-Analytics professionals, the conversation often feels like a binary: test everything with industry/academic standards, or don’t test at all.
I want to challenge this duality. To do so, we’ll build up our intuition of statistical concepts and focus our discussion on the themes of Knowledge vs. Velocity. We end with some estimates of ‘optimal’ experimentation strategies, along with the hidden tradeoffs present with any strategy.
Why we’re (usually) so bad at measuring impact
Let’s go back to a statement I made earlier: that teams are often woefully wrong about knowing if features will have positive impacts. Oftentimes, teams will launch a feature and then compare performance on key metrics before and after as a way to guard against this (also known as pre/post ‘testing’). Let’s develop an intuition around why/when that works, and why/when that fails.
We’re gonna make a simulated company - doesn’t matter what it does, but let’s say it cares about a single metric: conversion.
Imagine that this company has a dashboard, where they monitor their conversion rate. They’re shipping features one at a time, and using pre/post comparisons to estimate impacts. For now, we’ll use Kohavi’s estimates as a starting point, and say that every feature that they build has an equal chance to be: positive, neutral, or negative on conversion.
We want to evaluate how well this company detects good/bad features. An important input into this is how stable/noisy their conversion metric is. After all, in a world where their conversion rate is extremely predictable (every day it’s exactly 10%), it’s very easy to detect real changes!
We can see this in action with our first chart, representing a world where the Conversion metric for the company is very stable and not noisy:
Scenario 1: stable metric
Pre/post reads 98% correctly
(across 300 simulated dashboards)
Pre/post reads 98% correctly
(across 300 simulated dashboards)
Each triangle represents a feature that was launched, and the coloring shows if the company thought that the feature had a positive, negative, or neutral impact when looking at their conversion rate (blue) Pre/Post. The matrix on the right shows the distribution of feature launches by their actual impact vs. their estimated impact based on the data. Flip on reveal the truth to see which reads were right.
We can see that because the metric is very stable (look at the grey baseline), the pre/post estimates are almost always correct! This is unsurprising - we essentially have perfect understanding of the metric (it’s just gonna be flat, with a little wiggling over time), making it very easy to confidently measure changes in that metric because of our features.
The story gets more complicated when we evaluate a scenario where the metric is still stable (stays consistently around 10% as a baseline), but it’s very noisy:
Scenario 2: noisy metric
Pre/post reads 48% correctly
(across 300 simulated dashboards)
Pre/post reads 48% correctly
(across 300 simulated dashboards)
This time, the company is often wrong when it tries to determine the impact of any given feature. This is because the noise of the metric results in shifts which are large relative to the treatment effect we’re trying to measure! It’s hard to determine if any launch had any particular effect because the metric itself is moving wildly on its own. The company will often think that launches with real impacts (truly positive or truly negative) are actually having no impact because they can’t see through the noise.
We can quantify this by looking at how accurate the Company’s estimates of its features are as a function of how much noise there is relative to the effect they’re trying to measure:
Pre/post accuracy as a function of noise
In a low noise case (maybe the metric has daily fluctuations of about 1%, but the company is launching features that move the metric by 10%+), Pre/Posts are very accurate because it’s so obvious what the impacts are. However, the more noise there is in the metric, the less reliable these estimates become, until they are effectively useless.
The story gets even worse if, in addition to the metric being noisy, it is also impacted by things outside of the company’s control. For example, here’s a scenario where conversion is also moving due to seasonality, competitor actions, and a bunch of other things:
Scenario 3: noisy metric, many drivers
Pre/post reads 40% correctly
(across 300 simulated dashboards)
Pre/post reads 40% correctly
(across 300 simulated dashboards)
The company is having an even harder time understanding the true impacts of their features, because they cannot disentangle the impacts of unrelated factors from the impact of their features. There is lots of academic work around how you can control for these external factors when analyzing tests in order to try and be more accurate, but those approaches are nuanced, difficult to get right, and well outside the scope of this article (free resources here, here, and here).
Of course, Pre/Post accuracy doesn’t matter in and of itself! What we actually care about are the decisions we make because of what we think a feature did. So let’s take the above scenarios, and look at how accurate/inaccurate estimates impact what is or is not shipped, and therefore the Company’s ability to improve their metric.
We are plotting 3 lines in these charts:
- The baseline when no features are launched (grey)
- The optimal outcome, where the company magically knows all the features that work best, and launch only those as quickly as possible
- The actual outcome, where the company measures which features work using pre/posts, and only ships the ones that have positive impacts
Scenario 1: stable metric
Unsurprisingly, in the stable metric scenario when the Pre/Post analyses are very accurate, the company is performing optimally.
But the story changes drastically when looking at the noisy scenario.
Scenario 2: noisy metric
Because the company is often incorrectly estimating the true impacts of features, they are often:
- not shipping good features
- and mistakenly shipping bad features!
We can see below how metric noise greatly degrades the Company’s performance relative to its optimal performance:
% of total uplift achieved, as noise increases
So, these simulations show us that:
- If every idea you have is good, then just ship everything! In this scenario, the value of measurement is to estimate which ideas are best to help guide resourcing decisions and the future roadmap
- If you have a very stable, not noisy metric, Pre/Post measurements work great
Unfortunately, in real life metrics are often noisy and complex, and not all ideas will work. Pre/posts are the ‘cheapest’ knowledge you can buy; they have no extra technical requirements and potentially no velocity headwind. However, they only provide accurate knowledge if:
- your features have large impacts compared to the fluctuations of your metric
- and if the metric does not move due to other factors!
The more uncertainty you have in understanding why/when your metric moves, the more incorrect you will be.
This is a value proposition of experimentation - it reduces the sorts of mistakes present in pre/posts, at the cost of some velocity headwind. To understand how, let’s dive deeper into the exact types of errors we can make.
Alpha, Beta and Knowledge
In statistics, we have special names for the types of mistakes we made when trying to analyze the pre/post experiments above. Let’s go back to the matrix of outcomes from the previous section, but with extra labels:
In the matrices above 3 of the 9 boxes (green) capture scenarios when we are correct. The other 6 boxes are different types of errors we made in interpreting the impact of our feature. These 6 boxes can be summarized as:
- Alpha Error - our feature had no impact, but we thought it did something
- Beta Error - our feature had an impact, but we thought it didn’t do anything
- S Error - our feature had an impact, but in the opposite direction of what we measured (for example, we thought it was positive, but in reality it was negative). We’re gonna ignore this type of error for the rest of the article, but I included it here to be exhaustive.
You may be thinking, “great! Experiments allow us to ignore these errors, which is why we run them!”.
WRONG!
Alpha and Beta error are unavoidable parts of life! Just like how in the previous section, pre/post tests would get things wrong because of variance and complexity in the real world, experiments must also deal with the same Alpha/Beta errors.
Understanding Alpha
The reason for this is because experiments are random. They rely on creating 2+ groups for comparison:
- a control group, which does not get some change (your feature) that you’re measuring
- a treatment group(s), which does get the change
Different groups allow us to compare the performance of the treatment vs. the control in order to estimate the impact of the change.
An assumption we’re making is that when we create these two groups, they will be ‘the same’, so that any change in how the treatment group acts can be due to the change we introduced and not due to underlying differences between the treatment/control groups of people.
But, we must remember that individuals within the groups have different behaviors, and random assignment into groups could result in differences in the groups independent of whatever treatment you’ll apply. This could violate the ‘sameness’ assumption that we want to make. Let’s consider an example:
- You have 1000 users, where average conversion is 50%. This means of the 1000 users, 500 would convert, and 500 would not
- You take those 1000 users, randomly split them into two groups (a ‘control’ group and a ‘treatment’ group) and do NOTHING else
When you do this, by random chance, you may end up with a random ‘control’ group where no one was going to convert, and a ‘treatment’ group with everyone that was going to convert. You’d see a treatment conversion rate of 100%, vs a control rate of 0%! Even though your experiment did nothing, you may declare the treatment wildly successful (Alpha Error)!
We can conceptualize the risk of this by calculating the probability that something like this would happen due to random chance. We do this by simulating a ton of ‘fake experiments’, also known as A/A tests. An A/A test involves us
- randomizing users into treatment/control groups
- calculating the difference in conversion between the two groups (importantly, we don’t actually do anything to the treatment group)
- tracking of how often we see different amounts of difference between the groups
When we do this over and over again (20K times!) for our hypothetical example above (500 users in each group, 50% conversion rate), we get this distribution of potential outcomes:
Treatment vs. Control: differences when there's no effect
One way to think about this chart is that, on average, if we randomly create treatment and control groups, there will be 0 difference (that’s why the center of the distribution is around 0). But, due to random chance, we may see relatively large differences between the groups!
You may be wondering what those vertical lines in that chart are - those bounds are where Alpha Error = 5%! Essentially: given your sample size (500 users in the treatment group) and conversion rate of 50%, there is a 5% chance that you will have a treatment group with a conversion rate that is 6.2% different (absolute) from the control!
Alternatively, this means that if you wanted to use this experiment to measure a change in conversion of at least 6.2%, there is a 5% chance that the treatment group would show a difference in conversion at least as big as that even if the feature you were testing didn’t do anything. In this scenario, we may conclude that your feature had an impact, even if it didn’t. This is why Alpha is often called the “False Positive Rate”.
Alpha and p-values (p)
Alpha is directly tied to a common rule in experimentation: a result is called significant when its p-value (p) is below 0.05. The 0.05 in that rule comes from experiments usually utilizing an Alpha of 0.05. In reality, the rule is better generalized as p < Alpha.
P-values are surprisingly nuanced, so I’m not going to dive deep. If you’re curious I recommend reading up on this.
For this article, the most important thing to note is that because p < 0.05 is such a widely accepted decision criteria, it means that most people usually use an Alpha of 0.05 when making decisions. And based on the title of this article, clearly I think this is not ideal! To understand why we must now learn about Beta Error.
Understanding Beta
To conceptualize Beta, we need to think about a slightly different scenario: what if our treatment ACTUALLY has a real impact? Let’s go back to our conversion example again, but now our random sampling compares the 1000 people we had originally, vs those same 1000 people in an alternative reality where they ALL got the treatment. In the real world, we don’t have access to this alternate reality (if we did, we wouldn’t have to experiment!). If our treatment had a TRUE effect of increasing conversion rate from 50% to 55%, then in this “alternate reality”, 50 of the 500 people who didn’t convert in the “baseline reality” are now converting.
This does imply that there is some chance that, when we create treatment/control groups (which only have 500 people), all 50 of the ‘new converts’ don’t get added to the treatment group! In that scenario, we would incorrectly assume that the treatment has no impact!
Let’s go back to those simulations we did earlier to understand Alpha, but with a twist. This time we create random treatment/control groups with 500 people each, but source from two different populations:
- We get 500 people for the treatment group from the hypothetical treatment population (alternate reality where everyone got the treatment)
- We get 500 people for the hypothetical control population (baseline reality where no one gets the treatment)
- We calculate the difference in conversion rate between the two groups, and track the differences
Again, we repeat this 20K times and plot the distribution of differences generated by this random approach:
Treatment vs. Control: measured differences when there's a real effect (+5pp)
This time, the vertical lines are where Beta error = 0.64, meaning we have a 64% chance of thinking that the treatment did nothing, when it had a real impact!
You may be wondering why I plotted an arbitrary Beta error of 0.64. Well, the reason becomes clear if you overlay the Beta Distribution plot over the Alpha distribution plot:
Treatment vs. control: alpha vs. beta distributions
The right line showing the Alpha cut-off is at the same location as where the Beta cut-off is. This is intentional - it shows that the Alpha threshold and Beta threshold are actually the same cutoff. When you design an experiment, moving your Alpha threshold moves your Beta threshold! (Try it: change Alpha above and watch one error shrink while the other grows.)
In fact, when you design an experiment, you actually have 3 quantities you can control:
- The Alpha error you’re willing to risk
- The Beta error you’re willing to risk
- Sample Size (n) of your experiment
In our simulations above, we utilized an Alpha of 0.05, and a Sample Size of 500 (per group). When using these parameters, we’re saying that we don’t want to say that the feature had a significant effect unless the impact is larger than the Alpha threshold (6.2% in this example). This implies that if there is a real effect that is smaller than this, we won’t accept it as being real (Beta error).
It should be noted that because these inputs are interrelated, you could reduce rates of Alpha AND Beta error by increasing sample sizes. Here’s what the same plot above looks like if instead you had 5000 people in each group instead of 500:
Treatment vs. control: 5,000 users per group
More samples means big differences are less likely to happen due to random chance. So there is both:
- A tighter ‘Alpha Distribution’ - with 5000 users per group there’s a 5% chance you see a difference greater than 2% due to random chance, instead of the 6.2% change in the scenario where you only have 500 users per group
- A tighter ‘Beta Distribution’ - because the true 5% impact is well beyond the 2% Alpha thresholds, we’re unlikely to miss the impact of this change
We can generalize this by plotting a few curves showing how Alpha/Beta tradeoffs change as you change the sample size you utilize:
More data means fewer errors
But, you must remember that to increase sample sizes, you may need to run experiments for longer and longer! So the statistical power of an experiment comes at the tradeoff of velocity. Therefore, our choice of Alpha and Beta are inputs into velocity. This is where our theme of Knowledge vs Velocity comes back into this story, and illustrates how in experimentation, we must make tradeoffs:
- When experiments are designed, you have to estimate, “What impact do we think this feature will have?”. For example, a +5% improvement to conversion
- Then, using that estimated impact, you determine Alpha and Beta Error rates you are willing to risk. The textbook approach is to utilize an Alpha of 0.05 (remember p-value < 0.05?) and a Beta of 0.2
- Using those and your historical data, you can estimate how long your test will need to be. Reducing Alpha and/or Beta requires lengthening the test
The flip side is that if you’re willing to risk higher error rates, you can reduce the sample size you need. Therefore, by relaxing statistical guardrails, you can make experiments faster.
…wait, what’s the point of experimentation?
I have just spent a few paragraphs explaining why experiments can’t get rid of the Alpha/Beta errors that pre/post tests already suffer from, and that experiments require large sample sizes (and therefore time) in order to work. So, if you are a curious individual, you will (rightfully) be asking, “what’s the fucking point of experimenting if it has the same errors that pre/post tests have?”.
Well, dear reader, you get two advantages with experiments:
- You can control for external factors that move your metric (remember scenario 3 from earlier?)
- You know the amount of error you are willing to accept in advance
In the pre/post examples we discussed earlier, we are only able to show those 3x3 matrices because we simulated the data, and therefore, know the true impacts a priori. In the real world, you would have no idea if the true impacts match what you observed in your pre/posts. Your metric might be moving because of some external factor or because of noise in the metric. You would be shipping things without any understanding of how often you’d be correct and/or accurate. You could be inadvertently hurting your business, yet have no idea that you’re doing so!
So in reality, experiments are a way to quantify the confidence in our knowledge of what we’re doing, rather than a guarantee of knowledge. And, it takes more samples (and therefore, more time) to increase the confidence in our knowledge, which comes at the cost of velocity.
In the next section, I detail a conceptual framework to understand when (if ever) experiments generate ‘enough’ knowledge to be worth the velocity headwind, and how that may help us understand the ‘right’ Alpha/Beta tradeoffs to be made.
Company Simulations
In order to do this, let’s extend our simulations even further. Instead of simulating single experiments, we’re now going to simulate whole companies.
Each company will:
- Have one key metric (conversion) that they care about, and infinite feature ideas that they want to launch to improve this metric
- For simplicity, the team considered every idea to be equally good (ie. no expected differences in treatment effect)
- It takes the company 14 days to build a feature, and if they choose to test the feature, there will be some amount of time to test the feature
- If they are testing using pre/posts, then the next feature launch has to wait until the previous feature has finished testing. This is to ensure that the pre/post comparisons are ‘clean’, and multiple launches don’t overlap with each other.
- If they are testing with A/B experiments, they do not have to wait because multiple A/B experiments can be launched simultaneously
- We’ll start with a world where every idea from the team is good, and compare that to a world where some ideas are neutral or bad
For each company, there will be one of 5 possible CEOs, each with a different psychology/philosophy:
- Ship EVERYTHING: speed is critical, let’s not worry about measurement and just assume everything works
- Pre/Post Tester: use pre/posts to measure success
- Textbook Experimenter: lets ship experiments based on best practices in academia
- Beta Minimizer: we must launch positive features (low Beta), even if we incorrectly ship neutral features (high Alpha)
- Strict Experimenter: as a tech business, I can often run longer experiments to get very precise data
Before simulation, let’s see how CEO Psychology impacts their timelines for feature launches. Here’s a simulated Gantt chart, showing the build/test timelines for 5 features for each CEO:
Build and test timelines for the first 5 features
- build
- test (colored per CEO)
Defining Success
To evaluate how well the different CEO Psychologies perform under these different scenarios, we will measure 2 things after simulating each company for a year:
- What is the end uplift in their metric, because of what they shipped (using the true impacts of those features)
- The cumulative incremental conversions that their actions drove for an entire year
The former helps us align our simulations around the fundamental thing businesses care about: winning. Specifically, winning by improving key metrics as much as possible.
The latter is a bit more subtle, but we measure it to reward speed. Testing incurs a cost because we can’t launch the feature to 100%, meaning we don’t capture the full benefit of the feature during testing. In addition, testing a feature that is having a negative impact also has a cost, because it is hurting the company during the duration of the test. To help conceptualize this, let’s look at an example scenario, and how we calculate this cumulative incremental conversion metric:
Measuring Cumulative Incremental Conversions
Scenario 1: Only Good Features
Let’s start our simulations with a world where every feature created by the company has positive impacts on conversion.
Distribution of Feature Impacts
For every simulation and CEO, we run 300 simulations in order to make sure that performance is not impacted by randomness. In the Only Good Features world, unsurprisingly we see that the best possible strategy is to just ship everything:
Conversion Rate (average)
This is obvious - if every idea works, then what’s the point of wasting time with measurement? It should be noted that the Beta Minimizer does capture most of the uplift, indicating that that strategy is still quite effective in this world because it doesn’t sacrifice much velocity while obtaining lots of knowledge.
Cumulative Incremental Conversions (average)
I was a bit surprised at the poor performance of the Pre/Post Tester, but the reason for their struggles is due to the fact that their Pre/Post measurements have LOTS of Beta error:
CEO Error Rates
Ship Everything
correct: 100%
Pre/Post Tester
correct: 33%
Textbook Experimenter
correct: 82%
Beta Minimizer
correct: 93%
Strict Experimenter
correct: 89%
This CEO is measuring things, but their Pre/Post tests do not have enough statistical power to declare that their features are having an effect. So the Pre/Post CEO is often assuming that their features are not having an impact, and not shipping them. If this CEO also decided to ship neutral features, then they would be performing as well as the Ship Everything CEO.
Scenario 2: Include Bad Features
The story becomes more interesting (and more realistic) when we look at a world where not every feature is good. In fact, based on Kohavi’s Microsoft paper, we assume that ⅓ of ideas are positive, ⅓ have no impact, and ⅓ have a negative impact:
Distribution of Feature Impacts
When we use this distribution of potential impacts for features, we see a different conclusion from the earlier Only Good Features simulation:
Conversion Rate (average)
Because good and bad ideas cancel each other out in this world, the Ship Everything CEO does very poorly. They are shipping good ideas, and immediately undoing their hard work by shipping bad ideas. We also see that the Beta Minimizer is now the best performing CEO. In fact, we see that they generate about +14% incremental conversions throughout the year relative to the Textbook Experimenter and the Strict Experimenter:
Cumulative Incremental Conversions (average)
This is because the Beta Minimizer is:
- Shipping more good features than the Textbook Experimenter
- Shipping faster than the Strict Experimenter
In other words, the Beta Minimizer is the CEO that is appropriately balancing the benefits and tradeoffs of velocity vs knowledge.
We can see that although they are sometimes shipping features which actually don’t do anything, they are not missing features (Beta) as often as the other types of CEO:
CEO Error Rates
Ship Everything
correct: 34%
Pre/Post Tester
correct: 54%
Textbook Experimenter
correct: 83%
Beta Minimizer
correct: 84%
Strict Experimenter
correct: 90%
We should note that the optimal strategy is very sensitive to a key unknown: how good are the ideas that the company is having? In a world where every idea is good, there’s little point to test. However, in a world where good and bad ideas are equally likely and impactful, shipping everything is a horrible strategy.
When does testing beat shipping everything?
It turns out that all it takes is for 7% of ideas to be bad for the Beta Minimizer to actually outperform the CEO that doesn’t test at all. Of course, another assumption here is that the bad ideas are as negative as the good ideas are positive. I don’t think this is likely for a company that only cares about a single metric, so take these insights not as a rule, but a rough baseline for how perfect your team’s ideas need to be in order to outperform a Beta Minimizer approach.
I was also curious how different strategies performed as you changed development time required to build features. As AI speeds up software development, does Beta Minimization always outperform or is the story more nuanced?
Cumulative Incremental Conversions as Engineering Speed changes
The patterns we saw in earlier simulations were very stable: Beta Minimizers always outperformed. When we normalize these charts so that we can show how different Experimentation Philosophies perform relative to the Textbook Experimenter, we see this:
Normalized Performance
We see that no matter how much dev time it takes to build a feature, the Beta Minimizer will generate about 13-15% more incremental conversions relative to the Textbook Experimenter. This makes sense, since the Beta Minimizer is running experiments at about the same speed as the Textbook Experimenter, but with Beta Error rates being reduced from 20% to 5% (-15%!).
Why is Beta Minimization so effective?
What we consistently see is that the best performing CEO balances knowledge and velocity by utilizing experimentation, but focusing on minimizing Beta while relaxing Alpha constraints.
Fundamentally, I think this makes sense: the Beta Minimizer doesn’t miss out on real impacts, yet does so while still moving quite fast. Yes, they are shipping more features that don’t do anything, but in the simulation there is no cost to doing so. You could argue that the complexity cost in the real world (due to having so many features) is costly, but I suspect that the complexity cost of those features is low relative to the value brought by shipping good features fast.
This is why the title of this article throws shade at the common ship criteria used for experiments at companies; p < 0.05 implies the importance of Alpha while Beta is left unloved.
Cumulative incremental conversions after a year
- Textbook Experimenter
- Beta Minimizer
- Strict Experimenter
It should be noted that we haven’t actually explored a wide combination of Alpha/Beta parameters that an Experimenter could use. We picked Alpha = 0.3 and Beta = 0.05 out of a hat! What if Alpha = 0.2 and Beta = 0.07 is more optimal?
So, I created a series of heatmaps showing how the Cumulative Incremental Conversions after a year and # of experiments shipped changes based on different combinations of Alpha and Beta.
What we end up seeing is that the Beta Minimizer approach of Beta = 0.05 and Alpha = 0.3 works pretty damn well! The heatmaps do suggest that using an Alpha of 0.5 might be even better, but take these insights with a grain of salt. Like with all simulations this is an approximation of the real world. How effective this strategy is will depend on the business context, and the costs of shipping False Positives. And even for me, declaring that we should use Alpha of 0.5 seems questionable.
However, these simulations do suggest that the Textbook Approach that we as an industry tend to adopt unthinkingly may be very far from optimal!! Instead of p < 0.05, we may want to move towards p < 0.3 with much smaller Beta errors!
Holy shit Imesh, can you just let me leave
Ok fine, I’m done beating the corpse of p < 0.05. Let’s come back to the 2 main takeaways I want you to internalize:
- Running experiments is not always the right decision, but it USUALLY is - shipping without experimenting works if you believe almost all your features are good, and pre/post comparisons are sufficient if your metric is not noisy and unaffected by other influences. However, those are big assumptions to be making
- Analytics/Experimentation teams may be making the wrong tradeoffs - If you experiment at your company, your ship criteria may be undervaluing Beta error and overvaluing Alpha error. Think through the hidden implications of your ship criteria: are you paying the right price (velocity) for the desired amount and correct type of knowledge?
Given the lessons from the simulations, if I designed an analytics/experimentation function, I would focus on:
- Ensuring that all experiments are conceptualized as a knowledge<>velocity tradeoff, and not just one or the other
- Defaulting to faster experiments with relaxed Alpha requirements but stricter Beta requirements, while making exceptions for ‘important’ experiments
What’s next?
So why did I caveat the title of this article with (probably)? At the end of the day, the sort of “meta” evaluation of ship criteria and experimentation philosophy in this article is just the beginning! Our simulated companies only cared about a single metric.
In reality, there are 3 dimensions of complexity that could cause the ‘optimal’ experimentation strategy to be different:
- Companies care about many metrics - what is the right decision making framework when you have 100s of metrics, some are success criteria, others are guardrails, each has different sensitivities, and each team is only incentivized to care about their metric?
- Knowledge and discovery - a benefit of more accurate experiments is that it allows for more accurate resourcing discussions and discovery of more fruitful areas of opportunity. On the flip side, faster testing may be critical due to the potential of outlier ideas (recommend reading A/B Testing with Fat Tails by Azevedo et. al.)
- Experimentation impacts goaling - how does the choice in experimentation philosophy impact how teams take accountability for their metrics? For the Beta Minimizer CEO, they must ensure that relaxed experimentation guidelines do NOT reduce ultimate accountability for metrics
This article is already quite long, so I’ve left these nuances out. But I have good news if you got all the way here and want to read more: part 2 is coming! It will explore these nuances as I try to develop an understanding of how to build an ‘optimal’ knowledge discovery machine within an organization.
As always, if you have any questions/thoughts/concerns, reach out to me on LinkedIn, or at imesh.c.samarakoon@gmail.com. And if you want to explore these simulations yourself, here’s the code:
Run the simulations yourself
No Python experience required!
Install Python — skip if you already have 3.10+
Grab the installer from python.org/downloads. Need help? The official guide for your OS walks through it:
Download the kit
All the files for the simulation in a handy zip file:
p-less-than-5-kit.zipWhat's inside
experiment_sims/config.pythe "true" world every simulation starts fromexperiment_sims/stats_core.pyz-tests, power and sample-size formulasexperiment_sims/alpha_beta.pyA/A and A/B tests: measuring alpha and betaexperiment_sims/dashboard.pypre/post on a noisy dashboardexperiment_sims/company.pysimulating whole companies under five CEOsexperiment_sims/plots.pythe charts, in matplotlibexplore.ipynba notebook with sliders for every chart: run this!requirements.txtthe python packages you need
Set up your environment
Unzip it, open a terminal inside the folder, then:
cd p-less-than-5-kit python3 -m venv .venv source .venv/bin/activate pip install -r requirements.txtcd p-less-than-5-kit python -m venv .venv .venv\Scripts\activate pip install -r requirements.txtRun the simulation
Launch the notebook:
jupyter labOpen
explore.ipynb, then Run → Run All Cells. You'll get every chart from this article with sliders attached: change the noise, the traffic, the share of bad ideas or a CEO's alpha and beta, and watch who wins.
Open this article on desktop to grab the simulation code