| Pitfall | What it looks like | Why it matters for ML models |
|---|---|---|
| Peeking / early stopping | Stopping a test as soon as it looks significant | Inflates false positive rate, especially with automated model rollout pipelines checking results continuously |
| Network interference | Treatment group’s behavior spills into control group | Common in recommendation, marketplace, and social models where users interact with each other |
| Novelty effect | Short-term engagement spike that fades | New model outputs feel different at first, inflating short-window metrics that revert later |
| Offline-online gap | Better offline metric, flat or negative online metric | Offline metrics optimize a proxy, not the actual business outcome the test is meant to validate |
| Sample ratio mismatch | Traffic split does not match configured allocation | Signals a broken experiment before any result should be trusted at all |
Why A/B Testing ML Models Is Harder Than It Looks
Testing whether a new machine learning model is actually better than the one currently in production sounds like a straightforward experimentation problem: split traffic, measure a metric, compare the two groups. In practice, ML models introduce failure modes that a simple button-color test rarely has to deal with. Models influence downstream user behavior in ways that can leak between test arms, their apparent quality during offline evaluation frequently does not translate to the metric a business actually cares about, and the statistical machinery underneath the whole exercise breaks in subtle ways when teams check results too early or too often.
None of this means A/B testing is the wrong tool for validating ML models, quite the opposite. It remains the most reliable way to know whether a model change actually helps before it ships broadly. But treating it as a plug-and-play tool without accounting for these ML-specific pitfalls produces results that look statistically clean and are still wrong. The pitfalls below are the ones that recur most often across teams shipping model changes, roughly in the order a careful reviewer should check for them.
Peeking and Early Stopping Bias
Peeking, checking a test’s results before it reaches its planned sample size and stopping early because the result already looks significant, is one of the most common and most damaging mistakes in experimentation generally, and it is worse for ML model rollouts because monitoring dashboards make peeking effortless. Statistical significance calculations assume a fixed sample size decided in advance; checking repeatedly and stopping as soon as a p-value crosses a threshold inflates the true false-positive rate far beyond the nominal 5 percent most teams assume they are working with.
The fix is not simply discipline, though a pre-registered sample size and stopping rule helps. Sequential testing methods designed specifically to allow valid early stopping, and Bayesian approaches that produce a probability of superiority rather than a binary significance threshold, both offer more principled ways to look at results continuously without inflating false positives. The important discipline is deciding which approach you are using before the test starts, not switching frameworks mid-test because the data is not cooperating with the story you wanted to tell.
Choosing Between Frequentist and Bayesian Approaches
The frequentist approach, fixed sample size, a single significance test at the end, is simple to reason about and easy to audit, but it is unforgiving of any deviation from the pre-registered plan; peek once and the guarantees fall apart. Bayesian methods, which express results as a probability that one variant beats another given the accumulated data, tolerate continuous monitoring far better because they are not making a one-shot decision against a fixed threshold. The tradeoff is interpretability for some stakeholders: a p-value is a familiar, if often misunderstood, concept, while a posterior probability of superiority requires a bit more explanation to a business audience unfamiliar with Bayesian reasoning.
Neither approach is universally superior, and the right choice often comes down to how the organization operates. Teams that genuinely commit to a fixed test duration and do not look at results until the end can use standard frequentist tests without issue. Teams that want or need to monitor results continuously, because rollout decisions are time-sensitive or because stakeholders demand visibility throughout, are much better served adopting a sequential or Bayesian framework from the start rather than bolting continuous monitoring onto a frequentist test that was never designed for it.
Network Interference and the SUTVA Violation
Standard A/B test statistics assume the Stable Unit Treatment Value Assumption, commonly abbreviated SUTVA, which requires that one user’s assigned treatment does not affect another user’s outcome. This assumption holds reasonably well for isolated, individual experiences, but it breaks down constantly for ML models embedded in social networks, marketplaces, or any system with interaction effects between users.
| Setting | How interference happens | Mitigation |
|---|---|---|
| Social network feed ranking | A treatment user’s changed posting or engagement behavior affects what control users see | Cluster-based randomization by social graph community rather than individual user |
| Two-sided marketplace | Incentivizing treatment-group drivers reduces rides available to control-group drivers | Geo-based or market-based randomization instead of user-based |
| Recommendation systems with shared inventory | Promoting an item more in treatment reduces its availability or visibility for control | Time-based switchback experiments rather than concurrent user splits |
The practical takeaway is that the correct unit of randomization for an ML experiment is not always “the user.” When interaction effects are plausible, randomizing at the level of a geography, a time window, or a cluster of connected users prevents the treatment from contaminating the control group through the very interactions the model is designed to influence. Skipping this step does not make the interference disappear, it just makes the measured effect size wrong, usually biased toward zero because the control group has been partially treated without anyone noticing.
Novelty Effects and Primacy Effects
A new model’s outputs often feel different simply because they are new, independent of whether they are actually better. Users notice a change in recommendations, in ranking, in generated content, and that novelty alone can produce a short-term engagement spike that has nothing to do with the model’s real quality and everything to do with curiosity. The inverse, a primacy effect, can also occur when users are initially resistant to a changed experience before adapting to it.
The standard way to detect either effect is to segment results by user tenure: compare the treatment effect for brand-new users, who have no prior expectation to violate, against long-tenured users, who are more susceptible to novelty or primacy reactions to a changed experience. If an effect is strong for new users but fades or reverses for existing users over the test window, that is a signal the initial read on the metric was inflated by novelty rather than durable model quality.
Figure: Novelty Decay Curve
A treatment effect that spikes in the first days of a test and steadily decays toward a smaller, stable effect over subsequent weeks is the classic signature of a novelty effect rather than a genuine, durable improvement.
The Gap Between Offline Metrics and Online Business Impact
Perhaps the most consequential pitfall specific to machine learning is the gap between offline evaluation metrics and the online business metrics a test is actually meant to validate. A model can show a clear improvement in an offline metric, higher AUC, lower log loss, better ranking precision, and still produce no improvement, or even a decline, in the online business metric that matters, revenue, retention, task completion. Optimizing for the wrong metric is worse than running no test at all, because it creates false confidence that a shipped change is a genuine win when it may actively be hurting the outcome the business cares about.
This gap exists because offline metrics are almost always proxies. A ranking model evaluated on offline click-through prediction accuracy is not directly measuring whether users are happier with the ranked results, retention improves, or revenue grows; it is measuring how well the model predicts a historical click pattern that may itself have been shaped by the previous model’s biases. Closing this gap requires treating the online A/B test result as the actual source of truth and the offline metric purely as a cheap, fast filter for deciding which candidate models are even worth the cost of an online test.
Sample Ratio Mismatch: The Test You Should Always Run First
Sample ratio mismatch, commonly abbreviated SRM, occurs when the actual observed split of users between control and treatment does not match the configured allocation, for example a planned 50/50 split that actually lands at 54/46. SRM does not measure whether the model is good or bad; it measures whether the experiment itself is trustworthy at all. Common causes include buggy randomization logic, a hash function with poor distribution properties, bot traffic that disproportionately lands in one arm, or uneven ramping schedules where variants are rolled out to different user cohorts at different times.
A chi-squared test comparing observed to expected allocation is the standard SRM check, and it should run automatically before any team looks at the actual outcome metric. An experiment with a detected SRM should be treated as invalid regardless of how compelling the headline result looks, because whatever caused the imbalance is very likely also biasing which users ended up in which group in ways that have nothing to do with the model being tested.
| SRM cause | How it’s typically found | Fix |
|---|---|---|
| Broken or biased hash-based randomization | Chi-squared test flags a split significantly different from configured allocation | Replace the hashing function or randomization seed and rerun |
| Bot or automated traffic skew | Anomalous allocation concentrated in specific segments or time windows | Filter known bot traffic before computing allocation and outcome metrics |
| Uneven ramp-up across cohorts | Allocation ratio drifts over the test window instead of staying constant | Ramp all arms on the same schedule, or restart the clock after ramp-up completes |
Common mistake
Teams check the headline outcome metric first and only investigate sample ratio mismatch or novelty effects after a surprising result prompts suspicion. Running the SRM check and segmenting by user tenure should be the first two things any team does before interpreting any A/B test result, not a diagnostic step reserved for results that already look wrong.
What worked
Teams that pre-registered their stopping rule, sample size, and primary metric before launching a model test, and that automated the SRM chi-squared check as a gate before any dashboard displayed the outcome metric, reported far fewer post-hoc disputes about whether a result was real.
Multiple Comparisons and Metric Fishing
ML experimentation platforms often surface dozens of secondary metrics alongside the primary one, and the temptation to declare victory based on whichever metric happened to move favorably is strong. Running many comparisons without correction inflates the chance that at least one metric shows a spurious significant result purely by chance, a risk that grows the more segments, metrics, and sub-populations a team slices the same test data into. Pre-registering one primary metric before the test begins, and treating every other metric as directional and hypothesis-generating rather than confirmatory, keeps this risk contained.
Frequently Overlooked Details
- Pre-registrationCommitting to a primary metric, sample size, and stopping rule before launch prevents the test design from silently adapting to favor whichever result appears first.
- Ramp schedulesGradually increasing traffic to a new model arm can itself introduce sample ratio mismatch if not synchronized carefully across all arms.
- Tracking lossBrowser privacy protections and tracking prevention mechanisms increasingly cause silent gaps in client-side experiment data, especially across sessions or devices.
- Segment heterogeneityAn average treatment effect can hide a model that helps one user segment while actively hurting another.
- Long-run holdoutsA small permanent holdout group that never receives model updates helps distinguish cumulative model drift from genuine long-term treatment effects.
Designing Tests That Hold Up to Scrutiny
| Design choice | Recommended practice |
|---|---|
| Primary metric | Chosen and documented before the test launches, tied to the actual business outcome, not a proxy |
| Sample size and duration | Calculated in advance based on minimum detectable effect, not extended reactively when results are inconclusive |
| Randomization unit | User, cluster, geography, or time window, chosen based on whether interference between arms is plausible |
| Stopping rule | Sequential testing or Bayesian framework agreed on before launch if continuous monitoring is required |
None of these choices are exotic statistical theory; they are closer to basic experimental hygiene. The teams that consistently ship model changes with confidence are not the ones with the most sophisticated statistics, they are the ones that make these decisions before a test starts rather than justifying them after the results come in. A short written test plan, covering the primary metric, sample size, duration, randomization unit, and stopping rule, reviewed by someone other than the person running the test, catches most of these pitfalls before a single user is exposed to the new model.
How This Connects to the Rest of the ML Lifecycle
A/B testing sits downstream of model monitoring, since a monitoring alert or observed drift is frequently what prompts a retraining and retest cycle in the first place, a connection covered in our deep dive on model monitoring in production. The features a model consumes should also be consistent between the test arms, which is where a properly implemented feature store matters, discussed in our feature stores explainer. Reproducibility practices make it possible to trust that the model actually deployed in each test arm matches what was intended, covered in our reproducible ML guide. For teams testing causal claims beyond a simple randomized experiment, our piece on causal inference for product decisions covers complementary techniques for situations where a clean randomized test is not possible.
Glossary
- Peeking
- Checking a test’s results before reaching the planned sample size and making a stopping decision based on that early look, which inflates the true false-positive rate.
- Sample ratio mismatch (SRM)
- A statistically significant deviation between the configured traffic allocation and the actual observed split between test arms, signaling a broken experiment.
- Novelty effect
- A temporary change in user behavior caused by the mere fact that something is new or different, independent of whether the change is actually an improvement.
- SUTVA (Stable Unit Treatment Value Assumption)
- The assumption underlying standard experiment statistics that one unit’s treatment assignment does not affect another unit’s outcome, frequently violated in networked or marketplace systems.
- Switchback experiment
- An experimental design that alternates the treatment on and off across time windows for the same population, used when interference between concurrently assigned users is a concern.
Key Takeaways
- Peeking at results before reaching a pre-registered sample size inflates false positives far beyond the assumed significance threshold.
- Network interference violates the Stable Unit Treatment Value Assumption in social, marketplace, and shared-inventory systems, requiring cluster, geo, or time-based randomization instead of per-user splits.
- Novelty and primacy effects can inflate or depress short-term results; segmenting by user tenure helps distinguish them from durable effects.
- Offline metric improvements do not guarantee online business-metric improvements, because offline metrics are almost always proxies for the outcome that actually matters.
- Sample ratio mismatch should be checked automatically before any outcome metric is trusted, since it signals a broken experiment rather than a real effect.
- Pre-registering a primary metric, sample size, and stopping rule before launch prevents both peeking bias and multiple-comparisons metric fishing.
- The correct randomization unit depends on whether interference between test arms is plausible, not a default assumption that user-level splitting always works.
FAQs
What is peeking in A/B testing and why is it a problem?
Peeking is checking test results before reaching a planned sample size and deciding to stop based on that early look. It inflates the true false-positive rate well beyond the nominal significance threshold, because repeated looks at accumulating data increase the chance of catching a random fluctuation that looks significant.
Why does network interference matter for ML model testing?
Many ML systems, social feeds, marketplaces, recommendation engines with shared inventory, violate the assumption that one user’s treatment does not affect another’s outcome, contaminating the control group and biasing the measured effect, usually toward underestimating the model’s true impact.
How can teams detect a novelty effect in test results?
Segment the treatment effect by user tenure, comparing brand-new users against long-tenured users; an effect that is strong initially but fades or reverses over the test window for existing users indicates novelty rather than durable model quality.
Why can a model improve offline metrics but fail online?
Offline metrics like AUC or log loss are proxies for the real outcome, not the outcome itself; a model can better predict a historical pattern without actually improving the business metric, such as revenue or retention, that the online test is designed to validate.
What is sample ratio mismatch and why should it be checked first?
Sample ratio mismatch is a mismatch between the configured and actual observed traffic split between test arms, and it should be checked before any outcome metric is examined because whatever caused the imbalance likely also biases which users ended up in which group.
What randomization unit should be used for a marketplace or social product?
When user interactions can spill treatment effects into the control group, randomizing at the level of a geography, time window, or connected cluster rather than the individual user prevents the contamination that per-user randomization would introduce.
How does the multiple comparisons problem affect ML experiment dashboards?
Dashboards that surface many secondary metrics increase the chance that at least one shows a spurious significant result purely by chance; pre-registering a single primary metric before the test begins keeps this risk from inflating false confidence in a result.
Should a test ever be extended past its planned duration?
Extending a test reactively because results are inconclusive undermines the pre-registered sample size and stopping rule that make the statistics valid; if longer test durations are genuinely needed, that decision should be built into the original design rather than added after seeing early data.
- KDnuggets, A/B Testing Pitfalls: What Works and What Does Not with Real Data
- Statsig, A/B Testing for ML Models: Best Practices
- LatentView, AB Testing Validity Checks and Best Practices
- AB Tasty Documentation, Sample Ratio Mismatch
- Atticus Li, Sample Ratio Mismatch (SRM): The A/B Testing Quality Check Most Teams Skip
- Monetate, How to Fix Sample Ratio Mismatch in Your A/B Tests
- Kameleoon, How to Avoid Common Data Accuracy Pitfalls in A/B Testing
