Direct answer
What to do first
Begin with the decision plan, not the lift shown on a dashboard. Confirm the intended audience, traffic allocation, tracking and test version are sound. Then judge the predefined primary outcome alongside guardrails such as lead quality, revenue, cancellations or performance—only the measures that matter for this change. Read the size and uncertainty of the effect against the smallest change worth acting on. If the result is inconclusive, do not relabel it as a win or loss: retain the control, revise the hypothesis, repair measurement or rerun only if the decision is worth the added exposure.
Start with the decision, not the lift
A result page can put a percentage lift at the top and still leave you without a sound business choice. Start by replaying the reason for the test. What did the change propose to improve, for which users, and what would the team do if the evidence supported it? The primary outcome should already be a decision criterion, not the metric selected after an attractive chart appears. NN/g recommends defining primary metrics with guardrails, and Microsoft advises setting metric trade-offs before a specific test can influence them [1] [3].
Write the decision in plain language before you interpret the scorecard: implement the change, retain the control, investigate a risk, or test a revised idea. That line keeps the readout from becoming a tour of every number available. It also exposes gaps early. If nobody can say what size of change would be worth the effort or what downside would block implementation, the dashboard cannot settle the decision for you.
The pre-result record does not need a bureaucracy of documents. It needs enough detail for another person to understand what was supposed to happen and what evidence the team agreed would matter.
- The hypothesis and the exact version that was tested.
- The primary outcome, its denominator, and the audience the decision applies to.
- The guardrails and the kind of regression that would change the decision.
- The smallest effect worth the implementation cost or risk, plus the planned analysis and stopping basis.
Clear health questions before interpreting outcomes
Do not explain a conversion movement until the experiment itself looks credible. Microsoft recommends checking that the users in the scorecard match the intended population and that each variant’s user count aligns with the planned allocation. It also warns that a treatment can change logging or telemetry in one variant, which can make a metric harder to interpret as a treatment effect [1].
A sample-ratio mismatch is one health check. It means the observed traffic split is unexpectedly different from the split you configured. That can point to an implementation problem, filtering, bots, or another external influence. Treat it as a question to resolve, not a reason to panic or a license to ignore allocation. Optimizely says an imbalance does not automatically make a result useless, while severe unexplained imbalance can degrade experiment results [2].
Run the check at the level where the decision is made. A clean overall split does not prove that every filtered scorecard, triggered population, or metric denominator is comparable. If the health evidence is not trustworthy, pause the result discussion. The next action is diagnosis, not a story about why the variant won or lost.
- Check that the scorecard contains the intended audience and unit of analysis.
- Compare observed variant counts with the allocation you expected.
- Confirm the tested experience, tracking events, event definitions, and denominators are comparable across variants.
- Record collisions, release changes, outages, or targeting changes that could affect only one variant.
Read the primary outcome as a business signal
A label such as significant is not a business verdict. Put the observed primary outcome beside the baseline and the uncertainty produced by your analysis method. Then ask a more useful question: does the evidence distinguish between outcomes that would lead to different actions? Sample planning depends on the baseline, the smallest effect worth detecting, and the statistical approach; none of those inputs has a universal value [3].
For a rate, look beyond the ratio. Microsoft notes that a changed denominator can make rate movements contradictory or difficult to interpret, and recommends checking the denominator as its own metric. If a checkout conversion rate changes, inspect the relevant orders and eligible sessions before assuming the experience itself caused the movement [1].
An illustrative example—not a client result: a variant may show a higher request-submission rate while the number of people reaching the form changes at the same time. The rate alone cannot tell you whether the form became more persuasive, the audience changed, or measurement changed. The readout should name the observation, identify the competing explanations that remain, and avoid converting a correlation into a cause.
- Show the baseline, observed primary-outcome values, absolute change, relative change, and uncertainty from the chosen method.
- State the smallest effect that would be commercially useful rather than treating every detectable movement as valuable.
- For rate metrics, inspect the numerator and denominator separately before assigning a cause.
Treat guardrails as terms of the decision
A primary outcome answers the narrow question the test was designed to answer. Guardrails check whether that movement creates an unacceptable cost elsewhere. For a lead-generation change, a team might need a quality or sales-stage guardrail; for a purchase flow, it might need a revenue, cancellation, support, or performance measure. The right list depends on the change and the business, not on a generic dashboard template. NN/g gives the same principle in practical terms: a higher interaction rate may not be a positive business result without related guardrail evidence [3].
A primary gain paired with a guardrail regression is not a mechanical pass or fail. Microsoft describes this as a trade-off problem and says there is no standard way to evaluate every trade-off. Agree the decision owner and the acceptable trade-off before the result is available, then document why the result did or did not meet that condition [1].
This prevents a familiar failure mode: one team celebrates the metric it owns while another inherits the consequence. A usable readout keeps both movements visible and makes the decision rule explicit.
- Choose guardrails that match the likely downside of this particular change.
- Check whether a guardrail movement is real, interpretable, and material to the decision.
- Use pre-agreed ownership when the primary outcome and a guardrail point in different directions.
Give an inconclusive result a precise next action
Inconclusive should not be a polite way of saying nothing happened. It can mean the remaining uncertainty still includes effects worth acting on, the planned analysis could not resolve the question, or a health check prevented interpretation. Those are different situations. None justifies relabeling the variant as a winner or a loser.
If the experiment was healthy and the evidence does not support a useful improvement, keep the control while you decide what to learn next. NN/g recommends retaining the original design after an inconclusive test and considering another variation; it also cautions that A/B tests show how behavior changed, not why it changed [3]. Use research, session review, customer feedback, or product evidence to improve the hypothesis rather than rerunning the same weak question by habit.
When a result is surprising or hard to trust, reproduction can be the right next step. Microsoft describes rerunning with re-randomized users when an unexpected finding remained doubtful after an initial review [1]. The business case matters: repeat a test to resolve a decision with real value, not to manufacture certainty around a minor preference. Close the readout with the chosen action, the evidence behind it, the unresolved uncertainty, and the person responsible for the next step.
- Healthy but unresolved: retain the control and log what the test did not settle.
- Useful hypothesis, insufficient resolution: revise the alternative or bring in qualitative evidence before deciding whether another test is justified.
- Health failure: correct the setup or measurement issue before rerunning, if the business decision still needs an experiment.
- Clear and usable evidence: make the named implementation decision and retain the record of the hypothesis, metrics, and trade-offs.
Common questions
What should an A/B-test readout include?
Include the original hypothesis, the exact change, intended audience, primary outcome, guardrails, planned analysis basis, actual allocation, and the observed effect with its uncertainty. Add the data-quality checks that passed or failed, then state one decision: retain the control, implement, investigate, or rerun. Primary and guardrail metrics serve different purposes; both belong in the readout [3].
Does a sample-ratio mismatch invalidate an A/B test?
Not automatically. A sample-ratio mismatch means the observed traffic distribution differs unexpectedly from the intended split and can point to an implementation issue or another external influence. Investigate the cause before relying on the scorecard. Optimizely cautions that an imbalance alone does not make every experiment result useless, but a severe unexplained imbalance can degrade results [2].
What should we do with an inconclusive A/B-test result?
First say why it is inconclusive: the uncertainty may still cover effects worth acting on, the planned analysis may not resolve the question, or a health check may have failed. Keep the control while you decide whether to improve the hypothesis with qualitative evidence, fix measurement and rerun, or stop because the answer is not worth more exposure. An inconclusive result is not a win for either version [1] [3].
Sources and context
- [1]Patterns of Trustworthy Experimentation: Post-Experiment Stage
Microsoft Research guidance on checking scorecard audience and allocation, telemetry integrity, denominator movements in rate metrics, metric trade-offs, and reproducing doubtful results before a ship decision.
- [2]Optimizely's automatic sample ratio mismatch detection
Official Optimizely documentation defining sample-ratio mismatch, describing its potential causes and impact, and cautioning that an imbalance is not automatically grounds to discard every result.
- [3]A/B Testing 101
NN/g guidance on linking an A/B-test hypothesis to primary and guardrail metrics, sizing a test around the baseline and minimum detectable effect, interpreting practical significance, and handling inconclusive results.
