You're staring at your landing page, wondering whether the headline is the problem. Your offer seems useful, visitors arrive, and then too many leave without signing up. You've written a punchier alternative, but replacing the original feels like trusting a wizard who won't reveal what's in the potion.
You don't need a crystal ball. You need a fair duel, with the judging rules agreed before the first visitor arrives.
Table of Contents
- What A/B Testing Means in Marketing
- When A/B Testing Fits and When It Does Not
- Metrics to Track and How to Pick One
- How Much Traffic and Time a Test Needs
- How an A/B Test Runs From Start to Finish
- Misconceptions That Make Tests Lie
- Start Your First Test This Week
What A/B Testing Means in Marketing
A/B testing in marketing is a randomized controlled experiment that compares two versions of a page, email, or ad against one predefined outcome. The control (A) is your original version. The variant (B), also called the treatment or challenger, contains the change you want to evaluate.
That outcome might be click-through rate, conversion rate, revenue per visitor, or sign-ups. As Stanford Graduate School of Business explains, controlled digital experiments help distinguish the effect of a change from differences caused by audience mix or timing.
The repeatable answer to “What is A/B testing in marketing?” is this: it compares performance and, when properly designed, provides evidence about whether the tested change caused the difference.
What makes the duel fair
Random assignment means visitors don't choose their version, and you don't handpick who receives it. Your testing tool assigns them by chance, helping distribute different kinds of visitors across both experiences.
For a straightforward beginner test, change one clearly defined element. If you replace the headline, price, and checkout together, you can compare the complete experiences, but you can't identify which ingredient caused the result.
Think of matched potion batches brewed under the same moon, with a coin flip deciding which visitor receives which. Don't give the challenger enchanted spring water while making the control with swamp sludge.
The basic arrangement looks like this:
Say your invoicing software has a sign-up page. The control headline reads Save time on invoices, while the challenger reads Get invoices paid before lunch. Everything else stays unchanged, and sign-up rate, the share of visitors who register, is your primary metric.
A trustworthy result can support choosing a headline for the audience and conditions tested. It doesn't prove that headline is universally better, explain every visitor's motivation, or guarantee the same effect forever.
When A/B Testing Fits and When It Does Not
A/B testing fits when you have steady traffic, a measurable business outcome, and a change you can isolate. A page doesn't qualify just because you can imagine a prettier button.
A checkout with daily visits could test how it presents a price. A lead-generation page could compare its existing form against a shorter field list, while an email campaign with a sufficiently large list could compare subject lines. Each has a specific decision attached to an observable action.
Good candidates and awkward candidates
- Checkout price presentation: Test when purchase tracking works and the offer can remain stable throughout the experiment.
- Shorter lead form: Test when you can measure submissions and check whether the resulting leads remain useful.
- Email subject line: Test when the audience is large enough and recipients have time to respond before you need the decision.
- Sparse traffic: Don't prioritize a tiny adjustment on a page receiving only occasional visits. Gathering useful evidence may take longer than the decision is worth.
- Short-lived campaign: Don't start an experiment that can't finish before the seasonal offer or launch ends.
- Unmeasurable outcome: Repair tracking before comparing versions. A cauldron without measurement marks won't teach you much.
Adding challengers creates an A/B/n test. Optimizely's A/B/n guide explains that additional versions require more total traffic because every branch needs enough visitors for a dependable comparison.
Fit check: Choose an experiment you can finish while its answer still matters.
A complete redesign can be tested as a package, but it's a poor choice if your question is which individual element mattered. When controlled testing isn't practical, use customer interviews, usability feedback, and trend tracking to choose a more substantial improvement.
Testing is one part of conversion rate optimization, not its replacement. Before committing, calculate whether the page can support the sample your decision requires.
Metrics to Track and How to Pick One
Common test metrics and where they fit
| Metric | What It Measures | Best Used As |
|---|---|---|
| Conversion rate | Share of visitors who complete the goal | Primary metric for landing pages and checkout |
| Click-through rate | Share of viewers who click a link, button or ad | Primary metric for ads, emails and headlines |
| Sign-up or lead rate | Share of visitors who submit a form | Primary metric for lead-generation pages |
| Revenue per visitor | Total revenue divided by visitors in each group | Primary metric when price or offer changes |
| Average order value | Mean spend per completed order | Secondary or guardrail for upsell tests |
| Bounce rate | Share of visits that leave after one page | Guardrail for catching confusing changes |
| Refund rate | Share of sales later refunded | Guardrail for offers that inflate early sales |
One primary metric decides the experiment. Other measurements help diagnose what happened or act as guardrails, checks against harmful side effects.
Choose the primary metric according to the job you're asking the asset to do. Click-through rate can suit an ad or email when the immediate decision concerns generating visits. Sign-up rate suits a registration form, while revenue per visitor is useful when changing an offer could affect both purchasing frequency and spend.
Don't award the trophy for the wrong action
Suppose you enlarge a checkout button and make its wording more mysterious. More people might click to discover what happens, yet fewer might complete a purchase. Calling that a success would reward curiosity rather than sales.
Similarly, treat a faster page as a change whose commercial outcome needs measurement, not an automatic revenue victory. Watch the business result alongside engagement.
For broader campaign measurement, Purple Engage's explanation of using holdout groups to measure ROI offers useful context for comparing outcomes against a group that doesn't receive the intervention. The practical connection is choosing a comparison and outcome that answer the actual business question.
Before naming your primary metric, answer:
- Visitor purpose: What did this person arrive to do?
- Business value: What does the business gain when they do it?
- Potential harm: What could worsen without appearing in the primary metric?
Define how each event is counted before launch. Decide whether repeat submissions count, how purchases are attributed, and how long you'll allow delayed conversions or refunds to appear.
For bounce rate, use your analytics tool's documented definition consistently, since some tools measure engagement rather than single-page exits. Your guardrail is only useful when you know what it measures.
How Much Traffic and Time a Test Needs
Your minimum detectable effect (MDE) is the smallest change you design the experiment to detect. Choose it around an improvement worth implementing, not a lift so tiny that finding it would consume your entire marketing season.
For example, Optimizely's sample-size calculator illustrates that a 20% baseline conversion rate and a 10% relative MDE correspond to 2 percentage points, giving planning benchmarks of 18% and 22% around the baseline. Relative change and percentage-point change aren't interchangeable.
Those benchmarks describe the effect size you're planning around, not a magical significance boundary. An observed rate outside that range doesn't automatically establish a real effect, and a rate inside it doesn't automatically establish no effect. You still need the planned statistical analysis.
Set the strength of your detection spell
Adobe's sample-size guidance recommends commonly used design targets of 95% confidence and 80% power, with power meaning an 80% probability of detecting an effect at least as large as the planned MDE when it exists.
Confidence isn't the probability that your preferred headline is correct. It concerns the reliability of the statistical procedure under its assumptions.
Smaller effects require larger samples because you're trying to distinguish a subtler difference from ordinary variation. A low-converting funnel also generally needs more traffic than a high-converting funnel to detect the same relative lift, because conversions are rarer.
Planning rule: Calculate required visitors per variant before launch, then divide by the daily visitors that variant will actually receive.
Use an even traffic split for a straightforward comparison. Don't divide a per-variant sample requirement by all site traffic, especially when only some visitors reach the tested page.
For marketplace-specific planning, Headline Marketing Agency's guide to sample sizes for Amazon A/B tests provides additional context for sellers working within that environment.
Round the estimated duration up to full weekly cycles to include differing weekday and weekend behavior. Also allow time for the conversions you're measuring to mature.
For a first small-business experiment, use a practical go-or-no-go rule: if the estimate stretches beyond a few weeks, reconsider the idea. A substantially different offer may justify testing a larger effect; a button-color duel probably doesn't deserve an entire dragon's lifespan.
How an A/B Test Runs From Start to Finish
Return to the invoicing sign-up page. You're testing whether a benefit-led headline helps visitors understand the value well enough to register, not whether your team prefers lively copy.
A fixed-horizon test has an evidence target and evaluation point set in advance. Optimizely's statistical analysis overview describes planning the required sample before launch and evaluating after the required sample and minimum duration have been reached.
The workflow is simple, but each step protects a different part of the comparison:
Prepare the page and the judging rules
Write the hypothesis. Use A benefit-led headline increases sign-ups to connect a specific change to an expected behavior. Avoid vague ambitions such as making the page more compelling, which don't identify what you'll learn.
Choose the primary metric and guardrails. Use sign-up rate to judge the headline, with bounce rate as a check for confusion or mismatch. Don't switch to button clicks after discovering that registrations didn't improve.
Build the challenger. Replace Save time on invoices with Get invoices paid before lunch, keeping the rest of the page identical. Adding a new testimonial at the same time would prevent you from attributing the outcome to the headline alone.
Configure assignment. Split eligible traffic evenly and assign visitors randomly, preserving their version on return visits where possible. Don't send existing customers to one version and new prospects to the other, because that creates a different comparison.
Run, evaluate, and record
Set the evidence requirement. Calculate the necessary sample and choose a minimum duration covering full weekly cycles. Don't set an arbitrary deadline without checking whether your traffic can support it.
Let the experiment run. Monitor for broken forms, missing events, or serious business harm, but don't use an encouraging afternoon as permission to declare victory. Avoid changing the price mid-test, since that changes the conditions you intended to compare.
Make the planned decision. Evaluate sign-ups alongside the guardrails, then deploy the supported winner or record an inconclusive outcome. Don't force a winner just because the dashboard displays a higher number in one column.
For design-side preparation, Figr's A/B testing best practices can help you organize the challenger and review process. When choosing A/B testing tools, prioritize reliable assignment, event tracking, and understandable decision rules over a flashy winner badge.
Your finished report should state the hypothesis, sample, observed result, confidence interval or significance assessment, guardrail outcomes, and decision taken. Keep it readable enough that someone who didn't brew the potion can understand why you served it.
Misconceptions That Make Tests Lie
The most dangerous testing mistakes look convincing. A bright dashboard, an upward arrow, and a wizard's confident flourish can distract you from an unfair decision.
Keep these illusions in mind when reading a result:
Trick one, the early leader has won
Under a conventional fixed-horizon design, repeatedly checking and stopping at the first p-value below 0.05 can raise false positives above the intended 5%, as explained in research on sequential experimentation. Every additional opportunity to stop gives random fluctuations another chance to look persuasive.
The correction is to declare the stopping rule before launch. Properly specified sequential methods can support interim decisions, but ordinary repeated peeking isn't a substitute for them.
Trick two, no winner means identical versions
An inconclusive experiment may have too little traffic, an overly ambitious MDE, or a noisy outcome. Mastercard's statistical-significance guide explains why sample adequacy and stopping decisions matter when interpreting results.
Record uncertainty. Keep the control when the evidence doesn't justify a change, rather than claiming you've proved equality.
For another way to think through experimental comparisons, consider this explainer:
Trick three, the same winner suits everyone
Mobile visitors and desktop visitors may respond differently, as may people arriving from different channels or with different buying intentions. Research on subgroup-based treatment assignment examines how treatment decisions can account for customer characteristics.
But rummaging through segments until you discover a flattering result creates misleading findings. Choose important segments in advance, account for multiple comparisons, and validate promising differences independently before personalizing the experience.
Intuition deserves the same skepticism: Bing reported that fewer than one-third of its ideas improved the metric they targeted. An attractive idea is a candidate for testing, not evidence of success.
A winner must be statistically credible and commercially meaningful, improving the primary outcome without unacceptable harm to guardrails.
Start Your First Test This Week
Choose a page you already understand, not the most complicated funnel in your spellbook. A steady stream of relevant visitors and working conversion tracking are more useful than an ambitious redesign.
Keep the first experiment focused:
- Choose the page: Pick an existing sign-up or sales page with consistent traffic and an identifiable point of friction.
- Write the prediction: Use a concrete hypothesis such as Changing the sign-up headline to lead with time saved increases sign-up rate.
- Build the challenger: Keep the page unchanged except for the headline, then verify that registration and analytics work in both versions.
Give your automation clear boundaries
A split-test draft is a challenger version of the page with a new headline, layout, or offer. For this experiment, choose the headline alone, and watch bounce rate alongside registrations; if the offer leads to purchases, include refunds among your checks.
If you use our automated conversion testing, the Conversion Alchemist familiar creates, runs, and promotes A/B split tests automatically, promoting winning variants and retiring losing ones. Automation can handle drafting, traffic allocation, and reporting, but you still own the hypothesis, primary metric, and stopping rule. An early promotion still needs sufficient sample and statistical significance, not merely a temporary lead.
Use the landing-page split-testing guide to turn the idea into a page-level comparison. Record what you expect to learn before activating it, including what would make you retain the original.
Run the experiment if the page can reach its required sample within a few weeks. Otherwise, investigate a bigger change or gather feedback and track trends while traffic grows. Choose your headline, write your judging rules, and brew the first potion.
Web Mage builds websites and sales funnels from plain-English descriptions, with built-in split testing to compare challengers against your original page. Visit Web Mage to create your page and give your first conversion experiment a clear hypothesis, a measurable goal, and a fair duel.