What is A/B Testing?
Comparing two or more variants on randomized audiences to measure causal impact on a chosen metric. Valid tests need a written hypothesis, sufficient sample, and guardrail metrics so wins do not hide damage elsewhere.
Example
A team tests a one-step versus three-step signup on randomly split traffic, watches completion as the goal with support-contact rate as guardrail, and ships only after the sample covers full weekly cycles.
What people get wrong
Peeking at results daily and stopping at the first significant-looking day. Early stopping inflates false wins; pre-commit to sample size and runtime before launching.
Frequently asked questions
How long should an A/B test run?
Long enough to cover full weekly behavior cycles at the planned sample size — calendar time matters as much as visitor counts.
Related terms
Hypothesis
A falsifiable prediction written before an experiment: changing X will move metric Y by a stated direction. Hypotheses force teams to name the mechanism, which makes both wins and flat results interpretable.
Randomization
Assigning users to test variants by chance so groups differ only randomly. Sound randomization is what lets an A/B result claim causation instead of correlation with pre-existing user differences.
Guardrail Metrics
Secondary metrics watched during experiments to catch harm the primary goal misses: latency, error rate, support contacts, unsubscribe rate. A test that lifts conversion while spiking unsubscribes fails its guardrails.
Funnel Analysis
Measuring stepwise conversion through an ordered series of events to find where users drop off. Funnels turn vague activation complaints into a specific step with a specific leak to fix first.
Autocapture
Automatic collection of clicks, pageviews, and form interactions without per-element instrumentation. Fast to start but noisy: retroactive data comes with vague names and mystery events that explicit tracking would have labeled.
Explicit Events
Hand-defined tracking calls placed at meaningful product moments: signup completed, checkout started, invite sent. Explicit events cost engineering time upfront and pay back in clean names and trustworthy funnels.