Paywall A/B testing mistakes that pick the wrong winner
TL;DR:
- A cheaper price almost always wins on conversion rate. If conversion is your success metric, your test will tell you to lower prices and call it a win.
- Most bad paywall calls come from a short list of mistakes: the wrong metric, a window that ends before renewals, too few conversions, blended platforms and geos, and edits to a live test.
- Each mistake listed below comes with the fix.
You ran the test.
Variant B lifted paywall conversion by 11% so you shipped it to 100% of traffic… and three months later MRR is flat.
That’s because paywall tests carry a trap: the revenue effect of a change comes weeks after the conversion.
A user who picks your cheaper weekly plan looks like a win in week one and churns six weeks later.
Your test reads the first number. Your P&L reads the second.
Fixing the eight mistakes below is how you’ll align the two.
Judging pricing tests on conversion rate
Conversion rate counts taps on "Subscribe." It ignores how much people paid and how long they stayed.
For anything touching price, plan mix, or trial, you'll pick the wrong winner.
Apps in Adapty's paywall experiments playbook that raised a weekly price from $7.99 to $9.99 lost 5–8% of conversions and gained 18–22% in ARPU.
Score that test on conversion and you keep the cheaper price.
Glam AI, an AI photo editing app at $50M ARR, hit the same trap from the design side. A video hero on its main paywall lifted view-to-payment from 1% to 1.25%.
It also pulled more weekly plans and more refunds, and control won on net revenue per user.
The team kept control. Their rule: "Conversion is a diagnostic."
ℹ️What to do: Pick the primary metric before launch and write it into the test description. Use ARPU or net revenue per viewer for pricing, plan, and trial tests, with churn and refunds as guardrails, and keep conversion rate as a diagnostic
Calling the test before renewals come in
A design test shows its effect the moment a user taps. A pricing or trial test shows it at the first renewal, sometimes the second.
Trial tests make this obvious.
Shortening a trial from 7 days to 3 lifted trial-to-paid conversion by 12–18% on average in Adapty's data.
None of that shows in week one, because every user in both variants is still inside their trial. Stop at day 7 and you're comparing trial starts, the metric that tells you least.
Weekly and monthly plans have the same problem on a longer clock. A higher price can convert almost as well on day one and churn harder at the first renewal.
If you close the test before that renewal you'll never see the churn.
ℹ️What to do: Split traffic for 2–4 weeks on design tests. On pricing and plan tests, split for 4–6 weeks, then follow the cohorts for 2–3 renewal cycles and make the call on LTV.
Trusting a test with too few conversions
The number that matters is conversions per variant, and we find that apps don't collect enough in the first week.
A paywall with 20,000 views at 2% conversion gives you 400 purchases across two variants. At that volume, random noise alone can open a gap that looks like a 15% lift.
Aim for 200 subscriptions per variant as a floor and closer to 500 to detect lifts under 10%. At 2% conversion, 500 per variant means about 25,000 viewers in each group.
With 500 new users a day, that takes around seven weeks.
Peeking makes it worse. You check on day three, see variant B ahead, and stop. Early leads flip because the first users through a test skew toward your most engaged installs and toward the weekday you launched on.
ℹ️What to do: Calculate the sample before launch, write down the stop date, and act only when you hit both. If your traffic can't reach the sample in 30 days, test a bigger change: a new price point moves results far more than a new button color.
Testing button colors before price
Our experiment data shows pricing tests deliver 2–3x more uplift than visual changes, with the best pricing wins reaching 80%.
Copy and CTA tests tend to land at 5–10%. Yet most subscription apps still run on the price they set at launch and spend their traffic on button and badge tests.
Order matters for a second reason. A design winner found at the wrong price may stop winning once you change the price, because a different price draws a different buyer.
Real case: A trip planning app working with Adapty started with price. A 30% increase cost it only 5% of conversions. After four months of pricing, plan, and onboarding tests, ARPU was up 102% while installs went down.
ℹ️What to do: Test pricing and plan structure first, trial offers second, placement third, and visuals last. If your price hasn't changed in 12 months, your next test is a price test. Autopilot benchmarks your prices against competitors market by market, so you start from a real hypothesis.
Testing the paywall without the flow before it
The user who reaches your paywall has already made half the decision. Onboarding sets their expectations, and the timing of the paywall sets how much intent they bring. Test only the paywall and you optimize the final step while everything before it stays fixed.
You see this as tests that plateau: five paywall designs, and none beats control by more than a couple of points. The ceiling sits earlier, in an onboarding that never shows the product's value or a paywall that fires before the user has done anything worth paying for.
The opposite mistake happens too. A team changes onboarding and the paywall in the same release, sees revenue move, and can't tell which change did it.
Flow & Paywall Builder lets you build onboarding, quiz, and paywall screens without code, run flow-level A/B tests, and see drop-off per screen. Changes ship without an app release, so a new variant goes live in hours.
Blending platforms and countries into one result
One result across all users averages people who behave differently.
iOS users usually pay more than Android users.
A $9.99 weekly plan is mid-range in the US and premium in Brazil.
Say a higher price wins by 15% in the US and loses by 20% in Latin America: the blended result looks like a draw, and you keep the old price everywhere.
Winners don't transfer between placements either.
Glam AI added a $119.99 Business plan above its $59.99 yearly plan on the main paywall, and net revenue per user rose from $0.261 to $0.292. The same structure on inner paywalls had zero effect.
The team's takeaway: "A win belongs to the placement and the segment it was won in."
ℹ️What to do: Segment by platform when traffic allows and run separate tests for your top two or three revenue countries, grouping smaller markets by purchasing power. Re-test a winner before you roll it out to a new placement. Paywall targeting splits audiences by country, platform, or attribution source.
Changing a test while it's running
Edit a live test and you split your data into two experiments that your report blends into one. Common versions: fixing a typo in variant B on day four, moving the split from 50/50 to 80/20 because B looks strong, or launching a paid campaign in one country halfway through.
Overlapping tests do the same damage more quietly. Run a price test and a layout test on the same audience and each user lands in one of four combinations. A layout might win only at the higher price, and neither report will show it.
Setup bugs are the hardest to spot. A variant that fails to load and falls back to the default paywall skews results in ways that look like real user behavior.
ℹ️What to do: Freeze the test once it launches. If you find a bug, stop and restart with fresh traffic. Run parallel tests only on separate placements or segments, and run an A/A test with two identical paywalls before your first real test on a new setup.
Throwing away tests that didn't win
Industry estimates put the share of A/B tests with no winning variant at 70–90%. Treat each loss as wasted effort and you start running fewer, safer tests, and safe tests rarely move revenue.
A losing test still answers a question. If a 40% discount badge didn't lift annual plan share, your users probably aren't price-sensitive at that step, so test value messaging or trial length next. If a higher price held conversion flat, you have room to go higher.
The bigger waste is repeating what the market already knows. Hundreds of apps have tested trial length, weekly versus annual defaults, and price anchoring. Starting from zero costs you six weeks of traffic to relearn it.
ℹ️What to do: Keep a test log with the hypothesis, primary metric, sample, result, and your read on why. Before each new test, write down what you'll try next if it loses.
Before you spend six weeks of traffic on a test, find out what happened when other apps ran it. Ask the assistant whether a 3-day trial beats a 7-day trial in your category, or what a weekly price increase does to ARPU, and get answers backed by results from real paywall experiments. You start from a benchmark instead of a blank slate. Test here 🧪.
Run fewer bad tests
The bottom line: Score pricing tests on ARPU, give them time to reach renewals, and keep the setup frozen until you hit your sample.
That alone filters out most false wins.
The other half is speed. Each test that waits on an app release or a developer sprint costs you a week you could have spent on the next hypothesis.
With Adapty's Flow & Paywall Builder, you build paywalls and full onboarding flows without code, A/B test them without an app update, and see per-screen results.
When you're deciding what to test next, ask the A/B test assistant what has worked for apps in your category and start with real numbers.
Book a demo to see how teams run paywall and flow experiments in Adapty.
