Myths and mistakes of paywall A/B testing

It’s not a secret that A/B testing is considered a powerful springboard for boosting revenue in mobile apps and the web. The statistics show that mobile app developers and publishers that run paywall experiments can make up to 100x more app revenue. Yet, this marketing tool has been lately covered with a plethora of myths concerning its effectiveness, safety issues, and being overrated in general. 

In this article, we’ll take the role of myth-busters and try to disprove some of the most popular myths circulating around paywall A/B testing. We’ll provide more context to the myths to show that they are mostly based on the lack of information or someone’s bad experience. In addition to that, we’d also cover several common mistakes that prevent people from utilizing A/B testing to 100%. Let’s start with the myths first.

Myths

I’ve met people from the mobile app industry who avoid paywall A/B testing like the plague, claiming that it’s unsafe, complicated, and takes more time than provides final value. In this part, we’ll take a look at a few common myths and misconceptions regarding paywall A/B testing and see that they are either not true or simply rudimentary prejudice. 

Myth #1: Using remote config for testing paywalls can get your app in trouble

This is one of the most popular and common myths circulating in the mobile app community. Everyone in the industry has heard stories of this or that app getting banned and deleted by the App Store for guidelines violations. Of course, Apple is rather strict when it comes to rules and regulations, but they’re also pretty consistent in this matter, meaning they actually go by the book and do not go beyond what’s included in their guidelines.

If you’re new to A/B testing: remote config is a feature that enables you to remotely change an interface element with the help of placeholder variables inserted in the code beforehand which can later be tweaked via JSON file. Such manipulations let you change elements in your app or software without having to release an update or a new version.

The requirement that is feared the most in such cases is 2.5.2 of the Software Requirements which states: “Apps should be self-contained in their bundles, and may not read or write data outside the designated container area, nor may they download, install, or execute code which introduces or changes features or functionality of the app, including other apps”. Well, it’s a pretty reasonable point, but it doesn’t relate to the use of 3rd party frameworks and remote config for paywalls. Here are a few points to bust this myth:

• The code is never changed. The working principle of remote config is the substitution of one piece of content with another (for example a background image). To do that, you insert placeholder variables in your app’s code in advance. Such a variable is attached to a certain content piece in the framework that you use. So the code is the same, but the chosen paywall element can now be changed multiple times.
• Even Firebase features remote config. Firebase, an app development platform powered by Google, also features an A/B testing feature based on remote config. The use of such a feature by the owners of the second-largest app store shows that it’s an absolutely safe-to-use instrument.
• 3rd parties’ software is not a threat. Many fear using the remote config of 3rd party frameworks because it may count as the “outside” functionality. But in reality, you can create the same in-house backend infrastructure and integrate it with your app. The main point is that it will be extremely expensive and hard to maintain, but still. 3rd party SDKs do not add any “outside” functionality, they work within the guidelines.
• Some of the most popular app publishers use it. If you think that most of the popular apps in your preferred app store made it to the top just because they provide valuable content, it’s far from reality. In order to grow in revenue and conversions, you need to constantly experiment and look for new ways to acquire a new audience. The only way to experiment with paywalls is through remote config (or paywall builder, but we’ll talk about it in a bit). You can get familiar with some of the remarkable experiments of our clients in our case studies.

To sum it up, the remote config or any other A/B testing platform that is based on it is safe to use in your app for testing or changing your paywall. Moreover, it may even be viewed as a test platform for providing the best user experience in your app, which is beneficial for the App Store itself.

Myth #2: Firebase is more than enough to test mobile paywalls

When you finally decide to try running A/B tests for your paywalls, you start searching for a tool that should be preferably free and well-known. The first framework that pops up is usually Firebase. And it’s pretty clear why: Knowing that it’s powered by Google and has tons of tutorials can make you think that it must be a pretty reliable framework, working with which is going to be a walk in the park. But it’s not that simple.

Even though Firebase enables app developers to run A/B tests in their apps, it was not designed to experiment with paywalls. We have already prolifically investigated this matter, but let’s go through the main takeaways once again:

• Not designed for paywall testing: The platform’s focus is on general mobile app experience testing and improving messaging rather than specialized paywall experimentation.
• Setup complexity: Setting up a paywall A/B test with Firebase involves several additional steps, such as integrating Firebase SDK and Google Analytics for Firebase SDK, codifying paywalls, and custom coding for capturing revenue data points. 
• Limited documentation: Firebase’s documentation and landing page for A/B testing do not explicitly mention paywall testing. This lack of emphasis indicates that paywall testing is considered “just another use case”.
• Insufficient data reporting: Firebase requires additional custom coding to set up recommended and custom events in Google Analytics to capture necessary data for paywall experiments. 
• Need for third-party analytics tools: The limitations in Firebase’s default analytics mean that users may need to integrate third-party analytics tools to collect missing revenue data.
• Limited segmenting for paywall testing: Firebase offers predefined audience segments and supports custom audiences based on certain data points. However, creating custom segments for specific paywall testing scenarios may require extensive custom coding.
• Lack of differentiation for paywall tests: Firebase does not differentiate between paywall tests and other experiments, potentially leading to challenges in planning, implementing, and analyzing paywall-specific tests. 

While Firebase may excel in general mobile app experience testing, it falls short in meeting the specific needs of paywall A/B testing, requiring additional steps, custom coding, and integration with third-party tools to achieve comprehensive and actionable results. Dedicated solutions like Adapty are more suitable options for streamlined and effective paywall experimentation.

Myth #3: A/B tests are difficult to run and complicated to understand

Another thing that scares people from even trying A/B testing is complexity. In big companies such experiments are run and analyzed by growth managers or analysts, which makes many app developers think that launching a test and “deciphering” the results is nothing else but secret lore. 

In reality, all you need is a basic understanding of mobile app metrics and a convenient platform. For example, launching a paywall A/B test in Adapty takes no more than 5 minutes:

• Create a paywall group with 2 paywalls, either with the help of remote config or Paywall Builder.
• Choose the placement you want these paywalls to be displayed in.
• Select the target audience and choose the traffic weight.
• Launch the test and wait for the results.

For the more statistically significant results, wait at least 2 weeks before stopping the test (if there’s no definite decline in core metrics, which you can check live in the analytics dashboard). When the experiment is over, the system will show you a detailed report on every paywall’s metrics, including Revenue, ARPPU, Purchases, Trials, Views, Refunds, and many more. But most importantly, you will clearly see which one has the higher P2BB percentage (Probability to Be the Best) along with a neat visualization of the experiment timeline.

Myth #4: It takes too much time, effort, and money to make a decent paywall

Following the default development scheme that’s been used for several years, we have the idea in our heads that every screen in your app must be created by the app designer, or at least made with their approval. Such an approach is important for the app’s consistency but it doesn’t mean that for every paywall A/B test you need to have a new paywall created by your team’s designer – not anymore.

With Adapty’s Paywall Builder, marketing or growth specialists, who usually run monetization experiments, can create beautiful paywalls with a native layout in a matter of minutes – with no developer involved. Paywall Builder is a visual no-code editor that enables you to design a paywall based on one of the numerous best-selling templates. Every element on the paywall can then be adjusted to your liking, including images, text font and size, button shape, color scheme, and more.

A high-quality paywall can actually be created in a few minutes and immediately launched in an A/B test. The same goes for recurring tests: if you’re not satisfied with the results of the previous test, you can apply new changes to the existing paywall pair and relaunch the test, which will also take no more than 5-10 minutes. The future of paywall testing is already here.

Mistakes

Now that we’re through with the popular myths, let’s talk about equally common mistakes people tend to make to this day. Along with it, you’ll find out how innovations in the sphere of in-app purchases and monetization can improve the process of paywall A/B testing and solve older problems in a more convenient way.

Mistake #1: Changing your paywall without running an A/B test

This may be one of the most common and painful mistakes an app developer or publisher can make. Broken updates are always a loss – both financial and reputational – that’s why we always test new builds before publishing them. For some reason, it’s not always the case with paywalls. A hypothesis may seem so promising that one may not want to waste their time on running a test for a couple of weeks and apply changes to the paywall right away instead.

It may work out, but very often it leads to churn, higher bounce rate, lower conversion rates, and, as a result, a decrease in revenue. Why? Because one could miscalculate things, didn’t pay enough attention to dependencies in the funnel, or still made a technical error that led to troubles with purchases or user flow. Such situations require a quick reaction and swift actions to be taken to restore the previous version of the paywall. But Murphy’s law is still in the game, so you may get into a situation where the follow-up update gets stuck at the store review stage, leaving you to sit and watch your money going down the drain.

Should we say this can never happen when you use Adapty? Primarily for two reasons:

1. We recommend running A/B tests for your new paywalls before making them live. Such an approach leaves you with the most profitable paywall at the moment and enables you to test if your new paywall actually works the way you expected it to.
2. Even if you decide to apply changes straight away to your live paywall, it’s always easy to go back to the previous variant, thanks to the remote config feature. Moreover, if you use the paywalls made with the Paywall Builder, you also reduce the risk of having a technical error to the minimum, since it handles all the coding for you.

Mistake #2: A/B testing paywalls with too many differences

On the one hand, it’s a basic mistake you learn about right before diving into the world of A/B testing, but on the other, it’s a mistake that even experienced growth managers tend to make from time to time.

Important note: It’s completely normal to test two drastically different paywalls when you intend to test a new paywall template or structure and your only goal is to see that based on your hypothesis the new variant performs at least as well as the older one. In our case, we’re talking about testing several elements within the same template/structure with the hypothesis of increasing this or that conversion rate or some other metric.

To see the most transparent results for any split experiment, you must have the minimum amount of changes in the testing pair. For example, testing only a background picture will give you a more or less accurate result to see which one is more appealing to the users. But testing a background image along with a different product layout and one more new product option will be a mess. 

I mean, you will get some kind of a result, maybe even a statistically significant one, and perhaps even the test version may win, but you’ll never know what exactly made the users turn to the paywall “B”. Was it a new product, or a new layout? Or maybe just a new picture was enough? It’s fine when your test paywall actually won, but usually, it’s the other way around, so in the end, you’re left with a failed test and don’t even know what happened. 

I mentioned earlier that even experienced managers tend to abuse this approach, as they feel they already know their product and their audience, but most importantly, they do so because, at some point, time becomes money. Product managers of apps with more than $500k MRR may not want to wait for ~2 weeks to test only one element. So sometimes it feels easier and cheaper for them to come up with a more complex hypothesis and test it right away. It may seem rather bold, but the road to success is fraught with risk.

Mistake #3: Running A/B tests for all users with no targeting

Once you’ve learned the beauty of paywall A/B testing, you may find yourself in a situation where you test one and the same paywall over and over for all the users who installed your app. On the one hand, it solves the problem of insufficient traffic, which we’ll talk about a bit later, but on the other hand, it may lead to you receiving less profit than you could from acquiring more high-quality subscribers. 

Targeting helps set up A/B tests for particular audiences based on a number of factors, like age, region, language, or other custom attributes. Having not only one general paywall but also a few paywalls targeted to certain audiences, may help you find the most profitable subscriber flow that you could run paywall A/B tests on and find the best product-price relationship. Sadly, not every platform that features paywall A/B testing allows for such targeting, but Adapty does. Adapty enables you to show different paywalls to different audiences, distinguished by one or several attributes, and as a result, run separate A/B tests.

Mistake #4: Not using localization and audience segmentation 

Continuing the idea of segmentation, it’s worth saying that paywall localization is another vital element that very often gets neglected. In most cases, when creating an app for the Western world, app developers simply leave the interface, along with the paywall, in English to make it more universal for users and easier for themselves. In other cases, they take 5-10 most popular languages and translate the app along with the paywall, either by Google translate or with the help of real translators. And in theory, it should be enough, but only in theory.

There may be a couple of problems here that could actually be depriving you of getting more revenue:

• The basic one – your locale translation either does not fit into the paywall interface or looks overloaded. It usually happens with German languages which have much longer words. This often gets neglected, as it’s quite difficult to properly check it in advance. So when a user sees such a careless paywall, they may get doubts about the overall quality of your app and may decide to look for something else.
• The paywall doesn’t correlate with the culture you’re aiming at. It’s usually relevant in the context of Western culture vs Eastern one. Sometimes it’s not enough to simply translate the words, you also need to adapt the paywall to the cultural characteristics of the audience. The least you can do is change the background image, choose a different product layout, and rewrite the benefits with regard to the culture. It doesn’t mean that the paywall with just translation won’t work, but the adapted one is sure to be more appealing and gain you much more.
• Your localized paywall is shown to the wrong audience. Sometimes you can’t know for sure if your localized paywall converts better or worse for a certain audience. For example, if a localized paywall is shown based on the location, let’s say Mexico, all the users from this location will be shown the Spanish version of the paywall when there’s a large number of native English speakers. Targeting is very important, but not every platform allows you to adjust it.

Why are we talking so much about the localization all of a sudden? Well, because all these problems can be easily solved by A/B testing, especially if you use Adapty. 

Translations for your paywall may be added with the help of the aforementioned remote config and adjusted on the fly, which reduces the risk of showing a visually broken paywall. Or if you decide to use the Paywall Builder to create a paywall, you can add new locales right in the editor. Every locale you add is broken down into strings of text for a corresponding paywall element, which makes the localization process much easier. Once the translations are added, you can preview the localized text on the mockup of your paywall before publishing it.

The same goes for the cultural aspect problem – there’s no faster way to find out which paywall variant works best for a certain culture than A/B testing. Launch a test with two different background images and see if it works. But make sure to aim at the right audience, which is also super convenient with Adapty’s audience targeting. Thanks to its paywall management system, you can set up precise targeting for A/B testing based on many criteria, including region, age, channel, language, etc. So you get both localization and targeted A/B testing in one place – problems solved.

Mistake #5: Running A/B tests with not enough user traffic 

Another childish mistake comes from actually getting into A/B testing with too much enthusiasm. It’s great to get into experimenting at the early stages of your app growth, but A/B testing of any kind requires as much traffic as possible, which can be a problem at that time.

To get statistically significant results, you need a significant amount of users – it’s that simple. If your user flow is less than 3000-5000 people a month, you risk getting unclear and unreliable results, which may lead you in the wrong direction. There are two ways out of this: EIther postpone A/B testing for a while and wait till your app gets bigger organically or purchase ad traffic. Getting paid ads is one of the most popular User Acquisition strategies, so if your funds allow it – go for it. 

To sum it up

Paywall A/B testing is a safe and efficient way to grow your mobile app, no matter what rumors you may hear. With current technologies, the process of split testing has become much more convenient, affordable, and easy to dive into. If you’ve been putting off tasting app monetization experiments, it’s the best time to start and Adapty is a great choice to use in this journey. Find out more about how it can help you boost your MRR.

Unlock Industry Insights

Gain the edge with key insights from our State of in-app subscriptions in the US 2023 report. Essential for app professionals!

Get your free report now