Paywall A/B testing for optimizing in-app revenue

The main goal of this series is to introduce you to the paywall A/B testing, give you some ideas that you could start with, and, hopefully, inspire you to achieve great results. We learned that running paywall experiments can be beneficial and profitable, but you need to know how to actually set them up. Let’s talk about the general aspects of launching an A/B test and the tools that will help you do this.

This is the third part of our paywall A/B testing guide, so if you haven’t, make sure to check out the previous articles to get some insights from the marketing approach to experimenting:

1. Paywall A/B testing guide, part 1: how to approach split-testing and avoid mistakes
2. Paywall A/B testing guide, part 2: what to test on the paywall
3. Paywall A/B testing guide, part 3: how to run A/B tests on paywalls and why it can be difficult
4. Paywall A/B testing guide, part 4: how to run experiments in Adapty

Choosing the A/B testing framework

To run A/B tests, you have to use a special framework that would be integrated in your app and have a console that would gather analytical data and “communicate” with your app via remote config (we’ll talk about it a bit further). There are two ways to implement this step: develop your own framework or integrate an already existing one.

Creating your own testing framework is a major undertaking that would require development in the back end, coming up with something resembling an admin console, setting up databases, implementing logic, and much more. Plus, it would require time to test the framework itself, fix all the bugs, and finally come up with a functioning prototype. Such a development would require several months and a couple of developers working non-stop. It’s challenging but may be rewarding in the long run. However, it may be more rewarding to integrate an already well-functioning side framework, which would take a couple of days, and start testing your ideas.

There are frameworks that allow developers to run A/B tests in their apps, providing certain analytics and even offering integrations with 3rd-party services. Some frameworks can be integrated and used for free (e.g. Firebase) and some have a free trial but offer pro plans for regular use (e.g. Adapty), but they usually offer more useful features and deeper analytics. It’s up to you which framework to choose, for starters, Firebase may be enough. But no matter which one you choose, the main aspects of their functionality remain the same.

Main aspects of an A/B testing framework

Let’s have a look at the general steps of how A/B testing frameworks function. First, you need to integrate the frameworks’ SDK, so that you’d be able to view all the analytical data from your app through the control panel. In that very panel, you’ll be able to set up and follow experiments.

The framework enables you to configure paywall elements with the help of  Remote Config – a piece of code that allows you to substitute one variable with another, almost in real-time. But it’s important to understand that your app must be ready for all the changes that you intend to make with the paywall. You must first include in the app’s code the “placeholders” for all the things you’d want to test: the number of products, possible pricing options, and any visual or textual element. Only then can they be controlled and changed from the console. Apart from the remote config code, you’d also have to insert certain fetch, activate, and get methods.

After the app is ready, you can proceed to create the experiment in the framework’s console. Usually, the steps are as follows:

• Name the experiment
• Set up the targeting: select your app, set the percentage of the target audience that will see the testing version of the paywall (50/50, 75/25, etc.), add some additional events to narrow down the audience
• Set the goal of the experiment. It could be the purchase income or any other metrics and conversion rates
• Create the A and B testing versions of the paywall. Choose the parameter that you’re going to test, which should be of a different version for each paywall. 
• Set up additional targeting methods: activation events, app version, localization, country, language, or any other user property
• Launch the test

A/B tests are usually based on testing one parameter, but the frameworks usually allow running tests with more than one. It’s not practical to run tests with 2 or more parameters, as in the end, you wouldn’t have a clear picture of what exactly affected the result of the experiment: was it one of the elements or all of them together? But this ability is provided because there may be design cases with 2 (or more) co-dependent elements, and changing one element requires changing the other one. For example, if you change the background color to a brighter tone, you may want to change the text properties as well, so that it looks darker and more readable. 

In some frameworks, the Remote Config feature is presented in the form of JSON, a simple format for data interchanging. Basically, you can decide in advance which elements of the paywall you’d want to test in the future and sign them with corresponding nicknames that will be used in the JSON file. We recommend creating a spreadsheet with all the variables and values they may possess (e.g ca_button – red, blue, yellow, for the call-to-action button and the colors it may have), and then turning this in the JSON format, on your own, or with the help of the developer. It will be easier to find and edit these variables in the Remote Config field afterward without asking for someone’s help.

Even though Firebase may be good for testing simple UI changes, or adding/deleting features, its analytical power may not be enough to properly assess much more complex experiments with a delayed economic effect. Moreover, Firebase allows you to track a lot of events, but the events’ parameters are limited only to 5, which is frustrating and inconvenient. It means that you can get high-level data, but won’t be able to go deeper into details, where usually the most insightful information lies. 

In this case, you’d either have to use BigQuery, which requires SQL knowledge, or any other 3rd-party analytics systems, which are mostly paid. 

There’s also a difference in the way Firebase and other paid frameworks work with events. Firebase processes all the attribution events on the client’s side only, as it doesn’t communicate with other servers. In more advanced frameworks, only the primary attribution event is processed on the client (e.g. initial purchase), and all the further events (renewals, cancellations, billing issues, etc) take place on the server (Apple or Google). This helps gather more relevant and true-to-life analytical data.

If you don’t want to bother with complexities, it would be a good choice to try some paid service that would have all the data in one place and in a convenient format. Apart from A/B testing, such services also provide additional analytical features that can be very helpful in the process of growing your app: cohort analysis, user profiles, segmentation by different parameters, native integrations with 3rd-party services, etc. If you are interested in trying Adapty as a framework for your test, in the next article we’ll explain how to integrate Adapty SDK and start running paywall tests in your app.