Subscriptions 101: learn how to get +23% app revenue Read more

Paywall A/B testing guide, part 3: how to run A/B tests on paywalls and why it can be difficult

Sergey Zubkov

July 6, 2022

14 min read

Table of Contents

62d51d1f43bd691be3c8dc1e paywall ab testing guide part4

The main goal of this series is to introduce you to the paywall A/B testing, give you some ideas that you could start with, and, hopefully, inspire you to achieve great results. We learnt that running paywall experiments can be beneficial and profitable, but you need to know how to actually set them up. Let’s talk about general aspects of launching an A/B test and tools that will help you do this.

This is the third part of our paywall A/B testing guide, so if you haven’t, make sure to check out the previous articles to get some insights from the marketing approach to experimenting:

  1. Paywall A/B testing guide, part 1: how to approach split-testing and avoid mistakes
  2. Paywall A/B testing guide, part 2: what to test on the paywall
  3. Paywall A/B testing guide, part 3: how to run A/B tests on paywalls and why it can be difficult
  4. Paywall A/B testing guide, part 4: how to run experiments in Adapty

Choosing the framework

To run A/B tests, you have to use a special framework that would be integrated in your app and have a console that would gather analytical data and “communicate” with your app via remote config (we’ll talk about it a bit further). There are two ways to implement this step: develop your own framework or integrate an already existing one.

62c58ae90d72c3836f604614 kccnyi072twwcuhl8gxg9puuyo12rdbs1g1jhyrwwxv gqyi5cp3i3mg6vhsp9 r6uklkjjbu81zrzr216mypvtz6msif j3mwrc3pb sfem2wr9g5g5aptjhsrtwkmudyjghiiaho6ogvmoca
Basic analytics in the Google Firebase console

Creating your own testing framework is a major undertaking that would require development in the back end, coming up with something resembling an admin console, setting up databases, implementing logic, and much more. Plus, it would require time to test the framework itself, fix all the bugs, and finally come up with a functioning prototype. Such a development would require several months and a couple of developers working non-stop. It’s challenging, but may be rewarding in the long run. However, it may be more rewarding to integrate an already well-functioning side framework, that would take a couple of days, and start testing your ideas.

There are frameworks that allow developers to run A/B tests in their apps, providing certain analytics and even offering integrations with 3rd-party services. Some frameworks can be integrated and used for free (e.g. Firebase) and some have a free trial but offer pro plans for regular use (e.g. Adapty), but they usually offer more useful features and deeper analytics. It’s up to you which framework to choose, for starters Firebase may be enough. But no matter which one you choose, the main aspects of their functionality remain the same.

Main aspects of an A/B testing framework

Let’s have a look at the general steps of how A/B testing frameworks function. First, you need to integrate the frameworks’ SDK, so that you’d be able to view all the analytical data from your app through the control panel. In that very panel you’ll be able to set up and follow experiments.

The framework enables you to configure paywall elements with the help of  Remote Config – a piece of code that allows you to substitute one variable with another, almost in real time. But it’s important to understand that your app must be ready for all the changes that you intend to make with the paywall. You must first include in the app’s code the “placeholders” for all the things you’d want to test: number of products, possible pricing options, and any visual or textual element. Only then can they be controlled and changed from the console. Apart from the remote config code, you’d also have to insert certain fetch, activate, and get methods.

Example of the Remote Config code for the paywall

After the app is ready, you can proceed to create the experiment in the framework’s console. Usually the steps are as follows:

  • Name the experiment
  • Set up the targeting: select your app, set the percentage of the target audience that will see the testing version of the paywall (50/50, 75/25, etc.), add some additional events to narrow down the audience
  • Set the goal of the experiment. It could be the purchase income or any other metrics and conversion rates
  • Create the A and B testing versions of the paywall. Choose the parameter that you’re going to test, which should be of a different version for each paywall. 
  • Set up additional targeting methods: activation events, app version, localization, country, language, or any other user property
  • Launch the test
62c58b332cb363c84727d904 333
Creating a Remote Config experiment in Firebase

A/B tests are usually based on testing one parameter, but the frameworks usually allow running tests with more than one. It’s not practical to run tests with 2 or more parameters, as in the end, you wouldn’t have a clear picture of what exactly affected the result of the experiment: was it one of the elements or all of them together. But this ability is provided because there may be design cases with 2 (or more) co-dependent elements, and changing one element requires changing the other one. For example, if you change the background color to some brighter tone, you may want to change the text properties as well, so that it looks darker and more readable. 

In some frameworks the Remote Config feature is presented in the form of JSON, a simple format for data interchanging. Basically, you can decide in advance which elements of the paywall you’d want to test in the future and sign them with corresponding nicknames that will be used in the JSON file. We recommend creating a spreadsheet with all the variables and values they may possess (e.g ca_button – red, blue, yellow, for the call-to-action button and the colors it may have), and then turning this in the JSON format, on your own, or with the help of the developer. It will be easier to find and edit these variables in the Remote Config field afterwards without asking for someone’s help.

62c58ae97dc7651222463543 el98vfahrbfwqhvmwgeibcssamzocvsqwfil bjcwtbxqefednikn6whvlwxnexqfxgm2hn3epxtme1wmctlo8xgwij
Prepare a spreadsheet with all the variables that can be tested in Remote Config as a JSON

Even though Firebase may be good for testing simple UI changes, or adding/deleting features, it’s analytical power may not be enough to properly assess much more complex experiments with a delayed economical effect. Moreover, Firebase allows you to track a lot of events, but the events’ parameters are limited only to 5, which is frustrating and inconvenient. It means that you can get high-level data, but won’t be able to go deeper into details, where usually the most insightful information lies. 

In this case, you’d either have to use BigQuery, which requires SQL knowledge, or any other 3rd-party analytics system, which are mostly paid. 

There’s also a difference in the way Firebase and other paid frameworks work with events. Firebase processes all the attribution events on the client’s side only, as it doesn’t communicate with other servers. In more advanced frameworks, only the primary attribution event is processed on the client (e.g. initial purchase), and all the further events (renewals, cancellations, billing issues, etc) take place on the server (Apple or Google). This helps gather more relevant and true-to-life analytical data.

If you don’t want to bother with complexities, it would be a good choice to try some paid service that would have all the data in one place and in the convenient format. Apart from A/B testing, such services also provide additional analytical features that can be very helpful in the process of growing your app: cohort analysis, user profiles, segmentation by different parameters, native integrations with 3rd-party services, etc. If you are interested in trying Adapty as a framework for your test, in the next article we’ll explain how to integrate Adapty SDK and start running paywall tests in your app. 

Further reading