Why Firebase doesn’t work for mobile paywall A/B testing (+ a better alternative)

Backed by Google, Firebase is an app development platform that offers several tools to help creators build Android, iOS, and web apps. Because Firebase also features an A/B testing functionality, app makers can use it to test different app experiences and find the most optimal ones.

While Firebase’s A/B testing functionality – which is still in Beta, by the way –  can be used for A/B testing mobile app paywalls too, its many limitations become obvious once you try doing the same. Why? Because paywall experiments are a specialized kind of testing, as we’ll see in this post.

In today’s article, we’re going to look at:

• How Firebase A/B testing works for general mobile app experience testing vs. paywall testing (and why it’s not designed for paywall testing from the ground up).
• What Firebase’s A/B testing functionality’s fundamental limitations are when testing paywalls (analytics!).
• How Firebase’s in-built segmentation capabilities fall short for running targeted paywall experiments.
• And why you’d want to use a dedicated mobile app paywall A/B testing solution like Adapty over Firebase to power your paywall experimentation program.

Let’s get started.

1. Firebase wasn’t designed for mobile app paywall A/B testing

In its documentation, Firebase explains how its A/B testing “gives you the power to test changes to your app’s UI, features, or engagement campaigns to see if they actually move the needle on your key metrics (like revenue and retention) before you roll them out widely.“

As its documentation highlights, Firebase A/B testing can help you run experiments inside your app and comes in handy for many use cases. You can use it to: 

• Test and improve your product experience 
• Find ways to re-engage your users by using the Notifications composer 
• Safely roll out new features 

As you can see from how they put it, Firebase’s experimentation capabilities are geared more towards helping you 1) build a better product (app!) and 2) improve your messaging.

Firebase’s A/B testing functionality’s use cases only become clearer when you see its case studies. Firebase’s A/B testing success stories mostly only demonstrate the solution’s focus on helping ship better apps rather than enabling seamless paywall experimentation programs (which are an essential for app businesses).

While, of course, you can test paywalls with Firebase, neither Firebase’s A/B testing landing page nor its documentation mention paywall testing even once. All of this goes on to show how, by design, paywall testing hasn’t been a focus area for Firebase’s A/B testing solution. With Firebase A/B testing, paywall testing is “just another use case.”

Why does this limitation matter?

When you use Firebase to run a paywall experiment, it reflects almost everywhere that it’s a general-purpose mobile app A/B testing tool. Setting up a paywall test, too, takes additional steps with Firebase. For example, to set up a paywall A/B test with Firebase, you need to:

• Sign up for Firebase.
• Integrate the Firebase SDK inside your app.
• Integrate the Google Analytics for Firebase SDK inside your app.
• Set up Analytics.
• Codify your paywall (or paywalls) inside Firebase.
• Find out how you can codify the different revenue data points (via custom events in Google Analytics) that you’d like to collect. (This step needs you to set up dozens of events for capturing your desired data with custom coding! We’ll zoom in on this in just a bit.)‍
• Launch your experiments.
• Dig into Analytics and see what you managed to capture. (You should understand how custom reporting works for this.)

Now compare this setup with how it works in a proper paywall experimentation solution like Adapty. For instance, with Adapty you:

• Sign up for Adapty. 
• Integrate Adapty’s SDK inside your app. 
• Codify your paywall (or paywalls) inside Adapty. 
• Launch your experiments.
• Access Adapty’s accurate data reports full of actionable insights – capturing everything you need to power your app’s financial decisions right out of the box.

A specialized paywall A/B testing, experimentation, and personalization solution like Adapty fastracks your paywall experimentation program. Testing new pricing plans, subscription lengths, and promos – and pretty much everything that lives on your paywall – is as easy as one-two-three. 

Note: This isn’t a limitation of Firebase… it’s just that it’s not a paywall testing or experimentation solution. 

While we’re at it:

A lot of times, when app makers use Firebase to build their app’s backend infrastructure, they add an SDK like Adapty to it. Why? Because solutions like Adapty offer an almost plug-and-play in-app purchase infrastructure to help developers sell in-app purchases like subscriptions. Even with Adapty’s free plan, you can launch in-app purchases and monetize your app within a day.

2. Insufficient data reporting for paywall testing

Firebase’s additional “Analytics” step in paywall testing

We touched upon this earlier. But let’s look at it in detail now.

Because Firebase can’t answer all your money questions by default, running a Firebase paywall experiment needs you to codify not just your paywall inside Firebase but also set up a series of analytics events to capture all the data you want your paywall experiment to capture. 

Said another way: You need to dedicate a host of resources for coding your Google Analytics events to gather the necessary data.

Here’s how it works.

First up, Firebase makes it clear that you need a bunch of 1) “recommended events” and 2) “custom events” to generate meaningful app data. All these, as Firebase notes, “require some implementation in order to see them in Analytics.” Read: Custom coding.

Under recommended events, you have events like refunds, earning or spending virtual currency, and selecting a promotion. Firebase recommends that you set these events up in your account but doesn’t enable tracking them by default as they “require additional context to be meaningful.“

And custom events can take pretty much any form. This is any data that you’d like to collect. If you can code it as an event, you can collect it. 

With Google Analytics for Firebase, Firebase lets you set up up to 500 events for custom reporting. However, as you can now understand, to use Firebase to run paywall experiments, you’re looking at a LOT of custom coding for setting up these events.

With solutions like Adapty, on the other hand, all you need to do is codify your paywall (so you can test different elements via RemoteConfig), and you’re done. You don’t have to write even a line of code to get any data. All the data that you need is captured automatically for you. 

Once you launch an experiment with Adapty, Adapty starts collecting dozens of metrics on your experiment without needing you to do any coding whatsoever. Check out the following Adapty report for a paywall A/B test to get an idea. This experiment, as you can see, is testing two versions of a paywall, each of which offers different subscription plans. Version A is offering 3 subscription plans, while version B has 2 plans up on offer. Adapty isn’t just telling you which version is winning but is also giving you granular data on every single subscription plan on offer in both versions. 

For each product (subscription plan), you’re getting rich data on its:

• Revenue 
• Proceeds
• ARPPU
• The purchase conversion rate
• The trial conversion rate
• The actual number of purchases
• The actual number of trials
• Purchases
• Trials (Canceled) 
• Refunds 
• Views

When it comes to making data-backed financial decisions like pricing your app, the more you know, the better. With Firebase, you don’t get access to all these metrics, definitely not without adding a lot of events to your reporting.

Adapty, in contrast, automagically logs multiple events inside your app to collect all this data. Check out Adapty’s events for more:

That’s not all.

Did you know that with Adapty, you could also pass on such event data to a push notification solution (like OneSignal) and send personalized notifications? Such notifications can both increase subscribers and reduce churn. An example of this would be dropping a notification just as a user’s trial period begins. Check out our integrations on how you can use such event data with the rest of your app tech stack.

Having a “third-party” analytics tool is a must

Let’s say you’re able to work with your analytics team and add a host of events to Analytics to gather data. Sadly it might still not cover all your needs. Even if you create custom events to log your desired revenue metrics, you’ll still be boxed in the platform’s limitations. You’d eventually end up needing a third-party paywall SDK or analytics solution to collect all the missing revenue data for you.

How missing data leads to lost conversion opportunities

When you use a paywall A/B testing solution like Adapty, you have access to some very high-quality first-party data on your users.

For example, if Adapty shows a lot of refunds – Adapty automatically collects refund data – you might want to see if there’s a mismatch between what you’re promising on your app paywall and what you’re shipping.

Likewise, if Adapty reports multiple billing issues – again, this is data Adapty captures by default – you may want to review your billing ops. 

If a sizable audience seems to be in the grace period, a win-back experiment might make a difference! 

You get the drift.

3. Firebase’s limited segmenting for paywall testing

By default, Firebase comes with two predefined audience segments: 

• All users (These are “app users who have ever launched your app.”)
• Purchasers (These are users who have made an in-app purchase.)

But with its support for custom audiences. Firebase lets you filter out users based on age, gender, language, location, and device, among other data points.

You can also use events to filter segments.

You can add up to 100 audiences.

But while you get support for building custom audiences, many user segments you’d want to target with your paywall experiments aren’t available directly in Firebase.

For example:

Say you wanted to run a paywall experiment and target users who canceled their subscription during the trial period. However are you going to get this segment in Firebase?

The solution lies in custom coding. 

But as we just saw, that’s also the problem.

Likewise, if you wanted to offer a personalized paywall to users with the highest lifetime value… How would you filter out such people?

Again, you’d need custom coding.

And let’s say you wanted to run an experiment and only target users whose payments failed… How would you reach such people?

With Firebase, you’d need a good deal of custom coding to create such custom segments. 

Here, too, you may need a third-party solution to create custom audience segments for paywall testing for importing to your Firebase console for testing.

In contrast, in a paywall experimentation solution like Adapty, you can create such segments on the fly. Adapty lets you define segments using a host of metrics like: 

• Available attributes
• Age
• Calculated subscription state
• Calculated total revenue USD
• Country (from IP)
• Country from store account
• Creation date
• Device
• Gender
• OS
• Platform
• Subscription expiration date
• Subscription product
• Attribution Source: Organic, Non-Organic, Unknown
• Attribution Channel
• Attribution Campaign
• Attribution Ad Group
• Attribution Ad Set
• Attribution Creative
• Custom Attributes

Read more on Adapty’s segments here.

There’s no differentiation between paywall tests and other experiments

Paywalls sit at the heart of an app’s monetization mix.

Experiments that run on the paywall directly impact an app’s revenue potential.

A good paywall experiment should go like “Hypothesize – Build – Test!” and shouldn’t need your team to: 

• Write (and maintain) complex coding to capture critical revenue data that answers your real questions
• Endlessly figure out how to work out experiment logistics like segmenting and targeting
• And spend hours figuring out how to create custom reporting from all the custom events

By keeping the process of 1) planning, 2) implementing, and 3) analyzing tests effortless, paywall A/B testing and experimentation solutions like Adapty decouple paywall ops from the rest of the app, so you can constantly innovate on the marketing side. The results? You get to run a high-quality testing program that moves fast.

Install Adapty’s (free) SDK today and monetize your app with in-app purchases. You can learn a great deal about your audiences with Adapty, about what brings them to your app and also what converts them. And when you’re ready to run A/B tests, upgrade to Adapty’s pro plan: it unlocks A/B tests, personalizations, and even more detailed analytics.

Wrapping it up…

As you just saw, while Firebase’s testing functionalities can help you offer improved app experiences – which can translate to converting and retaining more users – they aren’t cut out for paywall testing, at least not out of the box.

Setting up its analytics, so it gives you actionable insights that improve your app’s financial health, is its single most challenging aspect. And assuming that you’re still somehow able to work it the way you want, you’ll still need to plug the “missing” pieces of insights you need to run your revenue ops by adding third-party analytics tools. You might even end up needing a solution like Adapty to power your paywall tests and personalizations.

If you’re specifically interested in A/B testing your mobile app’s paywalls, go with Adapty. It has everything you need to run winning mobile app paywall tests and personalizations. With easy integration, simple implementation, and some best-in-class reporting, Adapty makes running revenue experiments that help you meet your app’s short- and long-term growth goals as easy as one-two-three. And if you’re a Firebase user already, find out how you can use Adapty along with Firebase. If you have questions, reach out.