Agent skill for A/B tests

The ab-tests skill runs regular A/B tests of paywalls or flows from your AI coding tool. Your agent plans the test with you, makes the pieces that are missing, creates the test as a draft, and starts it, stops it with a winner, or replaces it with a changed one. It works through the adapty ab-tests CLI commands.

Note

You can’t undo starting, stopping, replacing, or archiving a test. Before each of these steps, the skill tells you in plain words who sees what from when, and it runs the command only after your yes.

What you can ask

Ask the way you’d brief a colleague. You don’t need to name the skill:

Run a 50/50 test of my Monthly and Annual paywalls for users in Germany.

Test my new onboarding flow against the current one.

Which A/B tests are running in my app?

Stop the price test. B wins.

Change the weights of my flow test to 70/30.

Archive the old tests on the settings placement.

ab-tests

Run a 50/50 test of my Monthly and Annual paywalls for users in Germany on the Main placement.

Ran 5 read-only commands and created a draft

adapty paywalls list
adapty placements list
adapty segments list
adapty ab-tests create --title "Monthly vs Annual" --paywalls '[…]'
adapty ab-tests get <test-id>
adapty placements get <placement-id>

I created the test “Monthly vs Annual” as a draft. No user sees it yet.

Start the A/B test

Start “Monthly vs Annual” on the Main placement for users in Germany.

Now: users in Germany see the Monthly paywall.
From now: they see the Monthly paywall (50%) or the Annual paywall (50%). A change for a segment can take up to 20 minutes to reach every user.
Stops: the Monthly paywall stops for users in Germany. No test runs there now, so none stops.
Can’t be undone. Stopping the test later doesn’t bring back what it replaced.

Start it?

Example conversation with a sample app. Your placements and paywalls will differ.

Before you start

An AI tool that can run terminal commands: For example, Claude Code, Codex, Copilot CLI, or Gemini CLI. Chat tools like claude.ai or ChatGPT can’t run the CLI, so they can’t reach your Adapty account.
The Adapty CLI, version 0.13.0 or later: The skill installs the CLI if it’s missing. When your CLI has no ab-tests commands, the skill says so and sends you to the dashboard.

Install the skill

The ab-tests skill is part of the adapty-skills plugin. To install the plugin, in your terminal, run:

npx adapty skills install

Restart your agent to load the skills. For other installation methods, see the plugin README.

How the skill runs a test

The skill agrees on the plan with you: a paywall test or a flow test, the paywalls or flows to compare and their weights, the placement, the audience, and the goal. It asks for all missing details in one message.
It verifies each piece and makes what it can. See What the test needs.
It creates the test as a draft. No user sees a draft, so this step needs no yes.
It reads the test and the placement, then shows you the effect of the start: what the audience sees now, what it sees from the start, which paywall, flow, or running test stops, and which users move into the test.
It waits for your yes. An earlier “go ahead” about the plan doesn’t count. If you say no, it sends nothing.
It reads the placement again right before the start. If the placement changed since you saw the effect, it shows you the new effect and asks again.
It starts the test and reads it back to confirm what runs now.

Stopping, replacing, and archiving a test follow steps 4–7, with that action in place of the start.

The skill doesn’t start a test again after it ran, because users who were in it would get their old arm again. It offers a new test with the same arms instead.

What the test needs

A test can only compare content that exists. The skill verifies each piece and makes the ones the CLI can make:

Piece	What the skill does
A flow	Verifies that the flow is published. A flow joins a test only once it is published. To build or publish one, the flow-generator skill takes over.
A paywall from products already in Adapty	Creates it, after your yes.
A paywall with a new design	Asks you to make it in the Paywall Builder, or in your app’s code for a paywall your app draws itself.
A new product	Asks you to create it in App Store Connect or Google Play Console and add it to Adapty. See Products.
A segment	Creates it through the audiences skill, after your yes.
A placement	Creates it after you approve its developer ID, which can’t be changed later. Your app shows nothing from a new placement until its code fetches that placement. A test on the placement reaches no users until you release an app version whose code fetches it.

Which audience the test runs on

A test runs on one audience of the placement:

When the placement already has an audience for your segment, or the test is for All users, the test takes that audience. A test running there now stops.
When the placement has no audience for your segment, the skill adds an audience for your segment with the highest priority on the placement. Users of the segment who currently match another audience move into the test, and the skill names those audiences before you say yes.

To give the test a lower priority than another audience, ask the skill. The skill first adds your segment as an audience with that priority and with the paywall or flow those users see now, so nothing changes for them. Then it starts the test on that audience.

Change, stop, or archive a test

Rename it or change its goal: The skill updates the test. Users keep their arms, and the test keeps its start time.
Change the arms or weights of a live test: The skill replaces the test. The live test stops and keeps its results, and a new test starts on the same audience. Adapty assigns every user of the audience to an arm again. In the dashboard, this is Modify.
Stop it: Tell the skill which paywall or flow the audience should see afterwards — the winner, other content, or nothing. To choose the winner, read the results in the dashboard: A/B test results and metrics.
Archive it: The skill archives any test that isn’t live. You can’t restore an archived test.

With other skills

Find what to test: The growth-advisor skill reads your numbers and suggests hypotheses. When you pick one, growth-advisor verifies that everything the test needs exists, then hands the test to the ab-tests skill.
Build the content: The flow-generator skill builds a copy of your flow with the change to test.
Show content without a test: To show different paywalls or flows to different users without a test, ask your agent. The audiences skill makes the change.

Limits

No results: The skill doesn’t read test results. Read them in the dashboard, then tell the skill which arm won.
One placement per test: To run a crossplacement test, use the dashboard.
Old onboarding builder: Manage onboarding tests made in the old onboarding builder in the dashboard.
No unattended starts: In a scheduled run or a script, with no one to answer, the skill only lists and reads tests, and creates and changes drafts. For a start, stop, replace, or archive, it reports the effect and the command for you to run.

To run the commands yourself, without the skill, see Run A/B tests from the CLI.