A/B Testing #

Experiments are explored in the Experimenting with Audiences guide, however this guide shows how to use Experiments and Metrics specifically to run A/B tests in your game and analyze the results.

To achieve this you will do the following:

• Create an event
• Create a metric
• Create an experiment
• Define the variants
• Define the phase
• Have the client participate in the experiment
• Analyze the results

This example demonstrates the classic “purchase button color” A/B test where users are presented with a button that is either green or blue, and metrics are then used to analyze which color button resulted in more clicks.

Create an event #

Create an event that will be triggered by the client whenever the purchase button is clicked.

1. Navigate to Settings -> Events.
2. Click Create New Event.
3. For Event Name, type purchaseButtonClicked.
4. Leave the Type as string.

Create a metric #

Create a metric that will be used to count the number of time the purchaseButtonClicked event occurs.

1. Navigate to Metrics.
2. Click New Metric.
3. For Name, type purchaseButtonClicked and give it an appropriate Description.

4. For Type, choose Count.
5. For Order leave it as High.

Create an experiment #

Create an experiment that will be used for A/B testing the different button colors.

1. Navigate to the Experiments screen.
2. Click Create New Experiment.
3. For Name, type PurchaseButtonColorExperiment and provide a meaningful Description.
4. For Audience, select ALL.
5. For Goal Metric, select purchaseButtonClicked.

Defining the variants #

Create two new variants: green and blue.

1. Click on the Variants tab.
2. Click Create New Variant.
3. For Name enter blue and for Value enter blue.
4. Click Create.
5. Repeat steps 2-4 to create a second variant with the name green and value green.

Defining the phase #

Create a new phase that will last for 1 week and have a 50/50 split between blue and green values for participants.

1. Navigate to the Phases tab.
2. Click Create New Phase.
3. For Phase Name enter PhaseOne and provide a meaningful Description.
4. Under Variant, enter a Split % value of 50 for both the blue and green variants.
5. For Start Date enter the date and time you want the phase to start.
6. For End Date enter the date and time you want the phase to end.

Client participation #

For the experiment to be valuable, players must take part in it by first receiving the value, updating the game UI accordingly and then triggering the appropriate event when the player clicks the purchase button.

Getting the experiment value and updating the UI

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

var result = await client.GetExperimentsAsync(session, new[] {"PurchaseButtonColorExperiment"});
var experiment = result.Experiments.FirstOrDefault();

if (experiment != null)
{
    var buttonColor = experiment.Value switch
    {
        "blue" => Color.blue,
        "green" => Color.green,
        _ => Color.gray
    };

    purchaseButton.GetComponent<Image>().color = buttonColor;
}

Triggering the event when button is clicked

1

await client.EventAsync(session, new Satori.Event("purchaseButtonClicked", DateTime.Now, experiment.Value));

Analyzing the results #

When the experiment is over you can view the resulting metrics by expanding the Phase you wish to view. This will provide all the Phase details including the start/end date and the configured variant split, metrics results and participants list.

The Goal Metrics section will display the counts of how many users received each variant specifically, as well as (for Count metrics) a breakdown of how many users sent an event after receiving the result.

You can see in the screenshot below that half the users received the control variant and half the users received the expensive variant (which matches the 50/50 split that was defined). You can see in the Goal Metrics section, which variant group impacted our goal metric of increased revenue more positively.

There are many different kinds of metrics you can experiment with. For a more detailed breakdown of these please see the Metrics page.