A/B Testing #

Experiments are explored in the Experimenting with Audiences guide, however this guide shows how to use Experiments and Metrics specifically to run A/B tests in your game and analyze the results.

To achieve this you will do the following:

This example demonstrates the classic “purchase button color” A/B test where users are presented with a button that is either green or blue, and metrics are then used to analyze which color button resulted in more clicks.

Create an event #

Create an event that will be triggered by the client whenever the purchase button is clicked.

  1. Navigate to Settings -> Events.
  2. Click Create New Event.
  3. For Event Name, type purchaseButtonClicked.
  4. Leave the Type as string.

Create a metric #

Create a metric that will be used to count the number of time the purchaseButtonClicked event occurs.

  1. Navigate to Metrics.
  2. Click New Metric.
  3. For Name, type purchaseButtonClicked and give it an appropriate Description.
Metric name
The name of the metric must exactly match the name of the event.
  1. For Type, choose Count.
  2. For Order leave it as High.

Create an experiment #

Create an experiment that will be used for A/B testing the different button colors.

  1. Navigate to the Experiments screen.
  2. Click Create New Experiment.
  3. For Name, type PurchaseButtonColorExperiment and provide a meaningful Description.
  4. For Audience, select ALL.
  5. For Goal Metric, select purchaseButtonClicked.

Defining the variants #

Create two new variants: green and blue.

  1. Click on the Variants tab.
  2. Click Create New Variant.
  3. For Name enter blue and for Value enter blue.
  4. Click Create.
  5. Repeat steps 2-4 to create a second variant with the name green and value green.

Defining the phase #

Create a new phase that will last for 1 week and have a 50/50 split between blue and green values for participants.

  1. Navigate to the Phases tab.
  2. Click Create New Phase.
  3. For Phase Name enter PhaseOne and provide a meaningful Description.
  4. Under Variant, enter a Split % value of 50 for both the blue and green variants.
  5. For Start Date enter the date and time you want the phase to start.
  6. For End Date enter the date and time you want the phase to end.

Client participation #

For the experiment to be valuable, players must take part in it by first receiving the value, updating the game UI accordingly and then triggering the appropriate event when the player clicks the purchase button.

Getting the experiment value and updating the UI

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
var result = await client.GetExperimentsAsync(session, new[] {"PurchaseButtonColorExperiment"});
var experiment = result.Experiments.FirstOrDefault();

if (experiment != null)
{
    var buttonColor = experiment.Value switch
    {
        "blue" => Color.blue,
        "green" => Color.green,
        _ => Color.gray
    };

    purchaseButton.GetComponent<Image>().color = buttonColor;
}

Triggering the event when button is clicked

1
await client.EventAsync(session, new Satori.Event("purchaseButtonClicked", DateTime.Now, experiment.Value));

Analyzing the results #

When the experiment is over you can view the resulting metrics by navigating to the Phases tab and clicking the name of the phase. This will take you to the Phase Details screen where you can see the start/end date and the configured variant split.

The Goal Metrics section will display the counts of how many users received each variant specifically, as well as (for Count metrics) a breakdown of how many users sent an event after receiving the result.

You can see in the screenshot below that 50 users received the blue variant and 50 users received the green variant (which matches the 50/50 split that was defined). Of those 50 users who received blue, 41 clicked the button. Of the 50 users who received green, 46 clicked the button. Since this is the higher result and our metric was configured with the High Order value, the 46 / 50 result is highlighted in green as the winning result.

Metric results
Metric results

There are many different kinds of metrics you can experiment with. For a more detailed breakdown of these please see the Metrics page.