Experiments #

Experiments enable you to deliver feature configurations to audiences, allowing you to test the effectiveness and success of a particular new feature, design, configuration, etc. on a given audience.

These differ from feature flags in that experiments are only for a fixed duration, are scoped to a particular audience, and are measurement-oriented - testing your assumptions about what change a new feature will drive in a given metric.

Use experiments in designing your game to work iteratively and make data-driven decisions. Successful experiments can then be converted to permanent feature flags for your players.

Audience assignment #

Identities are assigned to experiments based on their audience membership. Within any experiment, you can specify the audiences that will participate in the experiment, and how the identities in those audiences will be split across the experiment variants.

For example, you can create an experiment to test a new game mode on your non-spender audience. You can then specify a 50/50 split of the identities in non-spender audience to be assigned to the control variant or new-game-mode variant. This does not mean that the audience will be equally split between the two variants, rather for each individually identity in the audience there is a 50% chance that they will be assigned to the control variant and a 50% chance that they will be assigned to the new-game-mode variant.

Changed Audience Membership
Once an identity is assigned to a specific experiment variant, it will remain in that variant for the duration of that experiment phase regardless if the audience to which the identity belongs changed in the interim (e.g. from non-spender to spender).

Creating experiments #

Click the + New Experiment button to open the Create Experiment modal, then enter the following details:

  • Name: Enter a unique name for this experiment
  • Description: Provide a description of the experiment
  • Audiences: The audiences to participate in the experiment
  • Hold Outs: The audiences, if any, to exclude from the experiment
  • Goal Metrics: The desired metric to measure based on the experiment
  • Monitor Metrics: Additional metrics to monitor during this experiment

Create a new Experiment
Create a new Experiment

Experiment details #

Select an experiment from the list to view its details page. This page contains three tabs: Details, Variants, and Phases.

Details #

View and edit the experiment’s details, including its name, description, audiences, and metrics.

Editing Experiment details
Editing Experiment details

Variants #

View, edit, and create new variants for this experiment.

View Experiment variants
View Experiment variants

Phases #

Create new experiment phases and view the experiment’s current and/or previous phases.

Phases are used to control the variants that are active during a given time period for a given audience split. At the conclusion of the phase, you can view the changes in the goal and monitor metrics for the experiment.

View Experiment phases
View Experiment phases