Experiments #

Experiments enable you to deliver feature configurations to audiences, allowing you to test the effectiveness and success of a particular new feature, design, configuration, etc. on a given audience.

These differ from feature flags in that experiments are only for a fixed duration, are scoped to a particular audience, and are measurement-oriented - testing your assumptions about what change a new feature will drive in a given metric.

Use experiments in designing your game to work iteratively and make data-driven decisions. Successful experiments can then be converted to permanent feature flags for your players.

Audience assignment #

Identities are assigned to experiments based on their audience membership. Within any experiment, you can specify the audiences that will participate in the experiment, and how the identities in those audiences will be split across the experiment variants.

For example, you can create an experiment to test a new game mode on your non-spender audience. You can then specify a 50/50 split of the identities in non-spender audience to be assigned to the control variant or new-game-mode variant. This does not mean that the audience will be equally split between the two variants, rather for each individual identity in the audience there is a 50% chance that they will be assigned to the control variant and a 50% chance that they will be assigned to the new-game-mode variant.

Creating experiments #

Click the + New Experiment button to open the Create Experiment modal, then enter the following details:

• Name: Enter a unique name for this experiment.
• Description: Provide a description of the experiment.
• Schema Validator: Select the validator to use for data from this experiment.
• Audiences: The audiences to participate in the experiment.
• Hold Outs: The audiences, if any, to exclude from the experiment.
• Goal Metrics: The desired metric to measure based on the experiment.
• Monitor Metrics: Additional metrics to monitor during this experiment.
• Split Key: Select how identities will be split across the experiment variants, either:
  • ID + Phase Start (dynamic): This is the default setting. Identities will be assigned to a variant based on their identity ID and the start time of the current phase. Variant assignments are stable across deletion and recreation of an identity.
  • ID + Phase Start + Create Time (random): Identities will be assigned to a variant based on their identity ID, the start time of the current phase, and the time the identity was created. Variant assignments are not stable across deletion and recreation of an identity.
  • ID (stable): Identities will be assigned to a variant based on their identity ID.

Experiment details #

Select an experiment from the list to view its details page. This page contains three tabs: Details, Variants, and Phases.

Details #

View and edit the experiment’s details, including its name, description, audiences, and metrics.

Variants #

View, edit, and create/delete variants for this experiment.

Phases #

Create new experiment phases and view the experiment’s current and/or previous phases.

Phases are used to control the variants that are active during a given time period for a given audience split.

You can click on any phase to view the changes - partial in the case of a still running phase - in the goal and monitor metrics for the experiment as well as view and edit the participants.

When selecting a phase to view its associated details there are two tabs: Stats and Participants:

Stats #

From this view, you can view phase details and the metrics results.

For still running phases, you have the ability to use the Halt Phase button to stop the current phase.

The Export Variants button can be used to automatically create audiences and/or flag variants based on this experiment phase.

Participants #

From this view, you can view the participants for this phase and their assigned variants, as well as edit the variant assignments or delete the identity from the experiment phase.