Heroic Labs Documentation

Experiments #

Experiments enable you to deliver feature configurations to audiences, allowing you to test the effectiveness and success of a particular new feature, design, configuration, etc. on a given audience.

These differ from feature flags in that experiments are only for a fixed duration, are scoped to a particular audience, and are measurement-oriented - testing your assumptions about what change a new feature will drive in a given metric.

Use experiments in designing your game to work iteratively and make data-driven decisions. Successful experiments can then be converted to permanent feature flags for your players.

Creating experiments #

Click the + New Experiment button to open the Create Experiment modal, then enter the following details:

  • Name: Enter a unique name for this experiment
  • Description: Provide a description of the experiment
  • Split Key:
  • Audiences: The audiences to participate in the experiment
  • Hold Outs: The audiences, if any, to exclude from the experiment
  • Goal Metrics: The desired metric to measure based on the experiment
  • Monitor Metrics: Additional metrics to monitor during this experiment

Create a new Experiment
Create a new Experiment

Experiment details #

Select an experiment from the list to view its details page. This page contains three tabs: Details, Variants, and Phases.

Details #

View and edit the experiment’s details, including its name, description, audiences, and metrics.

Editing Experiment details
Editing Experiment details

Variants #

View, edit, and create new variants for this experiment.

View Experiment variants
View Experiment variants

Phases #

Create new experiment phases and view the experiment’s current and/or previous phases.

Phases are used to control the variants that are active during a given time period for a given audience split. At the conclusion of the phase, you can view the changes in the goal and monitor metrics for the experiment.

View Experiment phases
View Experiment phases