Experiments

An experiment splits your customers into 2 to 5 variants and lets each variant receive different programs, rules or campaigns. Orishare then reports, per variant, how many customers came back and how many reached a goal you name, with the statistics needed to tell a real difference from chance.

Two shapes cover most questions:

• A holdout answers "is this program worth having?". A small share of customers (10% by default) goes without it; everyone else receives it.
• An A/B test answers "which version is better?". Each variant gets its own program, rule or campaign.

Reading needs the analytics:read scope and every write needs analytics:write. In the dashboard, experiments are under Experiments.

Variants and assignment

Each variant has a key, a name and a weight in basis points (weight_bps); the weights of an experiment add up to 10000. A customer's variant is decided by a hash of the experiment's private salt and the customer id, mapped onto the weights. That makes assignment:

• deterministic: the same customer always gets the same variant, on every server;
• stable: it does not change for the life of the experiment;
• free on the hot path: nothing is written when a customer is assigned.

The first variant is the control. Every other variant is compared against it.

Targets

A variant may list targets: rules, campaigns and programs, each named by type and key. While the experiment runs, a targeted resource is delivered only to customers whose variant lists it. Everything that is not a target of the experiment behaves as usual for everyone.

{
  "key": "streak-holdout",
  "name": "Daily streak holdout",
  "variants": [
    { "key": "holdout", "name": "Holdout", "weight_bps": 1000, "targets": [] },
    {
      "key": "treatment",
      "name": "Treatment",
      "weight_bps": 9000,
      "targets": [{ "type": "program", "key": "daily_streak" }]
    }
  ],
  "goal_event": "order.completed",
  "goal_value_property": "amount"
}

Here 10% of customers stop receiving the daily_streak program when the experiment starts, and 90% keep it. For an A/B test, give each variant its own target: [{ "type": "rule", "key": "double_points" }] for one and [{ "type": "rule", "key": "bonus_badge" }] for the other. Customers in the first variant then receive only double_points, customers in the second only bonus_badge.

Targets can also be friend quest and team quest definitions, by key: { "type": "friend_quest", "key": "weekly_duo" } and { "type": "team_quest", "key": "team_week" }. A withheld player cannot start that friend quest (403 forbidden, details.reason withheld_by_experiment), and a withheld team member is not paid when the team quest completes. See Friends and teams.

A resource can be the target of one running experiment at a time. Starting a second experiment on the same resource is refused.

Templates

For the common questions, start from a template instead of writing variants. Every template is a holdout: holdout (the control) goes without the measured targets and treatment receives them, 10% and 90% unless you set holdout_percent. GET /v1/experiment_templates lists them with what each measures and which targets to pick.

{
  "key": "streak-lift",
  "name": "Daily streak lift",
  "template": "streak_lift",
  "targets": [{ "type": "program", "key": "daily_streak" }]
}

A template only fills defaults: the experiment stores the resulting variants and goal, and goal_event or goal_value_property in the body override the template's. The same body works in the experiment.create change set operation.

Audience

An experiment may be limited to the members of one segment with "audience": { "segment": "<segment name>" } (on create, on a draft's update, and in the experiment.create change set operation). Only members take part:

• a member is assigned a variant, carries the exp: segment and has targets withheld as usual;
• everyone else gets no variant, no exp: segment and keeps receiving every target, as if the experiment did not exist.

Membership is checked each time a customer is evaluated: a static segment by its members at that moment, a dynamic segment by its condition over the event being evaluated and the customer's current attributes and balances. A customer who joins the segment later takes part from then on; one who leaves stops taking part (their variant does not change, only whether the experiment applies). The exp: segments of other experiments never count towards an audience.

For the results, a customer enters the experiment at their first activity after the start at which they are a member (see How assignment is counted). That membership is not evaluated again for measurement: when Orishare processes an event and finds the customer a member, it records that decision (the variant and the time it decided), and the results count exactly the customers delivery treated as members, from the earliest such event. So an audience condition that depends on time, attributes or balances gives the same answer for delivery and for the results. An event that was never processed by the rules engine (for example while rules are paused) does not enter anyone into an audience experiment.

The segment must exist when the experiment is created or started. A segment deleted or renamed later matches nobody, so the experiment then affects nobody.

The variant as a segment

While an experiment runs, every customer also carries a synthetic segment named exp:<experiment_key>:<variant_key>, for example exp:streak-holdout:treatment. It works anywhere a segment does, so you can branch without targets:

{ "path": "customer.segments", "op": "contains", "value": "exp:points-test:b" }

Use it in a rule condition, a campaign audience or a program audience. The segment is computed from the same assignment; it is not stored and cannot be edited.

Branching in your own app

Your app can read the signed-in customer's variants and change its own screens. This uses the customer surface: a publishable key and a customer session, never a secret key.

const res = await fetch(API + '/v1/customer/experiments', {
  headers: {
    Authorization: `Bearer ${publishableKey}`,
    'X-Orishare-Customer-Token': customerToken,
  },
});
const { data } = await res.json();
// [{ "experiment_key": "points-test", "variant_key": "b" }]
const variant = data.find((e) => e.experiment_key === 'points-test')?.variant_key ?? null;

A draft experiment is never listed. When an experiment is missing from the list (or the request fails), show your default experience. From a backend, GET /v1/customers/{id}/experiments returns the same list for any customer.

Lifecycle

POST /v1/experiments/{id}/start checks that every target exists and that none belongs to another running experiment, then starts delivery by variant. An experiment cannot be edited or deleted after it starts.

Change sets, agents and undo

Create, start, end and delete are also change set operations, so a person reviews them in the dashboard and an agent applies them under the environment's autonomy policy:

{ "commands": [{ "type": "experiment.start", "key": "streak-holdout" }] }

Propose with POST /v1/change_sets, apply with POST /v1/change_set_applications, undo with POST /v1/change_set_undos. The preview says in plain words who stops receiving what: for each variant its share of customers (or of the audience's members) and the targets it does not list, and for an end the consequence of the winner you chose. The targets are checked again when the change set is applied, so a resource another experiment took in between refuses the apply (rule_invalid). The plan binds the experiment's complete state (audience, variants, targets, goal and status), so any edit to it in between makes the apply preview_stale. An undo addresses the experiment by its id and is refused (invalid_state) when the experiment changed since the change set it undoes, so it never touches another experiment that took the same key.

In a live environment, an agent's own key cannot call the experiment routes directly unless the environment's autonomy is auto; it uses these operations instead. Under limits, creating and starting an experiment apply at once (one that withholds a rule, campaign or program from some customers is recorded as the advisory withholds_rewards, never refused), while ending one and deleting one (irreversible) wait for a person to confirm them in the dashboard. Undoing a start is an end, so under limits that undo waits for a person too.

Ending and winners

POST /v1/experiments/{id}/end takes an optional winner_variant.

• With a winner: every customer now receives the winner's targets. Targets that only the other variants had stay withheld from everyone until you pause or delete those resources yourself. This is deliberate: the losing program does not silently switch back on for all customers the moment you end the test.
• Without a winner ({}): all suppression is lifted. Every customer can receive every target again, exactly as before the experiment.

If a holdout variant wins (the program was not worth it), the program stays withheld from everyone; pause or archive it to finish the job.

Results

GET /v1/experiments/{id}/results returns, per variant, the customers assigned (units) and a list of metrics:

converted and goal_value are present only when the experiment names a goal.

How assignment is counted

A customer enters an experiment at their first activity at or after started_at. Customers who never show up while the experiment runs are not in the results, in any variant. Day-N retention is measured from that entry day (UTC days, as in Analytics), so day 7 figures cannot exist until 8 days after the start, and day 30 until 31 days after.

Results are recomputed by the hourly analytics build. Run POST /v1/analytics/builds to recompute at once.

Reading a result

{
  "metric": "converted",
  "units": 9000,
  "value": 0.25,
  "ci_low": 0.241,
  "ci_high": 0.259,
  "comparison": {
    "absolute_lift": 0.05,
    "relative_lift": 0.25,
    "ci_low": 0.02,
    "ci_high": 0.08,
    "p_value": 0.004,
    "significant": true
  }
}

• value with ci_low and ci_high: the variant's own figure and its 95% interval, the range the true value is likely to be in. More customers make it narrower.
• comparison (absent on the control): the variant against the control. absolute_lift is the difference (0.05 is five percentage points for a share); relative_lift is that difference as a fraction of the control's value, null when the control is zero.
• comparison.ci_low and ci_high: the 95% interval of the difference. If it does not include zero, the variants very likely differ. Here the variant is better by somewhere between 2 and 8 points.
• p_value: how likely a difference at least this large would be if the variants were really the same (two-sided). Small means "hard to explain by chance". significant is true when it is below 0.05.

The dashboard turns this into one of four verdicts: "Likely better" (significant, above zero), "Likely worse" (significant, below zero), "No clear difference yet" (not significant) and "Not enough data" (fewer than 30 customers in the variant or the control).

Two habits keep results honest. Decide how long the experiment runs before you start, and do not end it the first time a verdict turns green: checking repeatedly and stopping on the first significant reading finds differences that are not there. And prefer the interval to the verdict: "between 2 and 8 points better" says more than "significant".

Sample ratio mismatch

sample_ratio compares how many customers actually entered each variant with the planned weights (a chi-square test). mismatch is true when the split is very unlikely to be chance (p below 0.001). That usually means something other than the hash decided who showed up in which variant, for example a variant whose experience makes customers less likely to send any event at all. Treat the results with caution when it is flagged.

Not supported yet

• Changing weights, variants or targets while an experiment runs.
• More than one running experiment on the same rule, campaign or program.
• Revenue and lifetime-value metrics beyond the one goal value.
• Per-environment time zones: days are UTC.
• Sequential testing or automatic stopping. The p value assumes you look once, at the end.
• Customer merges do not move history: the surviving customer keeps its own assignment.