Skip to content

Balance the split within groups (beta)

An experiment that balances its split within groups gives each group of users the same split as the whole experiment, as they are assigned. With a 50/50 split balanced within tier, the users of each tier are split 50/50 among themselves, so a lucky mix of tiers cannot tip the result. You set balance_within on the experiment, and send the attribute in each assign request's context.

How it works

Within each group, each new user takes the next place in a block, a shuffled list of the experiment's split. Every user's chance of each variant is still its share. For the users balanced in the blocks, a variant's count in a group stays within a few users of its share at every moment, and is exactly its share each time the group completes a block: at 50/50, the two variants of a group never differ by more than one user.

The global holdout, the exclusion group and the targeting rules come first: only users who take part are balanced. A user keeps their variant and their group if their attribute changes later. An experiment that does not set balance_within assigns its users exactly as before.

Turn it on

Send balance_within when you create the experiment, or with PUT /api/v1/experiments/{id} while it is a draft:

{"balance_within": {"attribute": "tier", "values": ["high_value", "churn_risk", "new"]}}

The response echoes balance_within. Check that it does: a server version without this feature ignores the field and still answers 200.

  • attribute is a top-level key of the assign context, such as tier or country.
  • values lists the values to keep apart, 1 to 20 of them, each matched exactly: Gold and gold are different values.
  • A user whose value is not listed is balanced with the others in other, and a user without the key in unknown.
  • null turns balancing off on a draft. Once the experiment has started, balance_within cannot be changed.
  • A clone balances within the same groups, and its groups start from their first place.

In the dashboard

The Variants step of guided setup and the single-page form have Balance the split within groups, off until you turn it on. A draft's Edit variants and metrics, on the experiment page, has the same switch.

  • Group users by offers Platform, Country and User Tier, Another attribute… (type its name) and Saved segments….
  • For an attribute, Values to keep apart (up to 20) takes one chip per value. An attribute named web_device, web_visitor or web_browser needs no values.
  • For saved segments, Add a saved segment adds one at the end of the list, and each can be moved up or down: a user is balanced in the first segment in the list they belong to.
  • A split with a share that is not a multiple of 5, such as 34/33/33, shows the refusal under the allocations as you type it. Next and Create Experiment wait until it is fixed.
  • When the API refuses the setting, its own sentence is shown at the control it is about, and focus moves there.
  • A draft that is a bandit or a split URL experiment shows a sentence in place of the switch.
  • The experiment page's Balance row says what the split is balanced within, such as within Platform (ios, android, web), or off. Once the experiment has started the row is all there is: the setting cannot be changed.

Assign users

Put the attribute at the top level of context in every assign request:

{"experiment_key": "pricing-test", "user_id": "u-123", "context": {"tier": "high_value"}}

A nested key ({"user": {"tier": "high_value"}}) or a dotted one (user.tier) is not read, so that user is balanced in unknown. POST /api/v1/tracking/assign and POST /api/v1/tracking/assign/batch take their places from the same groups, so users assigned one at a time and users of an uploaded list are balanced together (see Assign a customer list). The same list sent in another order can split differently; each user keeps the variant they got.

Balance within saved segments

Instead of an attribute, balance_within can list 1 to 10 saved segments by their id:

{"balance_within": {"segments": ["8d4c6f0e-3a1b-4c2d-9e8f-7a6b5c4d3e2f", "1f2e3d4c-5b6a-4978-8a9b-0c1d2e3f4a5b"]}}
  • A new user is balanced in the group of the first listed segment they belong to, so for a user in two of them the order of the list decides. A user in none of them is balanced in other.
  • Membership is decided as it is for targeting rules: an ID list by the assign request's user_id, a rules segment by its context.
  • When the membership of a segment listed before the first one the user belongs to cannot be decided, the user is balanced in unknown.
  • A user keeps their variant and their group if they leave the segment later.
  • Every segment must exist and be active, and a rules segment's rules must be valid.
  • A segment that a draft, active or paused experiment balances within cannot be archived or made inactive: the request answers 409, as for a segment that targeting rules use. A clone answers 409 when a segment it would balance within cannot be used.

What is refused

The request answers 422 with the reason when:

  • the attribute has a dot, has a character other than letters, digits, _ and -, or is user_id or segment;
  • values is missing (web_device, web_visitor and web_browser need none: their values are a fixed set), has more than 20 values, lists other or unknown, or lists a value twice;
  • a share of the split is not a multiple of 5, such as 34/33/33 (50/50, 20/80, 35/35/30 and 40/30/30 are allowed);
  • the experiment is a bandit or a split URL experiment;
  • both attribute and segments are sent, or neither is;
  • segments has more than 10 ids, lists one twice, or names a segment that does not exist, is archived or inactive, or whose rules are not valid.

Changing balance_within after the experiment has started answers 400.

Users assigned without balancing

When the server is busy balancing other users at that moment, a new user is assigned by the experiment's ordinary hash instead. They keep their group, their chance of each variant is still its share, and they are counted in the experiment_balance_fallback_total metric by reason (timeout or busy). An older server version, during a rolling deploy or a rollback, assigns every new user by the hash and stores no group. Limitations lists what balancing does not do. The results by group count both kinds in users_outside_blocks.

Results by group

GET /api/v1/results/{experiment_id}/balanced-groups (beta) gives the experiment's results by the group each user was balanced in, the group stored when they were assigned. The experiment's own results and their decision do not change.

  • groups has one row per group, built as a breakdown's rows are: the groups with the most users, then other and unknown, each with a p-value and the Bonferroni threshold over the rows returned. Per variant, the rows add up to every user assigned.
  • overall gives one estimate per treatment of the primary metric: each group's difference from the control, weighted by the group's share of all the users. Its interval is at the experiment's confidence level, and the treatments are corrected together with the experiment's correction method. No CUPED adjustment is applied to it.
  • overall is null, with reason, for a primary metric that is not a conversion (mean_metric), for an experiment with sequential testing (sequential_testing), and when no treatment has users or no group has 2 users in every variant that has users (too_few_users).
  • A group with fewer than 2 users in a variant that has users is counted with the largest group that has 2 in every variant that has users, and listed in merged_groups.
  • users_outside_blocks is the number of users split without balancing. unknown_share is the share in unknown of the users whose group was stored.
  • An experiment that does not balance within groups answers 409.

How they differ from ?breakdown=

GET /api/v1/results/{experiment_id}?breakdown=tier puts each user in the value of their first event that carries tier (Where the segments come from). For users assigned through POST /api/v1/tracking/assign with a listed value, the two agree. Otherwise they can differ:

  • Users assigned from a list have no exposure, so the breakdown counts them under unknown until they send an event with tier; here they are in their group.
  • A user assigned without tier who later sends an event with it is in that value in the breakdown, and in unknown here.
  • A value that is not listed, a decimal number such as 3.0, a list or an object, or text of 101 to 255 characters is other here, and the value itself in the breakdown.
  • With more than 20 distinct values the breakdown can count a listed value under other; here every listed group has its own row.
  • A user an older server version assigned has no stored group, so they are in unknown here.

Method

The overall estimate is the stratified difference in conversion rates (Cochran, Sampling Techniques, chapter 5): the sum over the groups of each group's share of the users times its treatment rate minus its control rate, with the variance summed the same way from each group's two rates and the group's share squared. It equals the treatment's coefficient, and its HC2 standard error, in one regression over every variant with the group indicators centred and interacted with each treatment.