# A/B Testing

> Source: https://docs.recombee.com/ab-testing

> For the complete documentation index, see [llms.txt](/llms.txt).

# A/B Testing

A/B Testing lets you compare different recommendation configurations within a Scenario and measure their effect on your key metrics. Each experiment runs on top of a specific Scenario, with traffic split between a **Control** (the current Scenario configuration) and one or more **Variants**.

## Setting Up an Experiment

You can create an experiment either from the **A/B Testing** tab inside a Scenario's detail view, or from the standalone **A/B Testing** section in the Admin UI.

When creating an experiment, provide a title (or leave it blank to have one generated automatically), then select the **Scenario** it should run on. Set the **start date** — and optionally an **end date** — to define the experiment's time window.

[![Create Experiment form](/img/ab_testing/ab_testing_create.png)](/img/ab_testing/ab_testing_create.png)

### Metrics

Metrics define what you want to measure. You must choose at least one metric before the experiment can be created.

Recombee provides a **library of predefined metrics** organized by vertical (Video, E-Commerce, Articles & News, etc.) — including metrics such as _Conversion Rate_, _Click-Through Rate (CTR)_, _Watch Time from Recommendations_, and _Income from Recommendations_.

[![Choose Metrics panel](/img/ab_testing/ab_testing_choose_metrics.png)](/img/ab_testing/ab_testing_choose_metrics.png)

You can also create **custom metrics** based on [Insights](/insights). A custom metric lets you specify a data source (e.g., _Detail Views from Recommendations_) and apply filters — for example, to track views from recommendations only for items from a particular category, or to measure the purchase rate exclusively for items above a certain price threshold.

### Variants

Each experiment has a **Control** — the Scenario's current configuration — and one or more **Variants** that modify it.

When creating a Variant, you can change any combination of:

* **Logic** — the whole recommendation model ensemble or its individual parameters
* **Filters** — which items are eligible to be recommended
* **Boosters** — how items are ranked within results
* **Constraints** — limits on how frequently certain [Segment of items](/segmentations) appear

The Admin UI clearly shows what has been changed in each Variant relative to the Control.

[![Create Variant panel](/img/ab_testing/ab_testing_create_variant.png)](/img/ab_testing/ab_testing_create_variant.png)

Once Variants are defined, assign a **traffic allocation** (percentage) to each using the sliders. Use the **lock icon** to fix a Variant's allocation so it is not affected when adjusting others. Click **Distribute Evenly** to split traffic equally across all Variants.

[![Variants allocation sliders](/img/ab_testing/ab_testing_variants_allocation.png)](/img/ab_testing/ab_testing_variants_allocation.png)

The sliders always add up to 100%, so changing one row moves the others. A row you have typed a percentage into is locked automatically, which keeps that value from being redistributed away when you adjust another Variant — click its lock icon to release it.

[![Adjusting variant traffic allocation with the sliders, explicit percentages and the lock](/img/ab_testing/ab_testing_variant_ratios.webp)](/img/ab_testing/ab_testing_variant_ratios.webp)

## Report

The report shows the effect of each Variant on every selected metric, including whether the results are **statistically significant**. For each metric, the table displays:

* **Users** — number of users in the Variant and their share of total traffic
* **Improvement** — change relative to the Control baseline
* **Probability to be best** — likelihood that this Variant outperforms all others
* **Probability to beat baseline** — likelihood that this Variant outperforms the Control
* **Distribution** — the mean and variance of the metric's value across users

The best-performing Variant is highlighted in the results.

[![A/B Testing results report](/img/ab_testing/ab_testing_report.png)](/img/ab_testing/ab_testing_report.png)