Menu

A/B Split Test

Compare a control group of URLs with a treatment group over a fixed period.

Updated

On this page

What it shows

An A/B split test compares two groups of URLs:

  • Control: pages that you do not change.
  • Treatment: pages that you change.

After the test period, the tool compares the two groups with statistical tests.

This is a test on groups of pages. It is not a test that shows different versions of a page to different visitors.

When to use it

  • You want to test a change, for example a new page template, on some pages first.
  • You have similar pages that you can put in two groups.

How to use it

  1. Open Optimize → A/B Split Test, then start a New A/B Split Test.
  2. Groups: add Control Pages and Treatment Pages, with Manual or Pattern input. Each group needs at least 3 URLs. A URL cannot be in both groups. A test can have up to 50 URLs in total.
  3. Select a Test Duration: 7, 14, 30 (the default), or 60 days.
  4. Validate: check the URLs.
  5. Run: start the test. The test starts on the day that you create it, so make the change on the treatment pages on the same day.
  6. After the test period, open the test to see the results.

Make the two groups as similar as possible, for example the same page type and similar traffic.

How to read the result

ItemMeaning
Group ComparisonClicks, impressions, average CTR, and average position for each group.
p-valueThe result of a Mann–Whitney U test on the CTR of each URL. A smaller value means that the difference is less likely to be chance.
Confidence(1 − p-value) × 100.
Statistical SignificanceAbove 95% threshold when the p-value is less than 0.05.
EffectCohen’s d on the CTR of each URL: negligible (less than 0.2), small (less than 0.5), medium (less than 0.8), or large.
Confidence IntervalsA 95% range for the difference between Treatment and Control, for clicks, impressions, CTR, and position.
Page-Level BreakdownThe values for each URL. Look for one page that changes the total.
VerdictMeaning
Likely winnerTreatment had a higher CTR than Control, and the test is significant.
Negative signalTreatment had a lower CTR than Control, and the test is significant.
InconclusiveThere is no reliable winner yet.

PDF exports the result.

Example

All data is fictional. On example.com, you put 10 product category pages in Control and 10 similar pages in Treatment. You add an FAQ block to the treatment pages, and run the test for 30 days.

The result shows a p-value of 0.21, a small effect, and the verdict Inconclusive. The CTR interval goes from below zero to above zero. Thus, the test does not show a reliable difference. You can run a longer test, or test with more pages.

Limits

  • A small p-value does not prove that your change caused the difference. Other events can affect one group more than the other.
  • With few URLs, the test can miss a real difference.
  • The verdict uses the CTR test for direction and significance. Read the Group Comparison too.
  • An inconclusive result is a valid result. It is not an error.
  • All running A/B tests in one workspace can use 250 URLs in total. Above this, the App shows “URL Limit Exceeded. You can only add 250 active URL’s for A/B Tests.”

Choose which optional cookies we may use on kongmetrics.com and in the Kong Metrics App. You can change this at any time from "Cookie preferences" in the footer. Cookie Policy