Skip to main content

A/B testing (Scale plan)

The A/B testing module runs two rules of the same campaign side-by-side as control and variant - say, $50 threshold vs $75 threshold, or tote bag vs travel bag - and reports which one drives more RPV (revenue per visitor - total promotion revenue divided by the number of shoppers who saw the variant; a lift-friendly metric that adjusts for variant traffic differences automatically).

Experiments vs campaigns​

Quick clarification on terminology:

  • A campaign is your main promotion - e.g. "Spend $50, get a free tote." Every Valotrix Cart Rewards store has campaigns.
  • An experiment is an A/B test within a campaign - e.g. test "Spend $50" vs "Spend $75" inside the same campaign. You need at least two saved rules in the campaign before you can experiment between them.

Experiments live inside the campaign editor - each campaign edit page has an A/B Testing card. You don't create experiments in isolation; you pick an existing campaign first.

How to create an A/B test​

  1. Open the campaign you want to test from the campaign list.
  2. Make sure the campaign has at least two saved rules (e.g. two thresholds or two gift options). Newly-duplicated rules can't be picked as control / variant until the campaign has been saved.
  3. On the campaign edit page, scroll to the A/B Testing card (Scale plan only - lower plans see the card with an upgrade banner instead).
  4. Pick the control rule (the baseline) and the variant rule (the challenger).
  5. Give the variant a short, dashboard-friendly name (e.g. "Higher threshold" or "Tote bag").
  6. Choose a primary metric: RPV (revenue per visitor) is the default and is usually what you want; AOV (average order value) is right when you specifically care about basket size; Conversion rate is right when you're testing whether the gift drives more checkouts at all.
  7. Confirm and start. The experiment moves to running state and visitors get bucketed on their next storefront visit.

What you choose when starting an experiment​

  • Control and variant rules. v1 supports exactly two variants (control + variant). Both must be persisted rules on the same campaign - newly-duplicated rules that share IDs aren't pickable until the campaign is saved.
  • Variant name - a short label that shows on the dashboard.
  • Primary metric - one of three: RPV (revenue per visitor, the default), AOV (average order value), or Conversion rate (promotion orders / bucketed visitors). Pick the one that matches the merchant question you're testing.

Status state machine​

The experiment moves through five states (rendered as colored badges on the dashboard):

  • draft - set up but not yet exposed to traffic.
  • running - visitors are being bucketed; events are being recorded.
  • stopped - manually halted; results frozen but no winner declared.
  • concluded - a winner was declared (either by passing the 95% Bayesian threshold or by manual call).
  • archived - read-only historical record.

How visitors are split between control and variant​

  1. Visitor identity. Valotrix Cart Rewards gives every visitor a stable ID. Logged-in customers are identified by their Shopify customer ID; logged-out visitors get a long-lived ID stored in their browser. The same person is bucketed consistently across visits and devices once they log in.
  2. Pick a variant. A fingerprint of the visitor's ID is mapped onto your traffic split. A 50/50 split sends half the fingerprints to control, half to variant; you can also do 70/30 or 90/10 - any split that adds up to 100%.
  3. Remember the choice. When the visitor adds a gift, their variant assignment is remembered in the cart so the same person sees the same variant across page loads, refreshes, and the rest of their session.

Each experiment uses a fresh fingerprint, so re-running an A/B with the same campaign produces different visitor assignments - there's no carry-over from prior experiments.

How the winner is decided​

The dashboard shows a running Bayesian chance to win for each variant - our math estimating how likely the variant is to beat the other on your chosen metric, given the data we've collected so far. We use a Bayesian method (closed-form Beta-Binomial) so the number updates smoothly as orders come in; the full math is in the DevSource block at the bottom of this page.

In plain terms: if Variant A shows "78% chance to win", it means there's roughly a 78% probability that A is actually better than B at this moment - not that A has converted 78% of its visitors. The experiment auto-flags as ready to conclude when one variant crosses 95% chance to win; you can let it keep running, stop it manually, or declare a winner whenever you're ready.

A nightly job recomputes the chances against the latest order data. Even high-volume stores see results refresh well within a minute of the job kicking off.

Plan-tier gating​

A/B testing is Scale-plan only. On Free, Growth, and Pro, the A/B Testing card on the campaign edit page shows an "Available on the Scale plan" banner - no experiments can be created or run.

Privacy and data retention​

The visitor ID is essential for the experiment to work - without it, we can't keep a visitor in the same variant consistently across visits. It's not used for tracking, advertising, or fingerprinting; storing it doesn't trigger a cookie-banner requirement. When a customer requests deletion under GDPR, all of that customer's experiment records are stripped of personal data within 30 days. See Plans, billing & cancellation for the full retention picture.

Known v1 limits​

  • Two variants per experiment (control + variant). More are on the roadmap.
  • Bot traffic isn't filtered out yet. Googlebot, link-unfurl bots, and uptime-check bots get bucketed and counted, which can slightly depress conversion rate / RPV / AOV at low sample counts. Real-customer numbers stabilize as sample size grows.
  • The simulator can't force a specific variant today. Preview by sending traffic to the live preview URL - the bucketing applies the same way.
  • Renaming a variant mid-run is supported but not from the running dashboard inline yet - open the experiment edit page to rename. A small future polish.

Next: Per-customer limits →