Use a split test when you want to compare one upsell strategy with a variation, or measure whether showing the offer performs better than showing no upsell at all.
Order Editing assigns eligible shoppers to one test group and keeps that assignment consistent. The test reports both strategies throughout the same test window, so you can make the decision from comparable traffic instead of comparing unrelated date ranges.
Before you begin
- Start with an existing Active or Draft upsell strategy.
- A strategy already assigned to another split test cannot be selected.
- Confirm the app is Live and the strategy's placement is installed before launching customer traffic.
- Choose one primary question. Keeping the main difference between Control and Variant B clear makes the result easier to interpret.
Create the test
- Open Upsell Strategies.
- Open Split tests or select Create Strategy > Create split test.
- Select Create split test.
- Under Strategy, choose the strategy to use as Control (A). Order Editing copies it to create Variant (B).
- Under What do you want to test?, choose:
- A/B test to compare Control with an editable Variant B; or
- Holdout test to compare the strategy with a group that sees no upsell.
- Set Variant B traffic. The default is 50%, and the slider supports 5% to 95% in five-percentage-point steps.
- Select the Primary Metric that should determine the winner.
- Under Automation, choose how a winner should be applied.
- Enter a required Test name.
- Add a Hypothesis (optional) if you want to record what you expect to learn. It can contain up to 250 characters.
- Review the launch checklist, then select Create test.
For a normal A/B test, Order Editing opens Variant B in a paused setup state so you can finish and review the variation. A holdout test starts running after creation because the no-offer group does not need a strategy to configure.
Configure Variant B
Variant B starts as a copy of Control. Change the part you want to compare, then save it.
Good comparisons include:
- the same products with different offer text;
- the same design with a different product module;
- the same product with a different discount;
- the strategy against a holdout that shows no offer.
Where possible, change one main variable. If the products, discount, copy, design, and targeting all change together, the test can identify a better version but cannot tell you which change caused the result.
Before launch, confirm Control and Variant B use compatible placements, markets, and status. Order Editing blocks save or launch when the two arms have a status mismatch that would make the comparison invalid.
Choose the primary metric
The primary metric determines the winner recommendation. Other results remain visible so you can check for trade-offs.
Primary metric | Use it when |
|---|---|
Conversion Rate | You want the largest share of visitors to accept an upsell. This is the recommended default. |
Revenue Per Visitor | You want to balance how often customers accept with how much revenue each visitor produces. |
AOV Increase | You want to maximize the average amount an accepted upsell adds to an order. |
Overall Score (Composite) | You want a weighted view of conversion, revenue, and order-value impact. |
Choose the metric before launch rather than selecting whichever result looks best afterward.
Configure evidence settings
Open Advanced settings under the traffic split when you need to change the statistical thresholds.
Setting | What it controls | Default | Range |
|---|---|---|---|
Minimum samples per variant | The response volume each arm should reach before winner guidance | 100 | 50 to 10,000 |
Confidence threshold | How much evidence the test requires before reporting significance | 95% | 90%, 95%, or 99% |
A higher sample or confidence requirement normally takes longer to reach. The create page uses recent traffic from the selected strategy to estimate how long the test may need.
Choose how the winner is applied
Winner application | What happens |
|---|---|
Manual | You pick the winner yourself. The test keeps running until you review the result and select a winner. |
Auto-apply as soon as the test reaches significance | Order Editing applies the recommended winner when the configured evidence threshold is reached. |
Auto-apply the current leader on a date | Order Editing applies whichever arm is leading at 9:00 AM in the shop's selected timezone on that date. |
A scheduled winner can be applied before the test reaches significance. Use a scheduled date only when ending on time matters more than waiting for conclusive evidence.
You can review or change the automation settings from the test page while the test remains active.
Launch and check the test
- Open the test after creating it.
- Review Control and Variant B.
- Confirm both strategies have the intended status and configuration.
- Preview each arm.
- Select Launch Split Test.
- Confirm the test status changes to Running.
For Checkout placements, add a suitable product to the cart and continue to checkout. For Thank You and Order Status placements, use Shopify's checkout-editor preview. For One-Click Upsell, use the Order Editing preview. You do not need to place orders simply to check the two versions.
Real customer traffic is required for meaningful test results. Do not create artificial orders to reach the minimum sample.
Monitor the results
The test page reports the performance of both arms over the same test window. Depending on the test and available data, you can review views, accepted offers, acceptance rate, revenue, Variant B gain, probability to win, sample maturity, device breakdown, and holdout incrementality.
Use the page actions to:
- Pause assignment without deleting the test;
- Resume split test after resolving an issue;
- save internal notes, tags, or learnings;
- Apply Winner when enough evidence is available;
- review completed tests in split-test history.
If one arm becomes Draft while the test is running, the page warns that the comparison is not being served correctly. Restore compatible statuses before relying on new results.
Apply a winner
- Confirm both arms reached a useful sample and review the selected primary metric.
- Check the other metrics for an important trade-off, such as higher conversion but lower revenue per visitor.
- Select Apply Winner.
- Choose Control or Variant B and record any useful learning.
- Confirm the action.
- Open the winning strategy and confirm its status, placement, targeting, products, offer, and design.
Applying a winner completes the test and keeps the selected strategy for future traffic. If automation applied the winner, review the history to confirm whether it was applied at significance or on the scheduled date.
If you cannot launch or apply a winner
- No strategy is available: create or activate a strategy, or finish the other split test that already uses it.
- Launch is blocked: resolve the checklist item, install the placement, turn the app Live, or make the two strategy statuses compatible.
- There is no winner guidance: wait for both arms to receive enough real traffic, or review whether the minimum sample is realistic for the strategy.
- A scheduled date cannot be saved: choose a future date. Scheduled application runs at 9:00 AM in the shop timezone shown on the page.
- Results look one-sided: confirm both arms remained eligible and Active throughout the same period before interpreting the comparison.
