A/B testing on Shopify: tests that carry a decision

An A/B test answers exactly one question: does variant B produce a different value in a metric defined up front, compared to variant A, with a comparable audience. It compares two states under the same conditions. Anything beyond that is interpretation, not measurement.

What a test does not deliver is a reason. You learn that one variant performs better, but not why. The cause lives in user interviews, session recordings and support tickets. Reading the reason out of the result means building your next hypothesis on a guess.

A test also says nothing about periods it did not cover. A result from a promotional week applies to promotional weeks. A result from summer does not automatically hold during peak season. Ignoring that means transferring findings to situations in which they were never checked.

The sample size is the point at which you stop looking. Decide it while the test is running and you will unconsciously pick the moment when the numbers happen to look good. At that point the result is no longer a measurement but a selection from many possible interim states.

Four quantities determine how much data you need. The baseline conversion rate of the control, the smallest effect that would genuinely change your decision, the significance level as your tolerance for a false alarm, and the power as the probability of detecting a real effect. All four are choices, not natural constants.

The second one matters most and gets skipped most often. Not every measurable difference is a relevant difference. Ask yourself before launch: at what improvement would you actually roll the variant out and pay for the work involved. That number is your minimum detectable effect.

The logic is easiest to show with a pure arithmetic example. The values below are freely chosen assumptions and say nothing about your store. They exist only to make visible the order of magnitude such calculations typically land in.

Assume a baseline conversion rate of two percent, a relative improvement of ten percent that you want to detect, a significance level of five percent and a power of eighty percent. You then need a visitor count per variant in the range of several tens of thousands, roughly eighty thousand under those assumptions.

The practical consequence is uncomfortable. The smaller the effect you want to detect, the faster the requirement grows, and not in a linear way. Halve the effect you are looking for and the required sample roughly quadruples. That is why many testing programmes fail at the arithmetic, not at the implementation.

Runtime follows from two conditions that both have to be met. The planned sample size must be reached, and the test must cover full week cycles. If the sample is already there after four days, you keep running until the end of the week anyway.

The reason for full weeks lies in buying behaviour. Weekdays and weekends bring different people with different purchase intent. A test that starts on Wednesday and ends on Monday weights certain days twice. That does not distort both variants evenly when traffic sources shift across the week.

There is an upper limit too. Over many weeks, disturbances accumulate: campaigns launch, prices change, cookies expire and visitors end up in the other variant. What started as clean randomisation becomes a blend. Two to four full weeks is usually the workable corridor.

The effect is called peeking and it is the most common reason results fail to replicate later. Checking the report daily and stopping as soon as a significance value briefly crosses the threshold means you are not testing once but testing again every day. The error probability adds up with every look.

The interim curve makes this visible. In the first days the measured difference swings widely because a handful of orders carry a lot of weight. Those swings are noise, not signal. Reading them as a result means rolling out variants whose lead dissolves once more data arrives.

There are two clean ways out. Either you fix the analysis to a single point in time defined up front and look at nothing but technical error indicators before then. Or you use a method that accounts for repeated looks, such as sequential tests whose thresholds are built for exactly that. Mixing the two is not a solution.

Not every question deserves a test. Some changes are too small for the traffic available, others are not a legal or commercial option in the first place, and some endanger the very measurement you need to read the result. In all of those cases, deciding without a test is the better move.

Anything touching the checkout or the price display is particularly delicate. Differentiating prices between randomly assigned visitors breaks trust and creates tangible problems the moment two people in one household see different prices. Test discount levels through campaigns separated in time, not through parallel groups.

Client-side tests change the page only after it has loaded. Visibly that shows up as a brief jump from variant A to variant B, usually called flicker. It distorts more than perception: it distorts the metrics too, because part of your audience sees the control and then something else.

The second trap is assignment across consent. Visitors who do not consent appear differently in your analytics tool than in your testing tool, or not at all. When assignment and measurement work on different populations, the denominators of the two variants stop matching and the rate becomes worthless.

On top come Shopify-specific conditions. The checkout is largely no longer a freely editable area but runs through defined extension points. Caching and multiple active theme variants can serve visitors different versions between page views. And a noticeable share of traffic is bot traffic, which can distribute unevenly across the two groups.

A test result is not a single number but a range. The measured difference is the most likely value; the confidence interval shows the span you have to reckon with. If that interval runs from slightly negative to clearly positive, you have no basis for a decision, however good the point estimate looks.

The biggest trap after the test is post-hoc segmentation. Split an inconclusive overall result by device, channel, country and new-customer status long enough and a significant slice will always show up somewhere. With enough cuts, a chance finding is the rule rather than the exception.

Segments are not forbidden, they just have to be named in advance. If you write down before launch that you will analyse mobile and desktop separately, that is a hypothesis. If the split only occurs to you while looking at the results, it is a search for confirmation and should be recorded as such.

For many stores in the DACH region this is the normal case, not the exception. If the arithmetic says a test would have to run for months, that is not a reason for a worse test but a reason for a different approach. A test with too little data does not give a weaker answer, it gives an unreliable one.

The first way out is to test larger changes instead of smaller ones. A rethought product page layout can produce an effect that shows up even with modest traffic. A different button colour will not. With little data you have to change more boldly, otherwise you measure noise forever.

The second way out leads away from testing. Many improvements need no randomised proof because they fix obvious defects: broken forms, missing shipping information, unusable filters, slow loading on mobile. You repair those things and watch the trend afterwards instead of pitting them against a variant.