A/B Testing Experiments

A/B testing experiments are controlled tests in which you run two or more versions of a web page, advert, email or other digital element simultaneously to measure which variant performs better against a pre-selected objective. You randomly allocate your visitors across the variants and objectively measure which version generates more conversions, clicks or revenue. This allows you to make marketing decisions based on behaviour, not on assumptions.

How does an A/B test work in practice?

You start with a hypothesis: for example, that a green button converts better than a blue one. You then create two identical versions of your page, differing only in the colour of the button. Half of your visitors see variant A, the other half variant B. You run the test until you have collected enough data to draw a statistically reliable conclusion. Tools such as Google Optimise, VWO or Optimizely automatically split the traffic and calculate the significance. Only when a variant performs better with at least 95 per cent certainty should you implement it permanently. With small and medium-sized companies, we often find that a test needs to run for at least two weeks to account for seasonal influences and weekly patterns.

Why A/B testing came about and why it matters now

A/B testing originated in the pharmaceutical industry, where placebo-controlled studies are the norm. Online marketing adopted this principle when web analytics made it possible to measure how visitors behave. Instead of guessing which homepage text works better, you simply test it. This is relevant for Dutch SMEs because, with a limited budget, you want to get the maximum return from your website or online shop. A single successful test can increase your conversion rate by 10 to 30 per cent without any additional advertising budget. Over a period of months, that difference adds up to thousands of pounds in extra turnover.

What A/B testing experiments deliver for small and medium-sized companies

With A/B testing, you can gradually improve your website or campaign based on real-world behaviour. For example, you can test different headlines, call-to-action texts, product photos or checkout flows. Every improvement adds up. In practice, we’ve found that testing one element at a time yields the most insights for our clients: this way, you know exactly what’s causing the difference. A/B testing is a core component of a well-thought-out SEO strategy and conversion optimisation. You can also carry out tests in adverts, email marketing or your online shop’s product pages. It takes discipline not to stop prematurely, but the return on investment is measurable and repeatable. Want to know which element on your site has the most potential? Link A/B testing to web analytics and you’ll immediately see where visitors are dropping off.

A/B testing experiments aren’t just for large online shops. Even SMEs with a limited number of visitors can set up targeted tests that provide quick insights. The trick is to focus on elements that have a direct impact on your conversion goal: a quote request, a purchase, a newsletter subscription or a phone appointment. Below you can see where A/B testing delivers the greatest results in practice.

Optimising landing pages for advertising campaigns

If you’re paying for Google Ads or social media adverts, you want every click to deliver a return. By testing your landing page, you can increase your conversion rate without spending any extra advertising budget. For example, test the main headline, the length of the form or the position of trust signals such as quality marks. An online shop selling garden products tested a shorter headline highlighting an immediate benefit against a longer one featuring storytelling. The shorter version generated 18% more orders. By cleverly combining SEO and adverts with A/B testing, you can get more out of the same budget. Make sure you have sufficient traffic: with fewer than 100 conversions per month, a reliable test often takes four to six weeks.

Improving email marketing and newsletters

In email marketing, you can test almost anything: subject lines, sender name, the opening text, the call-to-action button or the time of sending. Platforms such as Mailchimp and ActiveCampaign offer built-in A/B testing features. A B2B service provider tested two subject lines: one phrased as a question and one as a statement. The question-based subject line resulted in 12% more opens. They then tested the button text within the email itself and increased the click-through rate by a further 9%. By testing step by step, you can build an email strategy that really works for your target audience. Don’t forget to factor in attribution modelling: sometimes an email doesn’t lead directly to a purchase, but it does lead to a conversion at a later stage.

Streamlining checkout flows in online shops

Abandoned basket rates often range between 60 and 80 per cent. Any improvement to your checkout process directly generates more revenue. Test whether a guest checkout works better than mandatory registration, whether a progress indicator helps, or whether omitting an extra field increases the number of completed orders. An electronics online shop tested removing the ‘company name’ field from the checkout. The result: a 14 per cent increase in completed orders, as private buyers felt less scrutinised. Combine A/B testing with professional online shop design and you’ll maximise conversion without needing to increase visitor numbers. Just make sure you test against a sufficient number of transactions: with fewer than 50 orders per week, a test could take months.

When A/B testing experiments are the right choice and when they are not

A/B testing works best if you have sufficient traffic to obtain a reliable result within a reasonable timeframe. As a rule of thumb: at least 100 conversions per variant. Do you have a niche market with few visitors? In that case, you’re better off focusing on qualitative user research or heatmaps. A/B testing is also not suitable if you want to change too many things at once: test one element at a time, otherwise you won’t know what’s causing the difference. Avoid running tests during periods affected by seasonal factors or campaigns that skew traffic. And never stop a test early just because variant A is performing better after two days: that’s coincidence, not proof.