How to do A/B testing with full page caching?

Question

I'm trying to find the best approach to perform some A/B tests on a site currently being built. The site is hosted on WPEngine and uses Wordpress. They use full page caching, and that should be enabled anyway. A/B tests require different HTML for each variant. As the element that changes is the site's main menu, it would be bad to dynamically load it by AJAX with another HTTP request.

I can't imagine any other approach than a completely client-side one. My idea is to include both variants in the page and display only one.

<script src="set-variant.js"></script> <!-- sets variant variable -->
<div id="variants">
<!-- <div id="variant1"></div> -->
<!-- <div id="variant2"></div> -->
</div>
<script>

if(variant==1){
   document.write(retrieveVariant1FromComment());
} else {
   document.write(retrieveVariant2FromComment());
}
</script>

It works, but it looks odd (I would rather avoid document.write...) and also I'm not sure about SEO.

Does anyone know a better way to do this when pages are cached? Any recommendations?

Answer 1

The general approach you're using, to render variant content in Javascript, is common, and is how tools like Optimizely work. (AJAX might be involved, but it might be organized and optimized to not be a problem.) You might just want to use Optimizely or something similar, and doing that might also allow your organization to do A/B tests without changing your application code every time.

Here are two different server-side approaches:

• Put one variant at a different URL (probably sharing code behind the scenes rather than actually copying the code for the entire page) and redirect users who should see that variant to that other URL. Google Analytics Content Experiments works that way. The obvious disadvantage is that the user sees a different URL, and might bookmark it, so you might need to redirect from the variant URL back to the normal URL long after the experiment is over.

• If you're using a caching system that allows you to specify the cache key, just add the variant to the cache key. The page will be rendered the first time each variant hits it and cached thereafter. I don't know whether this is possible with your tools.

Regarding SEO, Google does run Javascript, so don't show variants that you don't want to be indexed to search engines. (I don't know how Google's search engine handles Google Experiments.)