Testing is not just about Technology

Monday, April 21, 2008 4/21/2008 11:46:00 AM

Posted by Tom, Website Optimizer Blog Team

One of the things we've learned is that to be a successful experimenter, you need to master a few key concepts. These concepts include diagnosing problematic pages (Google Analytics is a good tool for this), coming up with ideas on how to improve these pages, and understanding how to use Website Optimizer to instrument tests. Last but not least, you need to have the discipline to wait for your tests to gather enough data before drawing the right conclusions. Let's break down each one.

Diagnose and Develop Ideas:
Look at a page that gets a lot of users. Does it have a high bounce rate? What's wrong with it? What's stopping people from continuing to dig further into your site? What are they expecting to see? Is it there? What's there that isn't important? If you asked visitors what they expect to see before the page loads, is their answer even close to what you're showing them? Figuring out what's best to test can be as much an art as it is a science, but it may be the difference between a 30% improvement in your page performance versus a test which concludes that the various versions of your page performed about the same.

Instrument the Test:
All the ideas in the world won't help you if you don't know how to set up the test. We have a bunch of demos and help articles to show you how to do it, but the best way to learn is simply to start using the tool to set up lots of tests. Before you know it, you'll be trying all kinds of advanced experiments. If it's your first one, we'd recommend a simple A/B test where you give Website Optimizer two or more URLs and we split the traffic among them.

Wait:
Sometimes the hardest part of running a test is not jumping to conclusions. Let the horses run a few laps around the track before declaring a winner. Website Optimizer will tell you the chance to beat your original for each variation. The lower that number is, the more likely the pages are simply not that different. We like to wait a minimum of two weeks and for a 95% 'Chance to beat original' before declaring a solid winner, but it depends on how sure you want to be.

Keep in mind that it's possible your test may never determine a winner or that the winner may even be your original page. This could happen if all your pages perform about the same or your original page is really better than the other ideas you tested. Sometimes those tests are the most valuable since you learn what not to show your visitors.

For ideas on a few top elements to test on any site, check out our recent webinar, 'Website Optimizer, What Should I Test?'.