Monday, July 06, 2009 7/06/2009 03:07:00 PM
When you run an experiment with Website Optimizer, you're testing new content to see whether it improves your conversion rate. We can often find big winners, but, of course, some of our variations are going to be losers in that they perform significantly worse than our original content.While you've been able to disable poor performing variations on your own for some time now, you needed to sign in, check your reports, and manually prune away the losers. No longer! Today, Website Optimizer adds a new feature: auto-disabling.
With auto-disabling, Website Optimizer will automatically prune poor performing variations for you. This keeps overall conversion rate high while testing. Additionally, by removing poor variations automatically, your traffic is focused on only the variations that have a chance at winning, which means experiments should complete faster. Lastly, we think auto-disabling gives you more freedom to try really big changes. If that crazy variation just doesn't work out, it'll be disabled. And if you've been anxious about running a test, you should be able to relax knowing that a losing variation won't hurt your overall conversion rate.
Auto-disabling is available for any Website Optimizer experiment with more than two variations (i.e. you can't use it for an A/B test, but you could for an A/B/C test). You can enable it from the experiment settings page. You can read more about setting up auto-disabling in this Help Center article.
15 comments:
is this like pruning 2.0? what's the difference between pruning and disabling?
Pruning and disabling are the same thing. You can think of this as auto-pruning.
Kudos on the new feature!
A screenshot of where the new feature is would be helpful for those that don't like to read :)
This is great, and even more great that you still allow us to keep all variants for analysis of cross-correlation across experiments
Yes nice work guys. Though to be honest, I'm a little disappointed as auto-pruning was going to be my suggestion for this year's "design a feature" contest... I thought I was shoo-in for the title two years running... back to the drawing board...
Very cool. I can see how this could save a lot of phone calls and emails, debating whether to prune.
If it wouldn't add too much complexity, it would be nice to be able to exempt certain combinations from pruning. (I might want a given combination to remain in the experiment even it it's performing poorly.)
Great new feature. Thanks!
Hey Neal, you'll need to be more clever this year.
Michael, thanks for the suggestion. I'll pass it along to the team.
great post! Love the feature
Haven't been using the Google Website Optimizer since it gave problems with IE7 and forms visibility due of the scripting code... will give it another shot again with this new feature... New feature looks cool.
OK, I gave it a try and the pruning seems to only prune if the combo is in the red. How do I make it prune faster?
There are three different settings of auto-disabling. Moderate and Conservative both wait for a variation to be in the red. Aggressive will prune things before they turn red but too much before. We don't want to prune things too early because they could come back as winners.
Of course, you can always manually prune away variations that aren't performing to your expectations.
Love this feature. A suggestion: Taguchi pruning! Automatically disable a version that contains a section that isn't working. E.g., if image X is doing badly in all versions of the page, I want to disable all page versions that contain that poor performing image, even if the version itself is in the yellow?
Neat feature idea! Speaking from personal experience though, I've seen winning combinations that contain a losing section, so I wouldn't want it on all the time. You can never be sure how different content will interact.
That sounds a bit big brotherish. Why don't you let us decide when something should be automatically deleted? Basically you're saying I should log in every hour and I can delete it myself manually but if I want to use your auto-disable feature, I can only use it when you say it's ok :-(
Still love the feature, just want more control over what happens!
You're in complete control of how your experiments are pruned.
With manual pruning you can, at any time, sign into Website Optimizer and prune away a variation manually. What often happens though is that experimenters feel the need to sign into their experiments very often to make sure there isn't a poor performing variation.
With auto-disabling, this isn't an issue since GWO will automatically prune away poor performing variations. What's important to note here is that you can turn auto-disable on or off whenever you'd like. You also can set the threshold for auto-disable between three different levels.
So you have pretty much complete control over what happens.
Post a Comment
You are welcome to comment here, but your remarks should be relevant to the conversation. We reserve the right to remove off-topic comments, or self-promoting URLs, and vacuous messages. Please keep your comments relevant, friendly, and respectful.
We moderate all comments so you may not see your comment show right away.