AB Testing… the Wizard of Oz technique

If you are thinking about running an AB test to try a new feature or new design out you must do it quick and smart. As many of you have read in Lean Startup book, it’s important to create a minimum viable product before a complete launch to the market of the feature/tool that you’re implementing into your website. To do this, you should use the Wizard of Oz technique. Here is a good definition of this technique:

User-based evaluation of unimplemented technology where, generally unknown to the user, a human or team is simulating some or all the responses of the system.

The technique has often been used to explore design and usability with speech systems, natural language applications, command languages, imaging systems, and pervasive computing applications.

In other sort of words: you must create a fake product. There are 3 characteristics that your fake product or prototype must have:

1. It doesn’t have to be complete
2. It should be easy to change or iterate on-the-fly with new requirements
3. It’s easy to retire if shit happens (and shit happens so often)

You don’t want to waste a lot of your team’s time creating something that, probably, you won’t ever implement definitely in your website. Because, to be honest, the major part of the AB tests that you’ll run in your company will fail. For example, if you want to change the layout of your website and create a new design, you’re not required to full equip the test version; you can skip those useless parts of the website that in terms of costs are expensive to recreate. You must focus on your main goal, which brings me to the main question you should do yourself:

What do you want to learn with this test? What’s the main goal?

You must think about the main goal you’re looking for with the test. And usually, the goal is related with metrics. Never forget that websites are metrics: visitors, pageviews, items sold, clickouts, etc… You must take care of all your metrics in each test you decide to run: No metrics, no evolution. You must think in afterwards and if what you are testing will help you to do an important breakthrough or not. Errors, usually are a goldmine, sure, but always taking care of the path you must follow.

Coming back to the Wizard of Oz technique, try to have:

  • Rapid iterations, particularly minor changes in wording or call flow, are immediately testable.
  • Allows the system to be evaluated at an early stage in the design process.

You and your team are able to imitate the real product in these 3 aspects:

  • Feel – What might it look like?
  • Implementation – What might this work like?
  • Role – What might the experience be like?

Basicly, you should maximize learning and minimize time wasted in iterations:

  • Make an interactive application without much code
  • Make the frontend interface easy and understandable
  • Control wisely the user interface interaction without a giant backend
  • Makes sense when it’s faster/cheaper/easier than making real thing. Sometimes you can’t do it, usually you do.

Of course, running an incomplete AB test (in terms of functionalities) with this WofO technique has some disadvantages you must have under control:

  • Simulations may misrepresent otherwise imperfect tech
  • Wizards can be inconsistent
  • Playing the wizard can be exhausting. Usually, you should do this wizard manually, and you can’t afford it for a long period of time.
  • Some features are impossible to simulate effectively

The Wizard of Oz technique is a highly cost-effective way to compare multiple designs, and you’ll find out that parallel design could be better than the serial design. It is, you can create a multi-variant test (not only AB, but ABCDE) and learn of all of them to continue iterating; you can save a lot of time. Prototyping is asking questions and the best way to have a good idea is to have a lots of ideas; as I said, high percent of your tests will fail, but you can extract good information from each one to reach the goal you’re pursuing.

To sum up this serie of ideas, this technique:

  • represents creating a fake product of your final product
  • lets you to quickly iterate and move continuesly towards your goal
  • saves you a lot of money in terms of cost (the time of your team is directly related with money)
  • lets you to learn of your errors without suffering any disaster and improve the product and metrics

…be Wizard, be Fake!