John Cow dot Com » testing

Three Common Multivariate Testing Mistakes

Jason Katzenback — Sat, 29 May 2010 15:13:15 +0000

As promised, I’m getting on with the Trusted Expert series and finishing up with the last few posts before everything moves over to the new site. When we left off, we were talking about testing, and the next post is about multivariate testing.

And I’ve been dreading it – not because multivariate testing isn’t worth doing, it so is, but there’s a reason why I’ve been stuck at this particular point for some time. You see, multivariate testing can be a valuable tool that can tell you some very important things about what is working on your site and what isn’t. No, multivariate testing can be an extremely useful thing -when it’s done right.

But if it’s done wrong, it can be a colossal waste of time that can also lose you money. And unfortunately, it’s easy to get multivariate testing wrong. And since I’m writing this post on multivariate testing, if I give you the wrong advice or don’t explain myself clearly, it’ll be my fault. No pressure.

So this post on multivariate testing will strive to be as clear and concise as possible. While firmly stand behind the importance of testing, I in no way claim to be any kind of testing guru or expert – my aim here is to give you a good grasp of the basics so that you’ll walk away with a good understanding of how things work, how it relates back to being a Trusted Expert. And if you want to go ahead and delve further into the more technical aspects of multivariate testing once we’re done, you’ll have a good knowledge base with which to do so.

So without further ado (that had to be the longest intro to a post, ever) let’s get into the nuts and bolts of multivariate testing. If you remember from way back in my post about A/B testing, in that case you can only compare one thing at a time – it’s built right into the parameters of the test. With multivariate testing, there is no such restrictions, so it’s up to you to make sure the things you choose to test don’t actually automatically negate any results you get.

As the name suggests, with multivariate tests, you’re dealing with more than one variable, and you can have any number of variations on each of those variables (more on whether that’s a good idea in a moment). Remember, with an A/B test, you could have any number of variations, but they all had to be about the same element (headers, checkout button, images, etc.) When you’re running a multivariate test, one would assume that the variables you choose to test together have some sort of relation to each other that will result in an overall improvement in performance, but that’s up to you to make the right choices (see why multivariate testing is so easy to screw up?).

Multivariate tests allow you to test the power of different combinations on your site, which can be a very powerful thing. We all know that all the elements on your site work together to influence the user, but with multivariate testing, you can figure out what combination of items will give you the best result.

And with multivariate tests, you get multiple possible combinations. For example, if you’ve got two areas you’re going to test, (say headlines and button color) and you’ve got 3 variations for each – that means that there are 9 possible combinations (3×3=9) that need to be tested. The more areas and variations, and the combinations will grow exponentially.

If you want to get your feet wet in multivariate testing, a good test to start with is a headline/main image combo, since they are viewed together and definitely have in impact on each other. Stick to no more than 3 variations for each – this will result in a test that will not take too long and give you results that are easy to interpret.

Since multivariate testing allows you so much more freedom than A/B testing, it can be easy to run into problems. Here are some common mistakes that people make:

testing too many things at once. When you start out with multivariate testing, it can be tempting to test a whole whack of things at once – Google Optimizer allows you to choose up to 1,000 possible combinations. The problem with this is trying to many things at once can muddy the waters. If your chosen combination has 5 different elements, it could be that, even amongst your highest performing combination, there are still things that aren’t working for your site. But since you had so many elements, it can be hard to pinpoint which one was the problem.
choosing too many variables. Similar to choosing too many elements, choosing too many variables will give you a lot of combinations – and that will make the test take longer. Some tests, because of the sheer number of combinations, can take YEARS to finish. Do you have that kind of time? Didn’t think so. Keep it simple, and see test results you can actually apply to your site in time for it to actually make a difference.
losing sight of your goals. Remember, going into any kind of test, you want to have some kind of hypothesis – the outcome that you expect. You need to keep that in mind when preparing any multivariate test so you don’t get mired down in the sheer amount of details. So for your headline/main image test, your hypothesis could be that an action-oriented headline along with a dynamic picture will be the combination that works best. You then use the multivariate test to either prove of disprove your hypothesis. And, as always, you apply that learning towards your next test and hypothesis.

Hope that helps you understand multivariate testing a bit. As I mentioned at the beginning, it can be a very useful tool to finding out what will perform the best on your site. But because of the sheer amount of freedom multivariate testing gives you, you need to make sure that you practice some kind of restraint to ensure that you’re getting the best possible results.

Split Tests Simplified & Explained – First, A/B Testing

Jason Katzenback — Thu, 12 Nov 2009 18:51:57 +0000

So what test should you start using?

Google’s Website Optimizer gives you two options: and A/B test or a multivariate test. But it’s not as clear-cut as that.

There are actually two ways to run an A/B test using Google’s Website Optimizer. One using the given A/B test option and one using the multivariate test option, and then there’s the actual multivariate test using the multivariate test option.

Confused yet?

I sure was when I started, but here’ a (hopefully) clear, non-jargon using explanation of what does what in regards to A/B testing (we’ll devote another post to true-multivariate testing).

The A/B test that Google gives you allows you either the capability to run a complex page comparison test or to completely waste your time by running an invalid test whose results are useless, depending on what your skill level with testing is going in. You can guess which end of the spectrum I was on when I started testing.

Here’s the reason for that:

Google’s A/B test runs a split-traffic test to multiple pages. ‘Pages’ is the key word there. Since you are testing whole pages, there is the ability to change multiple elements on each page. And as I’ve mentioned before, when you test multiple things willy-nilly, you get skewed results that you can’t trust because you don’t know what one thing caused the change in results. But thanks to me, you now know better, and so you’d better not do that.

**note** The once instance where changing multiple things can work is when you are testing one overlying thing, like a design scheme, which has multiple elements, but they are all tied together in some way and contribute the one thing that you are testing. But this is the only time I would recommend running an A/B test with multiple variations within one page.

You can run what is considered a ‘classic’ A/B test using Google’s version of the A/B test, you just need to practice some restraint, and make sure that you are only testing one element per page. Keep to that rule, and you’ll see some great results that you can actually use. The main point of the A/B test is to compare different versions of one element on a page. You could do an A/B test of a buy now button color and test red (control), blue, green, and yellow.

This is really an A/B/C/D test, but since it is only testing one aspect of one element on the page, it is an A/B test.

The A/B test is really the simplest to set up and run, so it’s a great place to start testing if you’ve never done it before.

Since the multivariate test allow you to isolate specific elements, you can also use it to run an A/B test. Doing this, and isolating only one thing to test, will ensure that you are not accidentally changing anything else, since the only part of the page that will be tested is the element that you identify. Everything else will appear exactly the same as before.

It’s up to you whether you choose to use Google’s A/B test of the multivariate test to run your classic A/B tests.

Listen to this post at Cinchcast.com –> A/B Split Testing

(This post is a contination of the series of Becoming a Trusted Expert. You can read the previous posts at http://www.johncow.com/how-to-become-a-trusted-expert-online/)

7 Steps for Successful Split Testing – Kinda Science-y Stuff

Jason Katzenback — Tue, 10 Nov 2009 16:00:34 +0000

This is a continuation of the Trusted Expert posts on testing…

For testing purposes and the examples we give, we’re going to be using Google’s Website Analyzer which is part of Google Analytics (http://www.google.com/analytics/). Why? Well, for one, because it’s free and therefore accessible to everyone; and secondly, because it’s a darn good testing platform… free or not.

Google’s Website Optimizer is free, powerful, fast and easy. What more could you want? It allows you to perform A/B and multivariate testing.

Why is it free?

Because Google knows that if you improve your conversion rates, you’re more likely to invest in more advertising campaigns, users are happy because they’re happier with the sites that they find through searches. Basically, everyone is happy and Google makes more money.

Makes sense when you put it that way, doesn’t it?

There are other, paid services that you can use to further your testing, measuring, and optimizing practices. But this series is all about making the most out of what you can get for little or no money spent, by concentrating on the buyer and conducting your communications strategy in a totally different way. For that reason, in regards to practical applications for testing that will be covered in this part of the series, we’ll be concentrating on how best to utilize the free Google tools.

The important thing to remember about the Google Website Optimizer is that it is a tool. It only provides the metrics, it can’t tell you what changes to make. You have to infer that for yourself. If you want the tests you run to be meaningful and give you the feedback you need to improve your site, then you need to make sure you’re going about it the right way, and making the most of the free tools that Google gives you.

Later in the series, we’ll be going through exactly what you should be doing to get you started with testing. Where you take it from there is up to you.

Before we delve into the nuts and bolts of actually running a test, there are a few things you need to remember for any test you run, whether it’s #1 or #1001:

Always start the test with a goal in mind. Know what you expect the outcome to be (although you may be wrong – very, very wrong; that’s what you’re testing.) In scientific tests, this is called the hypothesis. Your goal is the basis and the reason for the entire test.
Determine what you metric of success will be before you start the test. How much better is good enough to be considered a success? 5%? 10% 50%? Only you can decide that.
Remember not to muddy the waters. If you’re testing the color of the checkout button, don’t also change the shape or the font or where it appears on the site. You could get an amazing result, but you won’t know what variable was responsible for the change. Be especially aware of this when testing copy; any other changes you make, even inadvertently, to the layout or the font or any other aspect of the display will totally negate the validity of the test on the actual content.
Remember that you always need a control. Even when you’re dong multivariate testing, you still need to use the original version of the page to act as a control, something to compare the test results to.
Use descriptive names for your tests. You may be able to keep track of the fact that Tests 1-10 were about buttons and Tests 11-15 were about headlines, but what about when you’re at Test 345? If you give your tests clear, descriptive names, it’ll be easier for you to find the information and results when you need to refer back to them to plan more tests.
The tests mean nothing if you don’t learn anything from them. Your test isn’t really done (or of any use to you) unless you’ve analyzed the results and applied the learnings to your site. You then use that knowledge you’ve gained as the basis for the next test, because…
And finally, …You’re never done testing. There will never come a time when you can say “That’s it, it’s perfect, I’m done.” Even if your site is perfect and performing at it’s utmost capacity at that moment, the internet is a fluid, changing thing, and you need to make sure (through testing) that you’re keeping up with the changes and keeping you and your site relevant and maintain your presence on the web as a Trusted Expert.

(This post is a contination of the series of Becoming a Trusted Expert. You can read the previous posts at http://www.johncow.com/how-to-become-a-trusted-expert-online/)

Maintain Your Trusted Expert Status – The Importance of Testing and Where to Start

Jason Katzenback — Mon, 09 Nov 2009 13:13:00 +0000

Every single page of you website has elements in it which affect how people view your website. And each of those elements came to be there because of a decision you made.

But was it the right decision?

You’ll never know how much better you could be doing unless you test.

While it may seem like a daunting prospect with all of the potential testing situations available to you, you need to start somewhere. Any testing is better than no testing at all. But if you go off testing willy-nilly, you may be getting skewed results because of other factors that are influencing the results.

Here’s what you should be looking at when you’re starting to test, in this order:

Function – I know, duh, but you’d be surprised at how many people ignore basic functionality for testing brighter, shinier, more interesting things, and then don’t understand why they don’t see results. If half of your pages lead to nowhere, and you’ve got a ton of broken links, and your shopping carts keep emptying, or there is an error in your opt-in form, then how can you expect to see good results when you’re testing the picture on the front page?

Make sure everything works on your site first. The worst impression you can make is that your site is not properly run; it will undermine the trust level with your users so profoundly that the best content in the world will not save you.

Accessibility – The basics of accessibility is making sure that all kinds of browsers can load your website, and that it looks like it should on all of them. You can test this out at a site like http://browsershots.org/ which will test how your site looks in all the browsers you can think of, including Explorer, Firefox, and Safari.

Other aspects of accessibility that you may not have considered, but are equally important are thinks like:

font size
language issues
script (java, php, asp, flash) errors
problems with things loading properly.

This is when it is very useful to have developed your buyer personas before addressing these concerns. For example, if your personas consist mainly of an older demographic, then size on font and clarity may be an important accessibility issue for you to consider.

Usability – Now that your site is functional and accessible, you have to start thinking about how people use your site. This is another opportunity for you to customize your site for your buyer personas.

What kind of font or layout to they prefer?
How do they move through the buying process?
What kind of navigation are they looking for?

It could even be something as simple as whether they prefer a blue button or a red one. A lot of the A/B tests you’ll do will have to do with usability (we’ll go through that whole process a little later – don’t worry.) There are a lot of things that you can find out about your buyers and your website through usability testing, and the improvements you can see from a few small changes is huge.

Intuition – The intuitive feel of your site is a little more subjective and can be harder to pin down. It as a lot in common with usability, but the intuitive level of your site is more about the entire user experience than about any one specific element. Through testing of intuitive elements, you can find the things that are preventing your users from completing the desired action you want from them.

These elements can include specific things like:

point-of-action assurances
customer reviews
color scheme
graphic design of the site itself.

Intuitive elements are the things that evoke a certain emotional response from our buyers. By testing these elements you can make sure that the emotional elements of your site are hitting the right notes with your buyer personas and further establishing yourself as a Trusted Expert in their eyes.

Persuasion – The persuasion elements of your site can be some of the trickiest to identify and tie down. These are the elements that actually persuade your buyers to perform the desired action. You only deal with persuasion issues after you’ve resolved all the other problems with your site.

Where to Start?

It’s important to focus your efforts on the things that will effect the most positive change with the least possible expenditure of time, effort, and money. Once you’ve addressed the basic concerns (functionality, accessibility, usability) then you can move on to the more emotional aspects (intuitive and persuasive elements). That still leaves a whole lot of testing to do.

Here’s how you can narrow those choices down:

1) Take a look at your site’s metrics.

This will tell you where you need to start testing. **Note – if you don’t have a site yet, make sure to address all of the concerns in order and build your site to meet your buyer personas needs and you’ll save yourself a lot of testing and you can go right to testing the emotional elements of your site to get a better response rate. Everybody else, keep reading.

2) Identify the following:

Pages with the highest bounce rate
Pages with the highest exit rates
Pages with the lowest time spent
The 5 most important pages on your site

3) Rank & Prioritize

Once you’ve identified these pages, you need to rank them in order of importance to you and (which really means in order of importance to your buyer personas, since it’s all about them, remember?) and where they fit along the functionality to persuasive scale, starting at the bottom.

You now have your list of things to test, and in what order to test them.

Also, in these areas, you can identify where your site has been lacking in thought leadership. Take this opportunity to change these areas so that it reinforces the idea of You the “Trusted Expert” in your niche.

Now you have a game plan to start testing with. Too bad you don’t know anything about actually running tests…

You didn’t think I’d leave you hanging, did you? That’s the next part!

(This post is a contination of the series of Becoming a Trusted Expert. You can read the previous posts at http://www.johncow.com/how-to-become-a-trusted-expert-online/)

Even More About Testing (Because the Last Post was Getting Long)

Jason Katzenback — Fri, 17 Jul 2009 02:30:06 +0000

Continued from the previous post Continually Improving Your Communications

At the highest level of testing, you will need to test all the persuasive elements in your site so you can create the best system for closing the sale possible. And don’t underestimate the importance of any element on your site; something that seems unimportant to you could make all the difference to your users. Every element on your website is meant to persuade users to take a certain action; therefore, every element needs to be tested in some way. Luckily, on the internet, testing is easy and inexpensive (well, it can get expensive if you want it to, but it can just as easily be done cheaply.)

Don’t rely on what you would want, or what you think your users want; not everyone wants the same thing you want, or wants to be treated the way you want to be treated. That’s why testing is so important – so you give your users what they’re looking for, not what you think they’re looking for, or what you want to give them. You can apply all the research and knowledge that you’ve gained in your niche that you want, try to adhere all the marketing rules that you can find, but you will never know exactly what your users really want until you start testing.

Essentially, through testing, you will be letting your buyer personas design your site for you. You start out with a plan and a best guess, but be prepared to make changes and let go of your favorite design elements if they prove to not be what your users want. Ignore the ‘rules’ of online marketing. If the internet has taught us one thing, it’s that marketing rules were made to be broken. If the internet or online marketing never changed, then rules would be ok. But it is a fluid, constantly changing thing, and while there are some guidelines you could keep in mind, don’t be married to any one strategy if is stops producing results for you.

Through testing, you are empowering your users to decide what works best, and since they’re the ones paying the bills, it’s best to listen to them. Using Google’s Website Optimizer, you don’t have to guess what they want, the report results will quickly tell you what they like, what they don’t like, and most importantly, what will make them buy and what makes them not.

Remember – it’s about what your customers want, not what you like!

Continually Improving Your Communications

Jason Katzenback — Tue, 14 Jul 2009 12:10:14 +0000

In the previous post “Communication In Your Web Design” I talked about how your web design can actually communicate to your visitors as to whether you are a “Trusted Expert” or not. In this post, I talk about the final phase of your communications strategy and becoming (and staying) a Trusted Expert which is…

TESTING

Why do you run tests?

So you can be as certain as possible that your communications strategy is helping rather than hurting your online business.

You’ll never know if you have the right site for your customers unless you test it.

You need to examine all the elements of the site – every single thing on your site is part of what will persuade user to purchase. You can’t take your assumptions and intuitions for granted, or assume that traditional marketing practices will apply. You have to examine everything you do with a critical eye. To be a Trusted Expert, you must have the best and most functional site you possibly can.

One of the best ways to realize the full potential of your website is by testing, and to keep right on testing. Your website is like your spokesperson, and it works 24/7 – out there, interacting with potential customers, all the time trying to persuade them to take action. You need to know if the assumptions under which you built your site are, in fact, true and meet the needs and expectations of your audience.

There are three major components to your ‘testing’ process that you need to take into consideration: testing, measuring, and optimizing.

Testing is when you compare elements on your sites or within your advertising campaigns to see which variation or combination is the best for persuading users to complete the desired action, whether that be a sale or a sign-up.

Measuring is when you monitor what the visitors to your site do. Using an analytics program such as Google Analytics (which is free, so why wouldn’t you use it?) you determine key performance indicators – traffic, overall conversation rate, page abandonment rate, length of time spent on a page, etc.

Optimization is when you put your test results in action to maximize your conversion rate. You take your insights and learnings from the tests you ran and put the results into action on your site.

As you test, you’ll learn more about the assumptions you made when you crafted your communications strategy. You’ll reinforce some assumptions, and have to refine and change others. While no one wants to be proven wrong, the great part about this process is that you are learning things about your buyer personas, and learning what they like, in real time.

You can use this information to continually improve your communications strategy. When you know what your buyers want, you’ll find it easier to be a Trusted Expert to them. As you grow more familiar with your users and what they want, you can create more meaningful options to test to continue to further improve your site, improving your conversion rate the entire time.

So You’re Finally Testing – But Are You Testing Smart?

Jason Katzenback — Wed, 15 Apr 2009 12:20:51 +0000

Good for you. You’ve finally decided to start testing elements of your website using Google’s Website Optimizer. It’s a great idea – everyone’s site could always do better, and it’s hard to resist using a tool that’s free.

But are you using it in a way that will most benefit you? Or are you testing things, willy-nilly, and making changes at the drop of a hat? And you’re not seeing any real improvement in your metrics? That could be because you’re going about this in the wrong way.

To properly test, assess, and optimize your website, you’ve got to go back to Grade 10 science class. That’s right, I’m busting out the scientific method on y’all. By using the simple experimental guidelines that we all learned in high school, you can ensure that the test that you run will provide you with real information that you can then use to improve your site’s conversion rate.

It seems a strange thing to do, but, really, it makes sense. After all, what is a test but a form of experiment? And if you don’t perform an experiment properly, you get skewed results. So, by following the scientific method, you’ll ensure that you can trust the learnings you get from running the test.

Every good experiment must start with a hypothesis. The hypothesis is the answer that you expect to see at the end of the experiment. It could be something as simple as “A blue “buy now” button would have a better conversion rate than a red one.” You can’t just start out testing with thinking “I want to improve my conversion rate.” Well, yeah, that’s what everyone wants. You need to be more specific than that.

For optimal results, you’ll want to come up with a hypothesis that is a true/false statement. As you can see from the way I worded mine, either the blue button is better than the red one, or it isn’t. Depending on whether my hypothesis is proved true or false, I will know whether or not to have a red or blue “buy now” on my sales page.

By setting up a hypothesis at the beginning of the test, you ensure that you stay on track and have a very clear idea about what you want to accomplish with the test. It also gives you the method for evaluating your results. Then, you can take your learnings, and apply them to your next test. You know that blue is better than red, but would green work even better than blue? And you’ve got your next test already.

This is a grossly oversimplified method for testing, but it’s somewhere to start. The example I gave is one of the easiest A/B tests you can do, but you can take the same methods and apply them to more complicated A/B tests as well as multivariate tests.

Just start doing it! You don’t know how much better your stats could be until you start testing them.

The important thing is that you start testing using Google Website Optimizer, and you test in a way that makes the results valuable information. And that means you have to actually follow scientific guidelines. There’s much more that can be said about testing, enough to fill several books, but this is a place for you to start.