Blog Post

Aroxo: The 4-Stages of Testing Your Web Product

Editor’s Note: Matt Rogers is the founder of Aroxo, a novel retail and exchange website based in London. Matt has been chronicling his founder’s experience, and sharing the lessons learned with Found|READ. Earlier posts include How to bootstrap Your Startup and Getting to Launch. His latest installment is about site testing, and how best to “iron out the kinks” prior to launch. Aroxo is almost ready… mattrogers.jpeg

Aroxo recently turned a major corner in its development: We moved from closed functional testing to using real live alpha testers — people who’d never seen Aroxo before. Without doubt, this is one of the most revealing, painful, and valuable stages in creating your start-up.

When launching anything you want to ensure that it works, but also that people find it easy and natural to use. Your [internal] functional testing should cover the first objective, your [open] alpha testing should cover the UI.

We employed four different tests to get Aroxo’s system ready for launch, each at a different stage:

1) Over-the-shoulder. To determine the sticking points; where the system confuses users.
2) Task-driven testing. To determine how well the system stands up on its own.
3) Goal-driven testing.
Once you know your site is slick and functional, this is for discovering end-to-end flaws.
4) Beta testing.
To test site’s marketing points and highlight future development opportunities. It comes at the very end.

Not only is each test different, but you get different learnings from it. I discuss each testing stage below.

Over-the-shoulder tests

This test phase is the most fascinating for the founders, as you get to watch people actually interacting with your system. Do this test when functional testing is 80% done, or earlier if possible. It should be done independently, as those close to the system are inclined to guide the user towards positive responses, rather than getting genuine feedback. But if you are bootstrapping or operating on a tight budget, you may well be doing it yourself.
So, here’s what you do:

* Write out a list of scenarios or tasks which you are going to as people to do
* Find 5-15 people who represent your expected customer base
* Sit them in front of a machine and get going

Your first task should be to get them to look at the home page for a few seconds (no more than 30 seconds), then ask them what they think the site does. People should find it easy to understand the site from the home page. If not, probe a little to find out what you can change to make it clearer. Then get started on the scenarios.

Your job is to generally keep quiet and take notes as they use the system. You are particularly looking for times when something confuses them, you may notice the person lean forward and start to look confused. At these points feel free to interrupt and probe, ask them what they are looking for and what’s missing.

When probing, be very careful to use words like “what” (open) rather than “why” (implies fault). Try not to reveal any emotion, you’re not asking people to tell you how great your system is, you’re after honest feedback. Therefore it is also important not to make people think they are stupid for not being able to use the system, frame it as “we want to get some open and honest feedback”.

Expect some excellent and easy improvements to come out of this to really make the system substantially easier to use.

Task-driven testing

Task-driven is quite different and comes only after major “sticking points” uncovered in the OTS tests have been fixed. For a starters, you won’t actually be watching people, they’ll be running through tasks on their own. This is particularly interesting because for the first time people will be using the system without anyone to ask for help. This means you can get more people involved.

To perform this sort of testing there are a few things you need:
* around 20 participants to start with (expect about half to actually use it and 2-3 who give you really good feedback)
* a well defined list of tasks for them to perform
* a way to communicate the tasks to them and
* a way for them to communicate their findings back to you.

As you push these changes through to the alpha platform you should start adding more and more people onto the trial. You want fresh eyes to test your UI improvements, people who’ve already seen it, won’t test it as well, if at all.

Your objective here is the test how well the system performs with a small number of people performing defined tasks, so each task you ask people to do should come with some prompts to help them give you relevant feedback.

Goal-driven testing

Goal-driven testing differs from task-driven testing in two ways. Firstly, there are a lot more people involved in the trial (I suggest 150-200). Secondly, instead of having tasks which people have to perform on the system (e.g Register on the site). You set wide goals for the participants. In context of Aroxo this is something like “buy something” or “sell something” the UI and the website itself should then guide the users through the system, rather than the tasks.

Start your goal-driven testing when you’ve reached the point that you believe the UI of the system is strong enough that people know what they should be doing and how to use the system.

Beta testing

Beta testing isn’t really testing in the same way as the alpha testing which you’ve already performed is. You want a large number of people to see the system (200-250 minimum) and many of them will have registered on your site to do it, so you don’t know them.

This makes beta testing public. If it is public, it’s part of your marketing. Therefore it is essential that it shows your site running well and easy to use. What you may get is some useful learnings about how the system scales, or you may also be able to learn something from some click stream work.

However, what is most important about beta testing is that it should not be part of your functional and UI testing, it is too late to make any significant changes to the system as you’ll be moving towards a full launch within weeks of starting your beta.

My closing thoughts

One thing which I’ve not discussed in this article is the emotional impact which testing has on the founders. If you’ve been careful with your IP whilst you’ve been building your system it’ll be the first time which you’ve had large numbers of people looking at your system. This means you are going to get a lot of feedback.

It is important not to take this feedback personally. It should help make the system stronger.

You need to remember that you are asking people to look at your system and tell you the flaws and problems with it. You are not asking them to look at it and tell you how clever it is! This means you are going to get a lot of (hopefully) constructive criticism. You need to take this on the nose and think of creative ways to improve the system.

Finally, in the case of Aroxo something we learned : it isn’t possible to test your UI yourself: you are way too close to it. So it is important to have lots of fresh [independent] eyes to test each new release.
Good luck!

For more from Matt, check out his founder’s blog, Digging my own ditch, where you can also read the original text of this post, which appeared tehre on Feb. 14th.

12 Responses to “Aroxo: The 4-Stages of Testing Your Web Product”

  1. Very important article and I will advise everybody to read it.

    You have pointed out something that people (including me) keep forgetting. Don’t take it personal. As the owner / designer of the application, every critic that is not “wow, how did you do that” (or somewhat on that level of enthusiasm) may be a total turn off. I can say from experience that I have stopped projects just because one or two people told me it’s totally useless. And guess what, other people implemented the same ideas months later and are now rich. So you should learn to trust and love your own product before showing it to anybody else. Don’t let anybody convince you otherwise.

    Great article, kudos!
    — gil

  2. Hey Matthew. You should have dedicated testers which are included in your development, these guys should do unit (testing individual components written by the dev team) and integration testing (testing the whole system from end to end). This is a specialist activity and it is important to get people who know what they are doing.

    Then for this UI (User Interface) testing I’d definitely get in a UI specialist to help you through this – if you have the budget. You may want to involve them during the design stage too – the earlier the better. These days they tend to go under the name Interaction Architect/Designer.

    For your other questions I deal with this directly in my other posts which Carleen was linked to above, there’s also some stuff on my blog.

  3. Okay, I am digging this.
    Matt, what personnel make up the development team? Not looking for names, just positions and how they fit into the mix from concept to launch.
    Also, what is a good method for getting the concept down on paper so that others can understand your vision? Is there an application for organizing that detail or a set of tried and true rules?