GigaOM » beta-testing

Nautical Monkey: Like car sharing but for boats

Katie Fehrenbacher — Mon, 18 Jul 2011 07:00:46 +0000

Car-sharing companies tend to highlight that people only use their cars, on average, less than 10 percent of the time. But when it comes to boats, owners use them even less, or an average of two weeks out of the year — that’s according to startup Nautical Monkey, which is aiming to create a business around peer-to-peer boat sharing.

That stat makes boats a perfect capital-intensive good to share between a group of people, and the latest subject of the so-called “collaborative consumption” movement, which uses the web to share physical things between strangers that meet online. This movement, which counts companies like Airbnb, RelayRides, Zipcar, and ThredUP, are tapping into consumers’ desires to use goods as a service, rather than owning a good out right.

Nautical Monkey, which is in beta, provides a web and mobile platform to help groups buy into a boat together, schedule when each person will use the boat, create legal documents for usage and collectively manage maintenance and fuel costs. It might not be the most eco-friendly thing in the world — to boat around and burn fuel — but sharing a good is far more sustainable than buying an expensive good and rarely using it. It’s the equivalent of owning an apartment or cabin that you don’t use and renting it out via Airbnb.

Nautical Monkey is offering free service to its first users up until the end of the year. If anyone wants to co-share a boat, or has a boat that you want to rent out, please test out the service, and let us know how it works. GigaOM proper is currently boatless. Check out this infographic below on using the web to share stuff:

Image courtesy of epSos.de.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

iOS App Beta Testing Takes With TestFlight

Darrell Etherington — Sat, 22 Jan 2011 00:00:26 +0000

Beta testing iPhone apps has been described by some developers I know as the least fun part of the whole process. That’s something TestFlight aims to help improve upon. The service provides free, over-the-air app beta distribution, and it’s now available to all.

I’ve helped a few developers test out beta apps in the past, and the whole experience felt surprisingly low tech. Basically, you send a developer your unique device identifier (UDID), and then they send back an application .ipa file and a provisioning file for you to install on your device once they’ve added your iPhone to their list of testers. Apple requires that only registered devices can run pre-release apps, as a way of preventing people from simply distributing apps directly to customers without going through the App Store.

TestFlight acts as a go-between, facilitating the process for both beta testers and developers alike. It removes the need for developers to send out their files manually to each beta participant, and makes it much easier for beta testers, too. Instead of having to manually install apps using iTunes, TestFlight allows you to install beta apps right from the site’s dashboard in mobile Safari, once you sign up for a free account. You still have to send your UDID to the developer, but once it’s in the system, other devs can contact you and add you to their beta pools as well. To be a tester, you have to have at least iOS 4.0 installed on your device, since that’s when Apple introduced its wireless app installation protocols.

TestFlight is free for developers, too, and the easy-to-use interface ensures that users will face far fewer barriers to entry in terms of testing. The service also allows devs to separate testers into different groups, and assign different versions of each beta to those groups. That means you can show your investors only stable builds, while providing each and every bleeding edge version to your QA team. TestFlight also offers recruitment tools, which allow you to publicize your beta and find testers via tweeted and emailed registration links, and can push out notifications to tester devices when new builds are available.

It may not be the only over-the-air beta testing solution for iOS apps, but it is one of the slickest and most comprehensive, and the fact that it’s free definitely doesn’t hurt.

Related content from GigaOM Pro (sub req’d):

Aroxo: The 4-Stages of Testing Your Web Product

Matt Rogers — Tue, 19 Feb 2008 08:03:29 +0000

Editor’s Note: Matt Rogers is the founder of Aroxo, a novel retail and exchange website based in London. Matt has been chronicling his founder’s experience, and sharing the lessons learned with Found|READ. Earlier posts include How to bootstrap Your Startup and Getting to Launch. His latest installment is about site testing, and how best to “iron out the kinks” prior to launch. Aroxo is almost ready…

Aroxo recently turned a major corner in its development: We moved from closed functional testing to using real live alpha testers — people who’d never seen Aroxo before. Without doubt, this is one of the most revealing, painful, and valuable stages in creating your start-up.

When launching anything you want to ensure that it works, but also that people find it easy and natural to use. Your [internal] functional testing should cover the first objective, your [open] alpha testing should cover the UI.

We employed four different tests to get Aroxo’s system ready for launch, each at a different stage:

1) Over-the-shoulder. To determine the sticking points; where the system confuses users.
2) Task-driven testing. To determine how well the system stands up on its own.
3) Goal-driven testing. Once you know your site is slick and functional, this is for discovering end-to-end flaws.
4) Beta testing. To test site’s marketing points and highlight future development opportunities. It comes at the very end.

Not only is each test different, but you get different learnings from it. I discuss each testing stage below.

Over-the-shoulder tests

This test phase is the most fascinating for the founders, as you get to watch people actually interacting with your system. Do this test when functional testing is 80% done, or earlier if possible. It should be done independently, as those close to the system are inclined to guide the user towards positive responses, rather than getting genuine feedback. But if you are bootstrapping or operating on a tight budget, you may well be doing it yourself.
So, here’s what you do:

* Write out a list of scenarios or tasks which you are going to as people to do
* Find 5-15 people who represent your expected customer base
* Sit them in front of a machine and get going

Your first task should be to get them to look at the home page for a few seconds (no more than 30 seconds), then ask them what they think the site does. People should find it easy to understand the site from the home page. If not, probe a little to find out what you can change to make it clearer. Then get started on the scenarios.

Your job is to generally keep quiet and take notes as they use the system. You are particularly looking for times when something confuses them, you may notice the person lean forward and start to look confused. At these points feel free to interrupt and probe, ask them what they are looking for and what’s missing.

When probing, be very careful to use words like “what” (open) rather than “why” (implies fault). Try not to reveal any emotion, you’re not asking people to tell you how great your system is, you’re after honest feedback. Therefore it is also important not to make people think they are stupid for not being able to use the system, frame it as “we want to get some open and honest feedback”.

Expect some excellent and easy improvements to come out of this to really make the system substantially easier to use.

Task-driven testing

Task-driven is quite different and comes only after major “sticking points” uncovered in the OTS tests have been fixed. For a starters, you won’t actually be watching people, they’ll be running through tasks on their own. This is particularly interesting because for the first time people will be using the system without anyone to ask for help. This means you can get more people involved.

To perform this sort of testing there are a few things you need:
* around 20 participants to start with (expect about half to actually use it and 2-3 who give you really good feedback)
* a well defined list of tasks for them to perform
* a way to communicate the tasks to them and
* a way for them to communicate their findings back to you.

As you push these changes through to the alpha platform you should start adding more and more people onto the trial. You want fresh eyes to test your UI improvements, people who’ve already seen it, won’t test it as well, if at all.

Your objective here is the test how well the system performs with a small number of people performing defined tasks, so each task you ask people to do should come with some prompts to help them give you relevant feedback.

Goal-driven testing

Goal-driven testing differs from task-driven testing in two ways. Firstly, there are a lot more people involved in the trial (I suggest 150-200). Secondly, instead of having tasks which people have to perform on the system (e.g Register on the site). You set wide goals for the participants. In context of Aroxo this is something like “buy something” or “sell something” the UI and the website itself should then guide the users through the system, rather than the tasks.

Start your goal-driven testing when you’ve reached the point that you believe the UI of the system is strong enough that people know what they should be doing and how to use the system.

Beta testing

Beta testing isn’t really testing in the same way as the alpha testing which you’ve already performed is. You want a large number of people to see the system (200-250 minimum) and many of them will have registered on your site to do it, so you don’t know them.

This makes beta testing public. If it is public, it’s part of your marketing. Therefore it is essential that it shows your site running well and easy to use. What you may get is some useful learnings about how the system scales, or you may also be able to learn something from some click stream work.

However, what is most important about beta testing is that it should not be part of your functional and UI testing, it is too late to make any significant changes to the system as you’ll be moving towards a full launch within weeks of starting your beta.

My closing thoughts

One thing which I’ve not discussed in this article is the emotional impact which testing has on the founders. If you’ve been careful with your IP whilst you’ve been building your system it’ll be the first time which you’ve had large numbers of people looking at your system. This means you are going to get a lot of feedback.

It is important not to take this feedback personally. It should help make the system stronger.

You need to remember that you are asking people to look at your system and tell you the flaws and problems with it. You are not asking them to look at it and tell you how clever it is! This means you are going to get a lot of (hopefully) constructive criticism. You need to take this on the nose and think of creative ways to improve the system.

Finally, in the case of Aroxo something we learned : it isn’t possible to test your UI yourself: you are way too close to it. So it is important to have lots of fresh [independent] eyes to test each new release.
Good luck!

For more from Matt, check out his founder’s blog, Digging my own ditch, where you can also read the original text of this post, which appeared tehre on Feb. 14th.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.