Usability Testing
By Simon Bisson
Usability is about more than delivering a good user interface, but how can you test something that’s so subjective? Simon Bisson finds out.
HardCopy Issue: 52 | Found In: Development | Published: 01/05/2011 | Last Revision: 28/06/2011
All applications have users, and as much as engineers make jokes about “problem exists between keyboard and chair”, they’re an important part of the development process. Applications need to be designed to be easy to use, and usability needs to be considered at all times, right from the very start of the development process and beyond, even as an application transitions from development to operations.
It can be hard for developers to appreciate the need to design for usability. There’s little contact with end users, and often little guidance on what needs to be done to ensure a good user experience. Often conflated with accessibility, usability is as much a mix of psychology and design as it is a development methodology. Usability isn’t about dropping controls on a page or a form; it’s about understanding how an application fits in with a user’s workflow so it doesn’t jar or distract, but instead makes sure that users remain productive.
Perhaps the biggest problem facing a development team wanting to implement usability tests is that usability is a non-functional requirement. You can’t define usability in terms of application inputs and outputs, and it’s not something that can be handled by automated tests (though you can use automated tools to test for the closely-related accessibility). While it is possible to test for usability as part of the development workflow, by making users part of your development team and having them interact with prototypes and with non-functional builds, it’s something that’s more likely to be part of your overall acceptance tests and part of any operationalisation process – and possibly even as part of a marketing exercise for external facing applications or Web sites.
Defining usability
A good place to start is the ISO definition of usability: “The extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency, and satisfaction in a specified context of use.” It’s key to understanding how to build usability into an application, and how to test it.
It’s also worth considering usability guru Jakob Nielsen’s and Professor Ben Shneiderman’s System Acceptance Framework which defines usability as a combination of the following:
- Learnability: How easy is it for users to accomplish basic tasks the first time they encounter the design?
- Efficiency: Once users have learned the design, how quickly can they perform tasks?
- Memorability: When users return to the design after a period of not using it, how easily can they re-establish proficiency?
- Errors: How many errors do users make, how severe are these errors, and how easily can they recover from the errors?
- Satisfaction: How pleasant is it to actually use the design?
There are plenty of tools to help engineering teams deliver a usable product. However, it’s a good idea for all involved to have a grounding in the principles of user-centric design, starting with Edward Tufte’s series of books on design, and Scott McCloud’s series on the semiotics of graphic storytelling initiated by Understanding Comics: The Invisible Art.
Developers should spend time with user interaction designers, understanding the constraints they work in, and the needs and cultural demands of end users. In practice this can be hard to organise, so another approach is to embed the designers in with the development team. Microsoft’s series of MIX conferences and its Expression design tools, along with Adobe’s Flash Builder and its Flex framework, are intended to facilitate this, providing ways to link traditional design tools with the engineering process and start a dialogue between the two very different disciplines.
The development lifecycle
You need to consider usability right at the start of the development lifecycle, as part of your initial requirements capture. It doesn’t matter what methodology you use: you need to understand user requirements whether you’re working with an agile process or a traditional waterfall development. CASE tools can be used to capture rough user interface and interaction prototypes as UML use cases, or as extreme programming user stories. Defining how users interact with a system at an early stage is important, and should specify just what they expect the system to do. Once they understand those user interactions and their expected results, developers and designers can work together to produce a usable user interface.
It’s also worth looking at the applications the users are already using. If they spend all day in front of Microsoft Office, for example, then it’s worth considering the components available from companies such as Infragistics and ComponentOne which offer a similar user interface. However this doesn’t obviate the need to test for usability.
Testing usability isn’t as easy as ensuring requirements are met, or managing unit tests, or even handling final acceptance tests. They’re something that need to be considered all through a development process, and may often depend on anecdotal responses. While some usability issues, like system response times and data formats, are quantifiable, much of a user interaction is subjective and depends on subtle pointers and cues, such as colour and button placement. A usable interface is one that embodies the flow of an application, guiding a user from one interaction to the next without distracting them – something that affects user productivity.
Testing Usability
You will probably have to design your own usability tests, and it’s a good idea to use the System Acceptance Framework as a basis for your tests. While it’s possible to carry out the tests at user desks, using surveys and other tools, it’s better to create a simple usability lab as the location for your tests. You don’t need to build a dedicated facility for the tests – an office with a couple of video cameras and some screen recording software can work very well. Each test needs to be built around a scenario, often derived directly from use cases or user stories. Scripts can be used to prompt users, and testers can be instructed to prompt users to carry out specific tasks, while monitoring their actions.
The books on information presentation and design by Edward Tufte are worth reading.
There’s a distinct difference between usability tests and marketing studies. While marketing studies can help drive product development by showing how users respond to specific features, usability testing is intended to give development and design teams a deeper understanding of just how users work with an application, how they learn to use it, and how they respond to it emotionally.
The System Acceptance Framework is a good place to start designing usability tests, looking at each of the five different usability elements. While you can test for them individually, in practice you can combine several elements into a single test – perhaps bringing together learnability and user satisfaction, or memorability and error rates.
You can test for learnability by recording the actions of new users. Screen capture tools help time interactions, while cameras can be used to monitor the user’s response to the software. It’s important to understand emotional responses, as frustration and delight are hard to quantify from traditional scripted tests or from survey responses – and are as important in usability as the actual screen layout. It’s a good idea to test learnability for both users who’ve never seen the application before, and for those who’ve been through a training course – letting you see how intuitive the interaction design is, and also the quality of any training material that has been developed for the application.
Similarly, efficiency can be tested by regularly monitoring user throughput, either on prototypes or on final code. Screen movies reveal the steps that users take, as well as differences between individual interactions. You can also use proven design testing techniques such as A/B testing, delivering slightly different versions of an application user interface to determine which has the best usability. You need to understand the aims of the project in detail to produce effective efficiency tests, and it’s best to work in conjunction with business owners and the design team here.
Memorability is harder to test as it requires multiple test interactions with specific users. You’ll need to record how the user works with the application on a daily, weekly or even monthly basis. Scripted interaction requests give a baseline, while video recordings show user responses to the interactions. Screen recordings can be used to give timings, and data analysis will show error rates.
You can also use interaction analysis tools to understand how users interact with on-screen targets – are they regularly missing key buttons or menu items? It’s a good idea to run these tests throughout the development process, starting with wireframes and working through prototypes and early builds.
Long term issues
You need to keep monitoring usability even after an application is in production. User needs change over time (as does the user population), and what worked when you launched an application may not work as business requirements change. User familiarity with applications could be linked to training, and once users start to informally train other users new usability problems may occur. It’s a good idea to work with the operations team and the business application owners to get regular reports of error rates. Users with excess error rates can be tracked to understand the root cause – and to determine if changes to interaction designs need to be made. It’s also important to understand how easy it is for users to make corrections, either editing inputs or re-keying data.
User satisfaction is probably the hardest thing to test. It’s subjective and extremely variable. Video of users can be analysed to determine emotional responses to a design, letting you understand if users are showing signs of delight or of frustration. In practice, neutral or slightly positive responses will be classed as a pass, as they indicate that users aren’t dissatisfied or actively disengaged from the application. On-going user research can be used to track satisfaction, with pop-up surveys or sentiment mining tools giving regular snapshots of the user experience.
The final part of a suite of usability tests is the most complex to analyse – but oddly one of the simplest to implement. Comparing throughput metrics between old and new applications tells you if the new version improved user productivity. If productivity is lower, then there are likely to be usability issues – especially if you’re tracking increased error rates. If this happens you need to work closely with user stakeholders and the operations team to understand just what might be causing problems, and what changes can be made.
One approach to dealing with usability issues is to work with off-the-shelf user interaction frameworks such as Infragistics Quince Pro. Frameworks like these often embody best practices, and can reduce the need for iterative user testing. The ability to share best practices with peers can make it easier to design for usability, by highlighting design patterns that work and providing a framework for improving marginal designs. Good usability testing has other effects: if you’re building a Web application, for example, you can streamline the testing process by mixing usability testing with market research, employing the same tools to determine what works best for your users, and what brings in the most users from your target markets.
Tools for testing usability
TechSmith is probably best known for its SnagIt and Camtasia screen shot and screen movie tools, but it also has one of the better known usability testing tools on the market in the shape of Morae. You get an in-depth set of analytic tools in Morae, which can be thought of as a usability lab in a box.
The Recorder module allows you to design your study, starting from its built-in templates. You can then add user tasks and discussion points, as well as complete surveys. You can use a mix of video cameras and Webcams to record your test users, in combination with Morae’s screen recording software. Tools like this can test Windows applications as well as Web applications, and work well alongside automated user interface test tools like Infragistics TestAdvantage. Once an application has passed an automated test, it can be passed over to the usability test team for more exhaustive user-focused testing.
Support for external cameras in Morae means you’re not limited to recording screen interactions. You can record user interactions in real time (time coding them with screen actions), so you can capture larger scale emotional responses as well as basic eye-tracking. Surveys can be used to capture additional information at any point in a test, and can be launched by the test script. There’s also an option to fully automate the testing process, letting you run tests remotely or even on users trying out a demonstration.
Remote observers can log into any test session while it’s being recorded, and can add notes and comments to the session, as well as calling out user actions for further investigation. There’s no need to have a large test team in a room, as it can change the user dynamic. Results can be graphed automatically, with support for common usability metrics such as efficiency and satisfaction. You can also drill down to markers set by observers, so you can see just what usability issues they were calling out. Multiple recordings can be compared, showing how different users responded to the same test scenarios. Final results can be collated and turned into an interactive report, with embedded video and test screenshots.
A COM server built into Morae lets you add usability testing hooks into your applications. You can use these to record your own data, perhaps sampling error rates or monitoring user throughput. You can also use the server to trap error messages from an application, or to show when users open help functions or find specific parts of an application.
Morae Recorder allows you to capture a great deal of information about the way in which the user interacts with your application.
Once you have captured the data, Morae Manager lets you view and analyse your recordings, which can include focus group meetings as well as Recorder sessions.
One option that simplifies early stage design and usability testing is ClickHeat, an open source tool that works by adding a small amount of mouse tracking JavaScript to the Web pages you’re testing. A server-side PHP application collects this data, displaying it as a heatmap of clicks which can be overlaid on the original page. You’re able to track just where your users click, seeing if they’re missing important user interface elements, and if your page’s cues guide them to the right places.
It’s a good idea to use free tools like this when you’re prototyping sites, or thinking about applying a new look and feel to an existing application.
The commercial ClickTale provides more information. Its Mouse Tracking Suite shows you just how your users interact with a site, revealing how they move through pages, where they click and when they scroll – as well as what they type. Mouse move tracking like this gives you some of the benefits of eye-tracking, as it lets you see just where your users are giving their attention. Tools like these are valuable, although they only give you part of the story as without understanding the user experience, user actions are unable to show you why users do what they do.
If you’re planning to run a set of usability tests on real world users, you can take advantage of the tools built into the Web-based service Loop11. You can use this to build a set of scripted tests that set goals for users. Loop11 then recruits testers and monitors their progress – showing you where they succeed, where they fail, and how they move through your (or a competitors) site.
Most usability testing tools are focused on the Web, with very few able to work with Windows applications. Mixing and matching tools with manual processes gives you the best possible coverage – but you do need to be a strong advocate for the process. Usability testing is important, but it needs a champion before it becomes an accepted part of a development lifecycle.