Test drive
By Simon Bisson
Simon Bisson shows you how testing tools can help you get the most from your development teams.
HardCopy Issue: 40 | Found In: Development | Published: 01/05/2008 | Last Revision: 07/07/2010
Software quality is a critical part of the application development process. Applications need to be fit for purpose before they’re deployed to end users. Compressed development times mean there’s little opportunity for a separate test phase in any application development, while n-tier and Web technologies mean that test cases need to handle multiple interacting components and systems. With businesses relying on software for key business processes you don’t have any second chances: your software needs to be right first time.
In the good old days applications could be written, handed over to testers and then deployed. Things are very different now: testing needs to be considered at all stages of the application life cycle, from requirements gathering to final deployment. Everyone’s a tester now, and everything in your application needs to be tested. Modern agile development methodologies are built around the concept of test, and developers need to be ready to design, build and verify their tests before they even start to write any code.
The testing lifecycle
There are several stages to software testing. Unit testing techniques are now a key tool in the developer toolbox which means that pre-release testing is becoming increasingly important. As components are completed and released on to the current code release branch, integration tests are needed to make sure that components work well together, and as more and more components are integrated, these tests inevitably become more and more complex. Integration tests are often closely linked to functional tests, which are used to make sure that application components meet functional requirements. You need to use several levels of integration test, initially testing applications on development hardware, then on staging hardware and finally in its production environment.
Visual Studio Team System
Microsoft’s Visual Studio Team System is a suite of tools designed to support key roles in the application development lifecycle. Building on the Team Foundation Server, the various products in the suite are versions of Visual Studio 2005 that have additional functions to support each role.
The key to VSTS is the Team Foundation Server which mixes version control and change management with a document portal. Intended to support design methodologies like CMMI, the Team Foundation Server uses an SQL Server database to manage code and SQL Server’s Reporting Services to produce reports on project status that can be shared through the portal.
One key feature of the Team Foundation Server is a build server which collates code that has been checked in, makes sure that it is valid by checking test status and then produces a build. This allows teams to have a single snapshot of the application that’s being built.
A new version of VSTS is under development and will include links to Microsoft’s Project Server project management tool, as well as more business-focused tools to help plan and manage development. It will also add tools for test case management and test automation. You’ll be able to set quality standards that must be met before an application is released.
Once all the components are together, you need to run system level integration tests to make sure the whole application functions as expected, and that it handles connections to external systems (which may be run by third parties). Performance tests are closely related to system integration tests as they ensure that an application meets the service levels defined in the specification. Automated stress testing tools come into their own here as they can be used to construct large populations of simulated users that behave in statistically valid ways. It’s important to test applications to breaking point – there’s no point in building an e-commerce application that will function well at July usage levels but fall over as soon as the Christmas rush begins.
Edge cases are important and need to be kept in mind when designing application tests. Out-of-the-ordinary events may be rare but they’re often the point at which an application becomes essential. Tests should also be used to explore failure modes, especially where business continuity measures are built in to the application. If an application is intended to scale then scaling effects should also form part of the testing strategy, and here virtual environments can be used to show the effects of adding additional hardware.
Your users need to be part of the entire spectrum of tests in today’s user-centric world but there’s one area where they’re critical. In most cases acceptance testing should be driven completely by the user as it is intended to make sure that an application can be released to them. Acceptance tests need to be designed with this in mind, and this needs to be a collaborative process that involves the business stakeholders and representatives of the user community.
Once an application is in production, testing doesn’t go away. It should become a key component of any application maintenance strategy, with regression tests used to make sure that any changes don’t affect the live application. While regression tests are best done using an environment as close as possible to the production system, development teams can take advantage of virtualisation tools. These allow developers to use a snapshot of the production environment running in virtualised hardware to ensure that changes have little or no impact when deployed. Regression tests should also be run using a copy of the production data, and if possible final tests should be run in parallel with actual operations to ensure that changes haven’t affected data integrity.
Getting your testing strategy right is a key part of any application architect’s role and it needs to be part of any project governance you put in place. Architect driven tools like Microsoft’s Visual Studio Team System can be used to make sure that developers adhere to a strategy, making sure that code isn’t committed to the source tree without passing the necessary unit tests.
When you look at a modern application lifecycle, and the governance requirements that go with it, it’s clear that test and QA need to be automated to a large degree. There’s a lot that today’s software testing tools can offer, from tools to help convert specifications into unit tests for application components, to complete QA frameworks that tie into project management platforms. Others lock down development environments to ensure that test procedures are carried out as required by project governance.
If you’re using software testing tools, you’re going to need an appropriate test plan. This will define a suite of test cases that are used to handle unit and functional tests. Each test case will be defined as a set of inputs with their expected results, often describing a set of steps that need to be carried out in order to see specific results. Test cases should be easy to define using the application specifications. If they’re not, then it’s time to revisit the specifications and the initial requirements.
Where to test
Testing tools can be used to manage the typical test process, assisting developers and architects in all phases of the test process.
The first phase of any test process is the initial requirements analysis. Developers, analysts and users will need to use this to determine what parts of an application need to be tested and how these tests should be carried out. The result of this should be an overall test strategy. Testing tools can be used to build this plan, often as part of any requirements gathering tool or as part of a use case modelling tool.
Once the initial test strategy is in place, it needs to be refined. This can be done during the test planning phase, which should run in parallel with initial application design and architecture. If application prototypes are being produced, then prototype tests should also be produced. The resulting plan should define the overall set of tests to be produced, as well as outlining appropriate unit tests. Use cases and other software development modelling techniques can feed into test tools at this point. As cases are refined then more accurate test plans can be produced.
As applications move from design into development, test tools need to take the application’s tests along the same path. Unit test tools can be integrated into development IDEs, while test reporting and management features can be tied into project and portfolio management suites. Modelling tools can also be used to keep use and test cases up to date, especially if a joint development process is being used or if you’ve chosen an agile methodology. Unit testing tools need to be able to handle any development refactoring, as variable names and interface definitions are liable to change. As a project proceeds towards deployment then integration and functional test scripts need to be defined.
Integration testing benefits from automated test tools running complex test scripts that can exercise as many of the application functions as possible while stress testing to ensure performance specifications are being met. Errors found at this stage need to be fed back to the development team and the development toolset in order to speed up debugging and defect elimination.
If a defect has been corrected the application will need re-testing. Of course, not all defects will be dealt with in the application development cycle. Testing tools will need to be able to record the defect response agreed by development and user teams, making sure that consensus positions are detailed and that the responses are used to guide further testing, as well as feeding back into the overall application lifecycle where they can be used to plan future application versions.
Making the tools work
Well designed tests can also help project managers see where there may be problems in the development process itself. If a component is regularly failing unit tests it’s possible to use the test results to determine if the component is too complex, or if the developer needs help. A dashboard view of test pass/fail rates will give project leaders an overview of project progress, revealing possible delays before they show up in weekly project management reports. Tying testing tools into project management solutions is becoming increasingly important as it means that team leads and project managers can visualise project progress, as well as making it visible to business stakeholders and development team management.
Tests aren’t perfect, and even the most rigorous unit testing policy won’t catch every possible bug or defect. However, with well written tests and well-designed test methodologies, application developers will find that much less time is spent fixing bugs in deployed applications. That’s time that can be spent working on new versions of existing software and developing completely new applications. Testing tools and QA packages may add to the cost of development environments, but it’s well worth budgeting for as the time saved and the risk reduced can go a long way to switching away from software maintenance to developing new solutions that can add business value.
With test such an important piece of the application development lifecycle, it’s not surprising that there are many different testing tools available. Targeted at everything from the individual developer to the enterprise application development team, they offer a mix of tools for building and running tests, to reporting and publishing test results and analysis.
Choosing the appropriate tool can be hard: with so many different suites available it can be difficult to find one that matches your needs directly. Like the test process itself, it’s important to make any purchasing decision one that affects all the team members involved in the application lifecycle. It’s also worth keeping in mind your development methodology and any governance models you may have in place. Here are a selection of test tools that are worth a second look.
Microsoft VSTE for Software Testers
Microsoft’s Visual Studio Team System is designed to give the various roles in the application lifecycle the tools they need to do their jobs effectively. The suite contains two tools for testers, namely Visual Studio Team Edition (VSTE) for Software Testers and Team Test Load Agent.
Building on the familiar Visual Studio, VSTE for Software Testers is designed to use the application development lifecycle to manage test development. The Test Manager window is where you work with test cases, sorting and categorising so that tests are easy to find and work with.
There are also tools for developing and running performance tests, using existing test cases as the basis for load tests, and tying them into prescriptive server performance models. Builds can be automated, along with associated unit tests, and the resulting reports can then be shared with everyone who is involved with the project.
As today’s applications aren’t just desktop or client-server, there are also tools for recording actions that can be played back to test Web applications. The resulting scripts can be turned into test code that can be edited in the Visual Studio IDE and then used to build unit tests for components of Web applications.
The tool is designed to work with Microsoft Team Foundation Server which means that testers can collaborate with the rest of the design and development team. Testing shouldn’t be considered in isolation and too many tools put it in its own silo. The VSTS Team Explorer and Team Portal let all the members of a development team share and track information about a project, with a single view for everyone involved.
The Team Test Load Agent is an additional tool that is able to increase the load that a single test PC can put on an application. With Team Test Load Agent a single PC can simulate 1,000 users, or multiple PCs can be used to build complex distributed test scenarios.
HP TestDirector for Quality Center
HP’s purchase of Mercury added a wide selection of QA tools to its portfolio. Billing them as Business Technology Optimisation tools, HP is targeting enterprise application development teams with a mix of governance guidance, best practices and tools.
TestDirector for Quality Center is a key component of HP’s BTO strategy. Taking a modular approach, TestDirector uses a Web front end to bring all aspects of the test lifecycle together. Modules include HP Service Test Management module, Release Management, Requirements Management, Test Plan, Test Lab and Defects Management. All the modules can share information and deliver it to the members of a development team.
A repository is used to store test scripts, providing a library of scripts that can be reused and modified as required. Once in the repository, scripts can either be run manually or used as part of an automated test. The Release Management tools are able to manage groups of changes and can be used as part of an overall application lifecycle strategy as well as part of the QA process. You can then use the Service Test Management tools to work with all the components of a service oriented architecture, understanding how they interact to support business processes.
HP has taken a requirements-based approach to testing with TestDirector. This can be a very powerful approach as it uses business requirements to drive the tests. Test cases can be automatically generated by the Requirements Management module.
Reporting tools are key to HP’s QA platform and reports are automatically generated and kept up to date. It’s important to have the right information at the right time and tools like this can make a project manager’s life a lot easier. Reports are available through the Web interface or can be exported into Excel where you can take advantage of Excel’s business intelligence tools to get the most from your test data.
Compuware DevPartner
Compuware's DevPartner Studio includes tools that help you isolate bottlenecks in your code.
Like HP, Compuware is focused on how development teams can deliver the applications that a business needs. Part of this is what it calls Application Delivery Management and its DevPartner tools are intended to help development teams deliver applications quickly and with minimal defects. They also include tools to test security.
DevPartner Studio works with Microsoft’s Visual Studio IDE to extend its native test and analysis features. You can use it to manage tests and to enforce code standards – something that can help reduce defect rates significantly. It can also be used to profile and analyse your code, automating processes that would have otherwise required significant test resources.
You can use DevPartner Studio to automate source code reviews, using either your own or the built-in rules to identify coding errors and to test service calls. Error detection tools will identify poor programming practices, and validate calls to APIs, as well as making sure you handle transitions between managed .NET code and native unmanaged components correctly.
One useful feature is its memory analysis tools which can help track down memory leaks and monitor your applications’ memory usage. As it’s integrated with Visual Studio you can use the heap analysis tools to drill down to the individual lines of code responsible for problems, and then fix them. Additional performance analysis tools can help track down problems with threads and I/O, helping avoid excessive disk and network usage and making it easier to spot errors that arise as your application interacts with others. A server version of the DevPartner tools can also provide performance analysis of n-tier applications.
It’s often difficult to prove that you’ve tested an entire application. DevPartner’s code coverage tools will report on what code has been tested and how well it’s been tested. You can use the results to focus in on untested code and to determine what type of testing is required.
AutomatedQA TestComplete
AutomatedQA produces a selection of tools to help test, debug and manage application development. Its best known product is TestComplete although it also offers tools to help understand application performance bottlenecks and to manage the development lifecycle.
Finding out what's happening in your VBScript code with TestComplete from AutomatedQA.
If you’re looking for a test tool for a wide range of application development environments, from Java to .NET to Web, then TestComplete brings together a wide selection of test types. Unlike many solutions, it’s suitable for smaller organisations as well as large development teams. There’s support for a wide range of tests, from unit to functional, as well as support for regression testing as part of a change management process. As well as traditional applications, TestComplete can be used to test Web applications, offering load, stress and scalability tests.
TestComplete isn’t dependent on development tools, and many tests can be carried out by automating GUI actions. There’s even support for many popular GUI controls, simplifying automation. However, that’s not a suitable approach when unit testing components. Here TestComplete can access internal objects, methods and properties for applications developed in many popular environments, or which use certain common JVMs. There’s also support for applications that take advantage of the Microsoft Office development platform.
You can use the built-in code editor to define test scripts, as well as manage logs and test items. Other features let you work with Web services (important if you’re evaluating third-party services) and manage virtual users for load testing. Scripts can be written in several different languages so developers can get to grips with using it quickly.
TestComplete is extensible and has an SDK that you can use to develop your own extensions. These can be used to wrap groups of tests as objects, allowing you to simplify test scripts that need to work with complex sequences of actions.
IBM Rational Robot
IBM’s purchase of Rational was very similar to HP’s Mercury acquisition. Rational brought IBM a wide range of software modelling and application lifecycle management tools. These included QA and test tools like Rational Robot and Rational Performance Tester.
Rational Robot is a test automation tool designed to help QA teams run functional tests of applications. You can use it to work with many different user interfaces, and it will work with HTML, .NET and Java applications. Tests can be run across several different application configurations simultaneously, making it easier to track and identify problems.
User interface testing can be complex and Rational Robot is able to test all your application components, working with menus and a wide range of different user input technologies. It can also go down deeper than the UI and can test specific components in an application, including ActiveX controls.
Scripts are written in Rational’s own SQABasic language and these can be produced manually or by recording user actions. You can edit scripts in a built-in editor and scripts can be reused across different versions of the operating system.
IBM Rational Performance Tester
Rational Performance Tester is designed to test the performance of complex Web applications. There’s no need to write any code, as wizards can be used to build tests. Tests can also be built using its tree-based text editor. You can gather statistics from any layer in the Web application in order to determine just where bottlenecks are, and to determine where further testing is required.
Load testing is important for e-business solutions and Rational Performance Tester can be used to simulate user loads with a simple agent that can be deployed on a test group of systems. One controller can run the test and aggregate the responses. You don’t need to wait for a report as real-time alerts will show problems. The resulting data can be analysed after a test has run so you can then decide on further areas of investigation.