Functional testing
By Simon Bisson
You’ve built your application, but is it ready for use? Simon Bisson explores tools and techniques for functional testing.
HardCopy Issue: 47 | Found In: Development | Published: 01/02/2010 | Last Revision: 07/07/2010
Good code is tested in many ways. All through the development cycle, unit tests and integration tests make sure your code is good and bug free. But there’s one final hurdle to overcome: namely final release. That’s when your code is ready to be handed to the operations team and installed on the desktops of your users, on your servers or out on the Web. So how do you know that your code is finally ready? Unit tests don’t help as they only tell you that your classes and objects behave as designed, and the tests will have evolved during the development process.
This is where functional tests come in. Based on Use Cases, if you’re using a formal design process, or on User Stores, if you’re using an Extreme Programming (XP) methodology, functional tests make sure that your code meets the requirements agreed with the project stakeholders. As Jeff Canna of RoleModelSoft wrote in a recent IBM blog post: “Unit tests tell a developer that the code is doing things right; functional tests tell a developer that the code is doing the right things.” That’s an important distinction: your application may be full of code that works perfectly, but what it’s doing could be perfectly wrong – at least in terms of the system specification.
So how do you go about designing, building and running your functional tests? The best place to start is your initial design documents and the resulting functional requirements. These are a formal description of the functions that stakeholders and users require from a system. You’ll need to use these to plan and define the tests you’ll be using. It’s important to have a complete requirements definition at the start of a project, something that often isn’t the case. If one hasn’t been produced then you need to go back through your design documentation and agree a set of requirements with your test team.
Visual Studio 2010 comes with a new standalone test management tool for working with functional tests in Microsoft Test and Lab Manager.
The requirements definition can be used to partition your system into its main functional components. Once you’ve defined your application’s components you can verify their descriptions against your design documents, and can then start to define your tests, making sure that they model all your Use Cases. Again good documentation is key to good test design as the tests and test cases need to be mapped to the requirements specifications in order to ensure that all application functions are being tested fully.
With functional tests designed and added to a test plan, you can now start to test your application. Tests can be manual or automated, and familiar unit test tools can be repurposed to handle functional tests. Other test tools can be used to handle aspects of functional testing that may not be part of your unit test process, including load and performance tests. Code coverage tools can also be used to ensure that all aspects of an application are exercised, as well as providing reports to project managers and stake holders. Any defects found during testing can be handled by existing change management procedures, and can be handed back to developers. Once remedied, the appropriate tests can be carried out again.
Functional tests don’t have to be automated, and in many cases may be run manually. That’s often the case with user interface tests, where Use Cases need to be worked through in order to ensure that information is requested and presented in the correct formats. Testers can compare the screens that are presented by the application with their descriptions in the functional specification, without needing to know just how the application actually operates. Most functional tests can be carried out using this black box process, with the tester playing the role of the user. Automated tests can be carried out in a similar manner, with the results of scripted operations being compared with the original design model.
Testing in Visual Studio 2010
The upcoming Visual Studio 2010 includes a set of tools for building and running user interface functional tests. A UI test recorder captures test actions and generates the code needed to test an application. It’ll work with custom controls as well as the default Windows controls, using a control locator to manage control properties and generate appropriate test code for your application’s controls.
Microsoft describes it as “intent aware”, and code produced by the recorder can bundle up a set of actions into one easy-to-understand operation. It’s not just for Web applications, either, with support for both Windows Forms and WPF. Visual Studio 2010’s Coded UI Test tools can work with all classes of Windows applications as they can take advantage of the automation and accessibility tools built into both the Windows platform and the Internet Explorer browser.
As each control is added to a test, a tester adds assertions to ensure that responses are produced correctly. Field properties are compared with expected values, and the generated C# code can be edited to produce more general purpose tests: handling sequences of inputs rather than single application interactions, for example. As Coded UI Tests are focused on specific controls, they can be affected by changes in application design, and so need to be kept in step with application development. Editing the underlying XML models isn’t recommended, so you should use the tools built into Visual Studio to handle this. The process can be simplified by abstracting the test design, and using initialisation methods in the C# test code to set test-specific information.
Visual Studio 2010 has a role-specific version for testers which includes support for acceptance and user tests. If you’re using Visual Studio Team System to manage your application lifecycle you’ll find this addition extremely useful as you can use it to manage functional testing as well as more development-oriented tests. Functional tests can be defined as test cases in Team Foundation Server, and can be managed using Visual Studio’s built-in testing tools. Load and stress tests can be managed alongside more traditional functional tests, ensuring that performance metrics are handled alongside user interface tests. If you’re using an Agile development methodology then your user stories can be linked directly to test cases through Visual Studio 2010’s Agile template. More formal methodologies can use the CMM process template to manage requirements and use these to build test cases and tests.
Visual Studio 2010 comes with a new standalone test management tool in Microsoft Test and Lab Manager (originally codenamed ‘Camano’) for working with functional tests. Test cases stored in Team Foundation Server are managed using the TestRunner. This records test operations and includes a list of actions carried out as part of any test process. This is more than a screen movie, as alongside a screen capture there’s a complete stack trace that can be replayed within Visual Studio. This historical debug trace helps developers remedy any defects that arise during acceptance and functional testing.
TestRunner recordings can also be used to reply test actions, automating what could be a long sequence of manual tests. This can save time and reduce errors when reproducing bugs or retesting after a fix has been deployed. Recorded tests can also be used to build Coded UI tests, allowing further automation of the manual functional test process.
Managing functional testing is about more than just running and reporting on test results. If a requirement or user story isn’t linked to a test case, Microsoft Test and Lab Manager will alert the test designer or the tester as this can mean an incomplete testing procedure. Like the testing tools in Visual Studio 2010 itself, there’s also an impact analysis feature which will warn testers of tests that need to be rerun due to changes in requirements, or if the underlying code has been updated. Functional tests need to be part of any regression testing process, and this helps ensure that code changes don’t affect the overall functionality of a system.
Rational Functional Tester
Windows applications aren’t the only code that needs functional testing, and IBM’s Rational Functional Tester works with Eclipse to give Java developers a complete functional testing environment with support for both Web and native user interfaces. Tests are built using recorded actions, and you can use these to build a storyboard of your tests.
Application snapshots show the various states of your test, and these help you to add output and input verifications, as well as using natural language test descriptions to specify the actions taken by a test. This approach makes it easy to see just what a test will do, letting you match a test to the original requirement or to a user story.
Tests often need to be run more than once, with different data. You can use Rational Functional Tester to build scripted tests, using either Java or Visual Basic. There’s also support for data driven tests, with a wizard to help build data sets that uses a spreadsheet-like view to create and manage data. You can use this to build relatively simple scripted tests without writing any code. Dynamic data can also be built using regular expressions, so things like date and time can be automatically varied.
Rational Functional Tester will also work with other platforms, including .NET code, any Web application, and terminal applications running on mainframes. There’s support for newer rich Internet application platforms like Adobe’s Flex (and of course Flash), and AJAX frameworks like Dojo. You can even use it to test applications that output PDF documents alongside traditional screen-based output. You’re not limited to using Internet Explorer to test Web applications, as there’s Firefox support as well. You can also use Rational Functional Tester to work with applications running on both Red Hat and SUSE Linux, though you can’t use them to record and edit test scripts.
Service Oriented Applications
More complex projects can take advantage of the functional testing tools built into Parasoft’s SOAtest. Designed to test service-oriented applications, SOAtest is designed to incrementally build test suites, starting with unit tests, moving to integration tests and then finally to functional and acceptance testing. SOAtest also goes one step further, using functional test suites as a basis for security testing, as well as the more common performance and load tests.
One aspect of SOA application development that requires rigorous testing is an application’s APIs and interfaces, and especially any Web services it exposes. These need to be well defined, as they are used to connect applications and to build complex services, so they need to be tested to ensure that the correct response is delivered for any input. SOAtest test suites can be built from standard Web service descriptors, including WSDL documents and UDDI registries, as well as more specialised descriptors such as BPEL files. There’s also support for information held by commonly used enterprise frameworks and enterprise service buses, such as Sonic and Tibco. As Web service descriptors are functional descriptions of the services, they’re an ideal basis for functional tests.
As its name suggests, Parasoft SOAtest is particularly suited to the testing of service-oriented applications.
You’re not limited to the services SOAtest supports directly, and you can create your own tests that can work across multiple Web service requests. If you’re working with Web 2.0 services, there’s also support for REST connections, which run over HTTP. SOAtest can handle most common REST response types, including XML data and JSON. JSON support is increasingly important as it allows JavaScript applications to work directly with REST-based Web services.
It’s important for tests to be parameterised so that once a test has been developed, it can be quickly extended to support changes to an application or to the functional specification. SOAtest will work with data held in Excel spreadsheets, giving you an easy to develop source of complex data that can be used to compare input data with expected outputs held in another spreadsheet.
The latest generation of functional test tools integrate with other test suites, allowing you to automate even the most complex of user interface tests. Building functional testing into test environments also means that it can be tied into code coverage tools, ensuring that an application is fully tested and that any code changes are also tested. The result is functional testing that need not be a time and resource sink. That’s most definitely a win, not only for design and development teams but also for the end user.