A look at the new crop of client/server testing tools for beleagured developers.
Back in the good old days, traditional batch programs were relatively easy to debug. They typically read one or more flat input files to produce one or more output files. To test the program, an input file was fabricated to exercise program logic. Known input records generated predictable output values. If the output did not match the anticipated results, then the program had a bug. The bug was isolated and corrected and the testing repeated. A finite set of tests was sufficient to test the system completely.
Things have become a little more complicated now, and GUI applications require significant changes to this straightforward testing method. For starters, output from GUI systems is not well-suited to simple comparisons of flat files; and other testing complications include the partitioning of computing effort in client/server systems, and the fact that event-driven GUI software lets users perform actions in unpredictable sequences. Graphical interfaces provide a wealth of information and great flexibility to users, but unfortunately this makes the testing process significantly more difficult. According to a December 1995 International Data Corporation Testing Report: "As popular visual application development tools such as PowerBuilder and VB have facilitated more rapid generation of client-centric client/server applications, the time and effort to test these applications have actually increased." Furthermore, an October 1994 "Success in Testing" report from the Standish Group concludes that: "Only 16.2% of application development projects in mission-critical, client/server environments succeed in being delivered on time, within budget, with all functional capabilities intact, and operating."
To succeed in today's demanding, rapidly changing environment, client/server developers need to take a serious look at the testing tools and techniques they use. No longer can they put off testing until the very end of a project when there's generally little time and few resources available for testing. Testing personnel can no longer be determined by who hasn't already been assigned to the next project. Testing techniques can no longer be limited to feedback message statements and a debugger. To test all aspects of client/server applications successfully, testing must be more rigorous, more professional, and more thoroughly planned.
Fortunately, a new crop of client/server test tools is available to assist beleaguered developers. This new generation of client/server testing tools has the following characteristics:
* They were developed specifically for client/server applications.
* As with the development environments they test, they boast graphical interfaces.
* Capture-and-playback testing has become commonplace.
* Regression testing is simple and effective and should occur across the entire life cycle of the software development process.
* Again, as with the development environments, they are easy to use.
* Developers can weave test scripts together and run them unattended.
* A single tool can test diverse hardware and software platforms. Testing can be performed on the client, the server, or both.
* Test tools generate reports and other testing documentation.
* They recognize and seamlessly handle GUI objects in the development environment.
* Wizards within the tool can lead testers through the process.
* Stress testing simulating hundreds or thousands of users can be performed using only one or a small number of workstations.
* Tools can automatically generate realistically large and diverse volumes of test data.
* Scripting languages are becoming less proprietary.
* Test planning and management capabilities are more common.
* Testing tools are gaining the ability to test Web sites and applications.
I'll describe these features in more detail, with examples from various testing products.
Just as client/server development tools are relatively new on the software battlefield, modern testing tools have adapted to handle client/server testing requirements. Many of the vendors that have developed and marketed these client/server testing tools date back to the earliest days of client/server. Mercury Interactive was incorporated in 1989, Performance Awareness is currently celebrating 10 years of business, AutoTester Inc. has been in the industry since 1986, and Pure Software Inc. was founded in 1991.
One of the primary advantages of client/server systems is that their graphical nature enables users to be more productive. There's no reason why application testers can't experience similar gains in productivity, as most current testing toolsets are highly graphical: Users execute tests by pointing and clicking icons in the testing product's graphical interface, and testing tools are replete with menus, toolbars, drag-and-drop, drop-down list boxes, and other GUI features.
The fundamental concept underlying capture/replay testing products is that the testing tool captures all user interactions with the application and database server. This includes keystrokes, mouse movements, menu selections, drag-and-drop operations, and button clicks. Testing programs also recognize and capture responses that the user sees, such as windows changing, drop-down list boxes appearing, messages and dialog windows displaying, and so on. At a later time when the testing tool replays the script, it initiates the same actions that the original user performed. The test program also compares responses that occur during the playback testing phase with those that occurred during the original recording cycle. The test program then documents any differences in responses.
Before a tester steps through the functions of the application, most testing tools require the tester to run a "capture" program in the background. SQA Inc.'s SQA Robot has an intuitive interface. The tester initiates SQA Robot, sets the desired recording options, names the output file (or accepts the system generated name), and starts recording. The user can pause or terminate recording at any time. When the user is satisfied with the test procedure, the recording can be stopped.
While recording with SQA Robot, the tester also has the option of inserting any of the following into a test procedure:
* Test cases that capture specific conditions, states, or values during the testing procedure, and record and store the data as the expected baseline for the application being tested.
* Wait states provide a way to synchronize test-procedure playback with the application being tested. Setting wait states lets the tester define specific conditions for which the testing procedure waits before continuing with subsequent testing actions. Some examples of these conditions include waiting until a specified file exists, waiting until the contents of two specified files match, and waiting until a selected window matches the recorded window image exactly. (See Figure 1.)
* Timers allow the insertion of start and stop timer commands that record the duration of events in a testing procedure, such as database or network access.
* Program calls execute testing procedures or other applications.
These options let testers customize the testing process in numerous ways.
The goal of regression testing is to be certain that changes to the application have not caused it to regress, or move backward, and become more unstable or buggy. Enhancements that developers make to a system stand a fairly good chance of introducing bugs into code that previously worked correctly. Perhaps the most frequent reason for performing regression testing is to make certain that correcting one bug has not introduced other unexpected bugs into the system. Some other common potential triggers for regression testing include a new version of the operating system, a new release of the development tool, or changes made to the database. Regression testing is essential throughout the software development life cycle, especially during maintenance.
Client/server, GUI systems have some additional reasons for requiring especially thorough regression testing. The most significant reason is that many development tools rely heavily on inheritance. When windows, controls, and other components inherit functional capabilities (and properties) from ancestors, changes to ancestors can have unpredictable effects on the descendants. Modifying an ancestor should trigger the regression testing of all of its descendants.
Partitioning an application among the client workstation, the database server, and an application server is becoming more common and more expected. The ability to partition applications lets developers adjust the workload more dynamically. The results of partitioning should not include the introduction of errors into the system, but without regression testing this possibility cannot be eliminated.
The list of events that create the need for regression testing is long. How many projects teams have the time and resources to perform regression testing conscientiously for each circumstance that requires it? The realistic answer is none or very few. Fortunately, many, if not most, of the client/server testing tools make regression testing easier. For most tools it is simply a matter of designing and running the appropriate test scripts.
preVue-X from Performance Awareness Corp. is one example of a testing product that automates the regression phase of the development cycle. With preVue-X, testers have the flexibility to:
* test without special libraries or application hooks
* execute multiple tests overnight, with accurate, repeatable results
* schedule computer time efficiently during the test phase
* verify that received responses are correct, and measure and record the client/
host response times
* test new features while performing regression testing
Almost all of the new generation of tools are sufficiently friendly to allow application end users to work with them. Involving application end users in the testing process frees up developers to devote more effort to fixing bugs and completing the development work. For example, end users can use AutoTester to test client/server applications. The product includes an easy-to-use, menu-driven interface. Tests that users create with AutoTester are automatically documented in an easy-to-read narrative format, ready to execute at the touch of a key, and results are reported for easy analysis and review.
The ability to build test scripts and rerun them repeatedly is an extremely powerful feature. The next higher level of testing sophistication involves combining multiple test scripts and running them together. If multiple scripts run sequentially or concurrently, then an enormous amount of testing can be compressed into smaller windows of time. The ability to run scripts unattended and have the results recorded and documented enables developers to perform testing virtually around the clock.
Many testing packages support unattended running of test scripts. AutoTester Client/Server from AutoTester Inc. stores test results in a database that accumulates a complete history of errors and an audit trail of what has and has not been tested. AutoTester Client/Server also includes sophisticated automatic recovery capabilities that enable true unattended testing. Automatic recovery scenarios handle unexpected conditions as they occur, so that testing can continue without operator intervention. Tests can be run unattended at night, extending the amount of testing that can be accomplished each day.
One way in which client/server systems differ drastically from traditional, mainframe systems is in the diversity of hardware and software involved. Software for a mainframe system has to work with one CPU, one operating system, and one type of dumb terminal. Client/server systems, in contrast, must function in a heterogeneous world. The database server might be a Sun Unix box running Sybase or Oracle as the database engine. An application server might be a Windows NT system. The clients will most likely be a hodgepodge of different makes of workstations. To make matters worse, the client workstation will probably be running a medley of operating systems including Windows 3.11 or Windows for Workgroups, Windows 95, Windows NT, X-terminals, OS/2, and Macintosh.
To be really useful, the testing tool you choose must function on all of the operating environments that run in your shop. Most testing tools work under more than one environment. AutoTester works on Windows 3.11, Windows 95, Windows NT, Unix, and OS/2. Segue Software's QA Partner runs on Windows 95, Windows NT, Window 3.1, OS/2, Macintosh System 7.x, and more than a dozen flavors of Unix. Great Circle from Geodesic Systems works with Windows 3.11, Windows NT, and Windows 95, and will soon be released for OS/2 Warp 3.0 and Sun Solaris X86.
Performing the actual testing is only part of the testing process. The other, equally important requirement is to document and report the test results. Most testing tools are quite strong in this area. The ability to produce a number of standard reports, including graphs and charts, is common. Many products also let testers create customized reports.
preVue-X provides comprehensive post analysis utilities to analyze test results; it analyzes pass/fail bit images, pass/fail textual responses, detailed performance data, and detailed X-Protocol stream analysis for defect isolation.
GUI development packages build applications from objects such as windows, data windows, data controls, menus, check boxes, drop-down list boxes, buttons, and so on. To be effective, a testing tool must be capable of handling GUI objects and not be mislead by minor changes in an object's appearance or position.
The new generation of testing tools is designed to handle GUI and object-oriented development packages. For example, SQA Robot recognizes standard Windows objects such as check boxes, radio buttons, push buttons, combo boxes, list boxes, and edit boxes. SQA Robot even handles product-specific objects such as PowerBuilder DataWindows, Visual Basic Custom Controls (VBXs), and SQLWindows Table Windows. If objects in the user interface change location, tests will pass because SQA Robot scripts are not location-dependent. In addition, scripts require no maintenance as applications undergo cosmetic changes to the GUI.
Testing tools, like so many other types of software products, provide wizards to guide users through tasks. RapidTest from Mercury Interactive Corp. uses automated test-generation techniques combined with easy-to-use wizards. Wizards are specialized software assistants that provide a self-guided visual interface for specifying test requirements and test cases.
RapidTest provides three wizards to address key phases of the testing process: Plan Wizard, Script Wizard, and Cycle Wizard. Plan Wizard is an interactive tool that helps scope out subjects for testing, and identifies tests that need to be run under each subject. Script Wizard is a powerful utility that automatically creates a full suite of GUI tests from the tested application. Script Wizard learns the entire tested application user interface recursively, by automatically navigating its way through all available user interface paths, opening every menu, menu item, and dialog box as if an invisible agent were operating the application. Cycle Wizard provides a point-and-click visual interface to interactively assemble a "build" of tests, to apply to a particular aspect of the tested application.
Stress testing is the process of running client machines in high-volume scenarios to see if the application will "break" under the pressure. Examples of stress testing are running a client application continuously for many hours or running a large number of different testing procedures to simulate a multiuser environment. Types of errors uncovered by stress testing include memory leakage, performance problems, locking problems, concurrency problems, excess consumption of system resources, and exhausting disk space.
It is expensive, difficult, and time consuming to stress test an application adequately using purely manual methods. This is because of the large number of users and workstations required to conduct the testing process. It is costly to dedicate sufficient resources to tests, and it is difficult to orchestrate the necessary number of users and machines.
A growing number of testing tool vendors provide an alternative to manual stress testing. Their solution is to simulate a large number of users interacting with the system from a limited number of client workstations. Generally, the process begins by capturing user interactions with the application and the database server in a number of test scripts. Then the testing software runs multiple instances of test scripts to simulate large numbers of users.
PurePerformix from Pure Software Inc. performs multiuser testing simulations. It creates scripts that represent actual users and their daily, often disparate, operations. With its capture tool, it builds emulation scripts by recording user activities, including keystrokes, mouse movements, and SQL requests. It then arranges a representative mix of scripts and replays them against both the client and the server, testing the client and server simultaneously. It also charts response times that users would experience.
To a very large degree, testing performed on a computer system is only as good as the test database is realistic. A realistic database is one that is comparable in size (that is, number of rows) and complexity (that is, diversity of entries) to the database that the system will ultimately have when it is fully deployed and developed. An unrealistically small test database will not yield accurate timing predictions. Similarly, a test database that is constructed of homogeneous (that is, heavily duplicated) data will not push testing to its limits or test all potential cases.
Unfortunately, manually generating data for a sizable, complex database requires a significant effort. One solution to this problem is BenchWorks from INFOgy Inc. One of BenchWorks' many capabilities is its ability to generate data for populating tables according to the developer's statistical requirements and constraints. It lets developers design data with customized characteristics in each column in a very concise manner. For example, char or string data types will define the data generated by "Min String Length" and "Max String Length" parameters. The generated data may consist of alphabetical characters, digits, or a mixture of both.
Developers can create numerical data that adheres to the following types of distributions:
* ordered sequences of unique elements
* random sequences of random values
* cyclical distribution of the specified data type
* uniformly distributed random sequences
* sequences with a Gaussian distribution with a maximum in the center of the specified range
* exponentially distributed sequences
* integer-valued Poisson distributed sequences
* continuous-valued Gamma distributed sequences
* a random sequence of elements that the user enumerates with frequencies for each element in the distribution specified
* a random sequence of elements that the user enumerates with each specified element's exact number of occurrences
In many databases, dependencies exist between columns. For example, the invoice column in the accounts table might have a foreign-key constraint, referring to the invoice column in the receivables table. This requires that values placed in the account.invoice column also exist in the receivables.invoice column. Generating data that conforms to these requirements is an additional complication. BenchWorks has a powerful feature that checks for consistency and constraint violations before generating the database, and it resolves all potential data integrity problems before populating tables.
The syntax of test scripting languages varies drastically from standardized to proprietary. SQA's scripting language is identical to Visual Basic; Mercury Interactive's scripting language is C-based; BenchWorks' Scenario language is based on SQL with extensions and additions such as looping, embedded variables, templates, and stored procedure calls; and Segue Software's Visual 4Test scripting language includes an outline editor with auto-indenting, simplified syntax, syntax color coding, visual notification of syntax errors, and a multilevel expand/
collapse feature for easy viewing.
The ability to capture and play back test scripts is handy, reports to document test results are also useful, and the ability to stress test a system is extremely valuable. Unfortunately, these individual capabilities are inadequate if the testing process is haphazard and incomplete. To be thoroughly effective, a testing tool must provide assistance at a higher level. It should have the ability to help plan and manage the testing process.
Many testing tools can manage the testing process. For example, by using a powerful set of automated management functions and an open database repository to store and access test information, Mercury Interactive's TestDirector helps to organize the entire application quality assurance process, and provides bug tracking and automatic notification. It coordinates all the efforts of everyone involved in application development, including quality assurance personnel, software developers, system analysts, and information system managers, and this results in greater efficiency and productivity.
SQA Manager from SQA Inc. also provides powerful test-management capabilities. It lets users track software-testing information through all phases of the software development, test, and revision cycles. It provides features that all members of the software development and testing team -- including testers, developers, customer-service personnel, documentation specialists, managers, and end users -- can use. It enables users to plan testing strategies, track information related to test execution, track defects from discovery through resolution, and produce reports to manage the overall software testing effort.
The number of Web sites has been booming recently. More and more companies are developing Web-based applications to give them a presence on the new electronic frontier. Unfortunately, testing Web-site applications has so far been a manual or nonexistent process. Developers had no automated way of testing whether applications worked correctly, what level of load they could handle, or even whether hyperlinks pointed to pages or sites that really existed.
Mercury Interactive Corp. recently introduced WebTest, the industry's first testing technology designed specifically for applications that run on the Web. Developers can use WebTest to measure response time to browser requests and to determine the maximum number of hits a Web server can support. It can also help validate that the application will function correctly under varying conditions. To simulate multiple-user load, WebTest -- used in conjunction with Mercury's LoadRunner -- employs the concept of "virtual Web users." By running hundreds of thousands of virtual Web users, LoadRunner simulates real-life load, measures access response time, and displays the results using an extensive collection of graphs and reports. It's a good bet that most testing tool vendors will Web-enable their testing products and suites.
While client/server GUI applications are significantly more difficult to test than are traditional batch or procedural programs, a new generation of testing tools can help you automate and manage the testing process. Testing tools such as those I describe in this article have been designed from the ground up to test client/server applications thoroughly. With tools such as these available, developers have no excuse for testing the most complex client/server applications inadequately.
The tools described in this article can make the process of testing client/server GUI applications significantly easier and more thorough. They do not, however, make the testing process simple, fast, straightforward, or flawless. No matter how good the testing tool, someone still has to "drive" it. Even a testing tool that includes wizards capable of generating many of the test scripts automatically still requires someone to create the remainder of the test scripts. It takes a human being, one with considerable skill and experience to know what, where, and when to test. Flushing out bugs is still as much of an art as it is a science, even when the developer is using the best tools.
The testing phase of a project needs to be carefully planned. It needs to start early and be well thought-out. Without a carefully planned approach, testing will be haphazard and inconsistent. Testing products that apply a methodology to the testing process can be an enormous help. They can provide a general outline for how, when, and by whom testing should be done. Unfortunately, buying a product of this sort doesn't automatically install understanding into management, wisdom and insight into the test designers, or dedication into testing personnel.
Perhaps most important, no testing tool can decide when the testing process is done. It is very unlikely that the last bug is ever discovered and removed from a real-world application. No matter how well an application is tested, it will still have flaws. There comes a point when it is simply no longer economically feasible to continue testing. Reports can provide statistics on the rate of errors being discovered, but a human tester has to make the decision to cease testing and ship the application.
Testing tools can aid but not substitute for an experienced tester. In the right hands, the proper tool can improve the quality of an application significantly, but the best testing tool is worthless if it sits on a shelf or is used improperly. No product on the market is a silver bullet or a panacea for the testing process. If you or your management think that any of the tools will solve all your testing problems effortlessly, you're sadly mistaken. They are all tools, nothing more. Use them wisely and you will be rewarded, but don't be naive enough to presume that they will do the testing for you.
See the product chart (page 74) for company contact information.
