In the last Qt Quarterly issue, we presented QtTestLib, Trolltech's lightweight unit testing framework for Qt 4. We will now review KD Executer and Squish, two powerful commercial tools that enable you to create sophisticated automated GUI tests – increasing the quality of your application and reducing testing costs in the long run.

• KD Executor 2.1 by KDAB
• Squish 2.1 by froglogic
• Conclusion

• It records events for each widget individually rather than for the application as a whole. A pressed button located at position (100, 20) at record time will be pressed correctly during playback, even if it has been moved to position (400, 80).
• It records the intention rather than the actual action on many of Qt's widgets. For example, think of a QTabWidget, which consists of several tabs. During the recording session, you press the third tab. KD Executor will recognize a mouse event on the QTabBar and will record which tab was at that screen position.

This smart recording technology is available for many of Qt's built-in widgets. If you develop custom widgets, you can use KD Executor's extension mechanism to enable this smart recording technology for your own widgets.

Moreover, KD Executor increases the efficiency of the communication between testers (or customers) and developers. Instead of describing a defect in an application, testers can simply record the situation where the defect shows up and send the resulting platform-independent script to the engineers. Scripts can be recorded as XML or as JavaScript (using QSA, Trolltech's official JavaScript/ECMAScript implementation).

The KD Executor Shell uses a simple and convenient approach to testing a Qt application. It lets you create and manage test suites and test cases in a friendly graphical testing environment. One of the main design goals of the KD Executor Shell was to make it as easy to use as possible, even for non-programmers, while still keeping the full power of scripting behind it.

More advanced users who already have a test environment in place may use KD Executor as a powerful record/playback engine, by running and configuring it from the command line. This allows test cases to be executed automatically from batch or scheduled jobs.

Launching the KD Executor Shell gives you an overview of all test suites, test cases, and test runs gathered in one table. A result table is located behind the result tab, with all performed tests ordered and collected by date and time when the execution took place.

By recording user inputs, KD Executor quickly captures new tests. There are no complicated test scripts to write or debug. At any time during the capture process, you can probe your application to record key properties. The values of these properties become test benchmarks. KD Executor's sophisticated test management subsystem allows you to replay these tests in any combination and automatically validate the execution of your application against these test benchmarks.

Detailed Results shows more information about a particular test result. It displays the different test aspects, and whether the test case failed or passed.

The tab labeled KD Executor Errors will display any internal errors of the application under test that might occur during the execution of a test case. This is to prevent the test run from being locked up by any error dialogs.

Stdout and Stderr show the standard output and standard error data coming from the application being tested, and are often useful tools for finding out why a test case failed.

All information about test suites and tests is presented in a tree view. Test suites can contain both test cases and other test suites, whereas the test cases themselves contain all test runs executed on them. The user can create, import, edit, or remove any of these items by right-clicking on the item and selecting the requested operation from the context menu.

The test suite tree can contain several test suites:

• The Last Result for a test case can have four different values: OK, Failed, Needs Human Intervention, or Never Run.
• Everywhere in the KD Executor Shell, context menus provide fast access to the most common operations.
• Right-clicking on any of the items will display a menu with options for that specific item.

If you have recorded any Qt properties, you can view their baseline values or the values that were recorded during a test execution by launching the Property Editor which lists all properties picked during script recording. If the Property Editor was invoked on a test result, it will also show the actual value of the property during playback. A similar Assertion Editor lets the user view the assertions for a particular test case, as well as the results during test runs.

Logs and tests generated on one platform can be run on any other platform supported by Qt. KD Executor also provides test run reports in HTML format. Publishing those reports on an intranet site is a convenient way of communicating the test results within an organization.

There are two main ways to use the KD Executor core engine.

• Pre-loading: Simply type kdexecutor on the command line before the name of your application. The KD Executor configuration dialog will pop up, allowing you to set various options. Most settings can also be specified on the command line.
• Source code modification: It is possible to compile KD Executor into your binary. This is useful if you want to limit the usage of your software (e.g., for playback-only demo versions), or if you want to use KD Executor in an already tailored C++ test environment.

Squish 2.1 by froglogic

Squish 2.1 is latest version of the automated testing tool from froglogic, a German software vendor that was founded by two former Trolltech senior software engineers. Squish provides a fully-featured automated GUI test tool for Qt applications that competes with established tools such as WinRunner, Rational Robot, and SilkTest while adding the extra value of a tight and complete Qt integration.

Squish allows you to record tests using several popular scripting languages. You can add verification points to the scripts to ensure that the application is behaving correctly, and rerun the tests as unattended regression tests.

Squish can be used to test Qt 3.x and Qt 4.x applications on Windows, Linux, Mac OS X, and several commercial Unix variants. Squish can also be used to test Qt/Embedded 2.3, including Qtopia applications, running on the target device or inside Qt's Virtual Framebuffer The product is packaged with its complete source code, thus enabling a compile-time adaption to any particular Qt configuration.

The Architecture

For testing with Squish, no modification of the application under test (AUT) is required. On application startup, Squish dynamically injects a module into the application's process space. The module takes over control and provides a TCP/IP network connection between the AUT and the tool driving the test.

This architecture allows for applications to be tested on different remote hosts, as well as multiple AUTs in parallel, which can be useful for testing complex client--server systems.

Scripting Languages and Bindings

Squish currently supports Python, JavaScript, and Tcl, and it is expected that a future version will support Perl as well. Using a standard scripting language over a "vendor script" offers the following benefits:

• Familiarity with the language.
• Wide range of resources (documentation, tutorials, books).
• Rich set of language features (e.g., control flow structures).
• Availability of third-party extension modules.

Squish extends the native script language with test-related functions and complete bindings to Qt, enabling the creation of very powerful test scripts. It is even possible to automatically create script bindings for any C++ library or application code, which can then be accessed from all available scripting languages.

Recording a Test

Squish organizes tests in test suites. A test suite consists of a number of test cases and possibly shared data and scripts.

A test case contains a test script which can be implemented in any of the supported scripting languages. Additionally, we can attach test data to a test case.

To create a test script, we normally start by "recording" it, meaning that the test engineer performs the actions on the AUT which should be tested while Squish is running. The result is a script that can be replayed at will. Thanks to this recording facility, entire tests can be created without writing a single line of code. Apart from being convenient, this allows the involvement of test engineers who don't have coding skills.

The following JavaScript test script was recorded by Squish for Qt 4's Spreadsheet demo. The user changed the value of the expenses for "Hotel" from 300 to 400 and exited the application:

function main()
{
    doubleClickItem(":Spreadsheet.Table1.QWidget1",
                    "300", 95, 17, 0, Qt.LeftButton);
    type(":Spreadsheet.Table1.QLineEdit1", "<Backspace>");
    type(":Spreadsheet.Table1.QLineEdit1", "<Backspace>");
    type(":Spreadsheet.Table1.QLineEdit1", "<Backspace>");
    type(":Spreadsheet.Table1.QLineEdit1", "400");
    type(":Spreadsheet.Table1.QLineEdit1", "<Return>");
    activateItem(":Spreadsheet.QMenuBar1", "File");
    activateItem(":Spreadsheet.QMenuBar1.QMenu3", "Exit");
}

As this script illustrates, Squish records high-level actions rather than low-level events. For example, it records a double-click on a specific widget rather than mouse events for specific screen coordinates. Similarly, it records menu item activations instead of individual mouse events. This makes the script more readable and maintainable, but also more robust against changes in the application's user interface. This robustness is crucial if the test script is expected to run correctly on different platforms and computers.

The above script could now be run to see if the application behaves correctly – however, the goal of automated testing is to perform this kind of verification without any human interaction. This is accomplished using verification points. A verification point compares an actual result against an expected value. Comparisons are normally based on properties of objects (e.g., the text of a spreadsheet cell).

If at some point it is necessary to change a small part of a test script, Squish lets us run a test until a certain point and record from there, inserting the newly recorded code into the existing script instead of recording the entire script.

Object Property Verifications

To continue with our example, we now want to check that the value for total expenses is recalculated correctly after the value for hotel expenses was changed. Inserting a verification point can be done in Squish using a comfortable point and click interface.

Verification points are stored in XML files and can be edited using an integrated editor. The script command test.vp( "\"VP_ChangeHotelExpense\"") is inserted automatically into the test script and will perform the comparison. (VP_ChangeHotelExpense is a unique identifier of the verification point.) Property verifications can also be expanded into script statements. For example, the verification point from our example checking the hotel expenses and total sum in the table would result in the following script calls:

test.compare(findObject(":Spreadsheet.Table1.item_7/1")
             .text, "400");
test.compare(findObject(":Spreadsheet.Table1.item_9/4")
             .text, "22433");

It is also possible to add such verifications to the script manually to perform more sophisticated checks. These checks can use any control flow statement and any part of the Qt API. Here's a Python example that checks that a QListView contains exactly three items with the text "Banana":

listview = findObject("MainWindow.listView")
found = 0
item = listView.firstChild()
while not isNull(item):
    if item.text(0).contains("Banana"):
        found += 1
    item = item.nextSibling()
 
test.compare(found, 3, "Check number of Bananas")

There are some cases where the only way to check for correctness is to compare a widget to a previously taken screenshot (the expected result). This is usually the case for graphing widgets, drawing canvases, etc. Squish supports the creation of screenshot comparisons by choosing the widgets to verify.

To make screenshot comparisons more robust, test engineers can define which regions of the image are relevant to the comparison. This can be used to filter out labels in a diagram, which might change their appearance depending on the machine and font settings.

After having created a test, it can simply be executed in the Squish IDE. The results of the verification points and any other errors which might have occurred are listed in the result log. Clicking on a test result takes the user to the file and line of the corresponding test script.

The IDE also allows for a closer inspection of test failures by displaying the deviation between expected and actual results. In the case of screenshot comparisons, multiple ways are offered to visualize the differences. In the screenshot above, all changes to a chart diagram are displayed in white (the Chart example, located in Qt 4's "examples/itemviews/chart" directory, served as the AUT).

The ultimate goal of automated GUI testing is to run the tests in unattended mode. Squish includes a set of command-line tools that make it easy to integrate the tests into a cron job, a Windows service, or a test management tool. This needs to be done only on one machine on the network; the command-line tool executing the tests can run the tests on other machines, regardless of the operating systems in use.

Test results can be generated as XML files, which can be post-processed to create HTML output, or fed into a test management or bug tracking system.

To create robust and maintainable tests, test engineers will often edit the recorded tests to factor out common code. For example, the GUI actions required to open a file in the AUT might be put in a separate script function. Such functions are available to all the test scripts from a test suite.

If scripts are manually written, the script coders need to know the names of the objects and properties of the AUT. This can be achieved easily using Squish's Spy, a tool that inspects the object hierarchy and properties of a running Qt application.

Another Squish feature that eases the task of creating robust tests is the object map. It maps symbolic names of objects to the real names in the application. In scripts, symbolic names are used. So when some object names in the application change, only the real names in the object map need to be updated; no script code needs to be changed.

A common method used to make a test easily extensible is to introduce data-driven testing. With this appoach, the data is written into a separate file or a database, instead of being hard-coded in the script. The test is then executed for each data record. This way, tests can be extended without touching the test script at all. Squish provides APIs and tools to take advantage of data-driven testing.

More information about KD Executor, including free evaluation versions, is available from www.kdab.net/kdexecutor.

To learn more about Squish and request an evaluation version, visit www.froglogic.com or contact squish@froglogic.com.