Topic: Mobile Software Testing Tools

Test Tool Basics

There are different types of software testing tools that you should consider. Static tools, for instance, do not execute any code. Instead, they look through code statements and check for mistakes, such as syntax errors. The actions of a compiler fit within this static category.

Of course we’re usually forced to use a compiler every time we build a program. If errors are bad enough, the program won’t run, and you have to fix the defects before moving forward. It’s difficult to be lazy and skate past that one.

Believe it or not, many programmers don’t perform much testing beyond what the compiler accomplishes. They may perform a few functionality tests after coding, but checks are often not made to the internal structure of the program. This certainly isn’t enough effort to produce quality code! There may be a defect lying underneath that will appear when the right sequence of events occurs.

Dynamic test tools go a bit farther and actually execute the code. The debugger that is commonly part of an IDE, for example, runs the code and looks for various errors. There are many other test tools that fit into the dynamic category, and most of them offer distinct enhancements over using the debugger.

For example, the Palm OS Emulator monitors code while emulating the operation of the handheld device. If the code does something that falls outside of Palm's programming guidelines, the Emulator provides an alert. Maximize this type of testing before moving the code to the target hardware in order to make more efficient use of your time.

The Palm OS Emulator performs approximately twenty debug options, such as monitoring accesses to hardware registers. The Emulator ensures that the program doesn’t utilize memory ranges reserved for important devices, such as external LCD and USB controllers. Another monitoring function of the Palm Emulator that I especially like is memory leak detection. The Emulator checks for memory leaks on SysAppExit and records this valuable information to a log file.

Code Coverage Tools

When developing mobile software, you should define test cases that adequately exercise the code. Coverage testing is critical because it looks at the guts of the code and determines which portions that the test cases actually cover. This allows you to be certain that the testing will successfully find defects in the program.

Apply coverage test tools diligently during unit testing because failures in lower-level code has much less impact and is less costly to fix when found early in the development cycle. Avoid “big bang” testing at the end of the coding phase. Instead, you should have many of the defects found and corrected early on and only have minor tweaking to do just prior to release. So, employ coverage tools and reduce risks (and stress) toward the end of the project!

Coverage tools let you add test cases directly when the tool finds uncovered areas. I’ve found this process to be extremely helpful. You can keep adding test cases and re-running the tool until you’re happy with the amount of coverage. Redundant code is also found with coverage tools, which prompts you to do some smart pruning in order to make the code run more efficiently and consume less memory.

Based on test cases, coverage tools report various measures, such as statement coverage, decision coverage, and condition coverage. This lets you know specifically how much of the different aspects of the program that your particular test cases cover. Most coverage tools also have profiling, which provides statistical analysis of a program. For example, a profiler determines the percentage of execution time of each function, which helps you discover inefficiencies in the code.

For example, CodeTEST by Metrowerks works well within a Symbian environment and uses instrumentation technology to monitor the code. A CodeTEST test setup uses tags that inject into the code during the build process, which doesn’t affect the original source. This is important for when you’re done with testing because you don’t have to withdraw the tags from the original source code – you’re ready to move quickly forward. CodeTEST outputs a software execution trace similar to a logic analyzer, which comprises performance, coverage and memory analysis.

How much of the code do test cases need to cover? This is a common question, and I’ve found that 80 to 90 percent statement, decision and condition coverage is adequate in most cases just prior to release. This is often enough to avoid costly fixes after fielding the software. If safety is a concern, though, push for greater coverage.

Complete, 100 percent coverage would be ideal, but with most mobile applications, you’ll likely incur significant costs when nearing this. As an alternative, consider conducting formal code reviews to supplement the use of test tools, especially with critical functions.

Test Scripting Tools

Assurance that test cases exercise the appropriate portions of the code significantly advances the quality of the software, but it’s generally not sufficient. Also strongly consider making use of scripting tools to automatically drive the program with varying inputs. You can easily build a test harness with these tools and perform a much wider scope of testing as compared to using only human testers.

The Nokia Testing Suite, for example, supplies a free tool for Symbian that exercises user activities in an emulator and on the target device. In fact, Nokia requires the program to pass these tests to gain certification. In order to operate on the client hardware, you install a small amount of small code on the hardware device, which relays test results to a host system. Most other scripting tools work in both simulation and the hardware device as well.

The Palm OS Emulator provides scripting through the use of “Gremlins” that generate different user inputs. A Gremlin can perform a specific input function over a specified number of times or forever in a loop. You can setup a series of Gremlins to operate the program overnight, for instance, to thoroughly test the application.

Also, the TestQuest scripter tool allows you to enter user interactions via a graphical representation of the client device. With this tool, you can even execute tests between multiple hardware devices over Bluetooth wireless connectivity. A strong advantage of the TestQuest scripter is that it works with a variety of platforms, including Symbian, Intel PCA, Palm, and Microsoft mobile devices.When testing with actual client devices, be sure to exercise applicable failure modes while the test scripts are running. For example, operate a microwave oven within a few feet of the wireless client when client functions are accessing data from the server. The RF interference from the microwave oven will cause the wireless network to have lower performance while the oven is operating. Be certain that functionality is still acceptable in this situation.

Also, apply scripts that check behavior of the application while the device is taken outside the range of the wireless network and then brought back into range. You need to ensure that the application handles these scenarios gracefully and that no data is lost or corrupted on the mobile device or the server.

Additional Tips to Consider

Many mobile IDEs come with an emulator, but ensure that you have scripted test tools as well that operate in both the emulator and target hardware. Start with testing in the emulator to get rid of the majority of defects, and then move the code to the actual hardware for further testing. This allows you to minimize the time-consuming re-flashing of the hardware device with program changes by maximizing the correction of defects in the simulation phase. The result is often a shorter test effort.

Ensure that the chosen mobile test tools have utilities designed for your target device platform. For example, Metrowerks test software and Borland C++ Builder 6 Mobile Edition have tools compatible with Symbian Series 60 code, but others don’t. Start your search for mobile testing tools with the target platform in mind. Make certain that the tools have emulator skins that match the layout of your target device.

With some test tools, you may not experience certain problems, such as memory leaks, until the software runs on the actual device platform. Other system functions, such as wireless connectivity, may also not come into play until using the software on the actual device. After running tests on the emulator and making necessary medications, move forward with the real thing.