Topic: Testing Framework for ILU

There are two components to this testing framework. The first is the individual test script, written in Python. Each script can make use of a set of Python classes which encapsulate standard testing behavior, and have access to the configuration information for ILU. The classes provided encapsulate things like a ClientServerTest, which runs a server, then runs a client against the server, and looks for an error-free completion of the client. The testing framework takes care of things like establishing a temporary binding directory, or running a temporary simple binding server.

The second major element is the automated framework that builds ILU and runs all the test scripts. This component reads descriptions of ILU configurations from a configuration description file, then builds and installs an ILU tree, either from an ILU tar file, or from an RCS tree, then runs the test scripts in the various `example' subdirectories of the installed ILU, once for every configuration tested.

Typically, there are zero or one test scripts in each subdirectory of `ILUSRC/examples/'. However, there may be more than one in a directory. These scripts are installed into the `ILUHOME' tree during the make Install step, and are used for testing the installed examples.

Each script is written in Python, and will be invoked as script input to the Python interpreter. Typically, a script creates one or more instances of a test object, an object which at its most typical binds together the running of a client and server. After creating a number of instances, the script will run each test object, which causes the test to be run. If all of the test objects run successfully, the script exits with a status of 0; if any of the tests exits unsuccessfully, it should exit with a non-zero status.

The test scripts may make use of a number of data structures and classes defined in the Python module ILUTesting. The simplest way to access them in a testing script is to import ILUTesting at the top of your script.

A number of test classes are defined. They are all subclasses of the abstract class ILUTestingMachinery.Test. Each test should be defined by creating an instance of one of these classes. The test can then be run by calling the run method on the test instance (though this is usually left to the function run_tests). The following kinds of tests can be defined:

• SimpleClientServerTest (testname, client command, server command) -- this object encapsulates the normal client server test. The server is started by executing the server command in a subprocess; a pause then ensues, for the server to start up; the client is then run. If the client exits with a zero status, the test is successful. If it exits with a non-zero status, the test is unsuccessful, and the log files of both the server and client are written to standard output. This object has an instance variable, server_startup_delay, which defaults to 5 (seconds); this value may be changed after the instantiation of the object if a longer delay is required for a specific test.
• ClientSBHServerTest (testname, client command, server command) -- this class embodies a client-server test where the server explicitly outputs a string binding handle to standard output, and the client then uses this SBH to bind to the server. The client command may use the string "$SBH$" in its arguments, and that string will be replaced by the generated SBH before the client command is executed. This object type also supports the server_startup_delay instance variable, as described for SimpleClientServerTest.
• SimpleSingleProgramTest (testname, command [, environment]) -- this object type encapsulates a single program, which should run and exit with a zero status. The command used to start the program is given with command; optionally, an environment may be specified which can be used to augment or override the default setting of various environment variables when the program is executed.
• MultiClientMultiServerTest (testname, client commands, server commands) -- this object captures a test where several servers, or several clients, may have to be run. It will start all of the servers, then run each of the clients sequentially. If all clients complete successfully, the test terminates successfully. If any client terminates unsuccessfully (exits with a non-zero status), the test itself terminates unsuccessfully. This test also supports the instance variable server_startup_delay, but it applies to each of the servers of the test.

Several dictionaries of configuration information are also available to the test script:

• iluconf_dict contains the contents of the file `ILUHOME/imake/iluconf.h', which is mainly configuration options and settings for the C compilations. Each macro defined in that file with a #define construct is mapped to a key-value pair, with the macro name being the key, and the macro expansion being the value. This can be tested for various optional attributes; for example, you can check to see if a particular protocol has been configured into ILU.
• iludefs_dict contains the contents of the file `ILUHOME/imake/ilu.defs.new', mapped as for ILUTesting.iluconf_dict, with only those constructs defined as C macros, with #define statements, being mapped to key-value pairs. Note that any constructs defined in `ilu.defs.new' with the form KEY = VALUE will not appear in iludefs_dict. This can be used to test for configuration attributes which do not appear in the `iluconfs.h' file.
• makefile_dict contains either the values defined in the `Makefile' in the current directory with the construct KEY = VALUE, or the values defined in in `ILUHOME/imake/ilu.defs.new', if no `Makefile' exists in the current directory. Note that the contents of the Makefile have already been passed through the C pre-processor, so the values here are the actual values used in the build phase of ILU. This can be used to check for various values, such as the location of a particular library.

A number of strings, which abstract commands for running a particular scripting language interpreter, are exported:

• java_command -- an invocation of the Java interpreter;
• python_command -- an invocation of the Python interpreter;
• lisp_command -- an invocation of the Common Lisp interpreter;
• guile_command -- an invocation of the Guile interpreter

The boolean value ilu_threaded will be TRUE if ILU has been built with support for threads.

Finally, the function run_tests, which takes a list of test objects as an argument, will run through the tests one at a time, printing the name and completion status of the test to standard output. If a test terminates unsuccessfully, the logs of the client and server will be copied to the standard output.