Whenever possible packages should be shipped with one or more test cases. This allows users to check that all packages function correctly in their particular configuration and on their target, which may be custom hardware unavailable to the package developer. The component framework needs to provide a way of building such test cases. For example, if a makefile system is used then there could be a make tests target to build the test cases, or possibly a make check target to build and run the test cases and process all the results. Unfortunately there are various complications.

Not every test case will be applicable to every configuration. For example if the user has disabled the C library's CYGPKG_LIBC_STDIO component then there is no point in building or running any of the test cases for that component. This implies that test cases need to be associated with configuration options somehow. It is possible for the test case to use one or more #ifdef statements to check whether or not it is applicable in the current configuration, and compile to a null program when not applicable. This is inefficient because the test case will still get built and possibly run, even though it will not provide any useful information.

Many packages involve direct interaction with hardware, for example a serial line or an ethernet interface. In such cases it is only worthwhile building and running the test if there is suitable software running at the other end of the serial line or listening on the same ethernet segment, and that software would typically have to run on the host. Of course the serial line in question may be hooked up to a different piece of hardware which the application needs to talk to, so disconnecting it and then hooking it up to the host for running some tests may be undesirable. The decision as to whether or not to build the test depends not just on the eCos configuration but also on the hardware setup and the availability of suitable host software.

There are different kinds of tests, and it is not always desirable to run all of them. For example a package may contain a number of stress tests intended to run for long periods of time, possibly days or longer. Such tests should certainly be distinguished somehow from ordinary test cases so that users will not run them accidentally and wonder how long they should wait for a pass message before giving up. Stress tests may also have dependencies on the hardware configuration and on host software, for example a network stress test may require lots of ethernet packets.

In the current implementation of the component framework these issues are not yet addressed. Instead there is only very limited support for building test cases. Any package can define a calculated configuration option of the form CYGPKG_<package-name>_TESTS , whose value is a list of test cases. The calculated property can involve an expression so it is possible to adapt to a small number of configuration options, but this quickly becomes unwieldy