Automated black box, GUI-level regression test tools are popular in the industry. According to the popular mythology, people with little programming experience can use these tools to quickly create extensive test suites. The tools are (allegedly) easy to use. Maintenance of the test suites is (allegedly) not a problem. Therefore, the story goes, a development manager can save lots of money and aggravation, and can ship software sooner, by using one of these tools to replace some (or most) of those pesky testers.

These myths are spread by tool vendors, by executives who don’t understand testing, and even by testers and test managers who should (and sometimes do) know better.

Some companies have enjoyed success with these tools, but several companies have failed to use these tools effectively.

In February, thirteen experienced software testers met at the Los Altos Workshop on Software Testing (LAWST) [2] for two days to discuss patterns of success and failure in development of maintainable black box regression test suites. Our focus was pragmatic and experience-based. We started with the recognition that many labs have developed partial solutions to automation problems. Our goal was to pool practical experience, in order to make useful progress in a relatively short time. To keep our productivity high, we worked with a seasoned facilitator (Brian Lawrence), who managed the meeting.

These were the participants: Chris Agruss (Autodesk), Tom Arnold (ST Labs), James Bach (ST Labs), Jim Brooks (Adobe Systems, Inc.), Doug Hoffman (Software Quality Methods), Cem Kaner (kaner.com), Brian Lawrence (Coyote Valley Software Consulting), Tom Lindemuth (Adobe Systems, Inc.), Brian Marick (Testing Foundations), Noel Nyman (Microsoft), Bret Pettichord (Unison), Drew Pritsker (Pritsker Consulting), and Melora Svoboda (Electric Communities). Organizational affiliations are given for identification purposes only. Participants’ views are their own, and do not necessarily reflect the views of the companies listed.

This paper integrates some highlights of that meeting with some of my other testing experiences.