Topic: How to Write a Helpful Bug Report

Imagine a customer telling you that your product is not working. You would probably have many questions:

    * How serious is the situation?
    * Is there a defect in your product, or is the defect in another part of the customer system?
    * Is the issue already known?
    * Is a solution already available?

Now imagine thousands of such incidents from around the world. How would you analyze these incidents as quickly, efficiently, and accurately as possible? After all, each report is a chance to make the product better for everyone.

This scenario is the challenge that the WebBugs team faces every day. Ideally, the team would be able to investigate every bug report in person. But this approach is not realistic, so we have designed a bug-reporting site to help us gather the information we need.

Our goal is to assist developers in creating a report that is complete and compact. A complete report allows our team's developers to reproduce the problem locally. A compact report helps us focus the investigation so that we can quickly determine the cause of the behavior.

Let's tour the bug-reporting site to see how we pursue this goal. And note that this document applies only to bugs. It does not discuss the trail for reporting documentation issues and requests for enhancements, which are other options on the bug-reporting site.
Report Elements

There are four basic parts of a report:

   1. Overview
   2. Routing information
   3. Test case
   4. Contact information

Overview

The overview provides us with a background for understanding the issue, as Table 1 indicates.
Table 1: Overview of a Bug Report
Synopsis
   
This becomes the report's name. A good synopsis is a unique identifier that offers a rough idea of the issue.
Description
   
This is a framework for understanding the problem and the test case, which you will provide later in the report. If you wish, you may include the cause of the problem and a proposed solution. Write as short a description as possible. Use a professional tone and avoid the impulse to vent your frustration. Venting is likely to distract and put off the report analyst.
 

Routing Information

Routing fields are the rudders that direct the report to the proper engineering group. Although we will try to correct any mistakes, filing an incorrectly classified bug report may cause a delay in the response.

Table 2 provides more detail.
Table 2: Routing Information in a Bug Report
Category and subcategory
   
Fill in the specific component that is exhibiting the problem. Choosing the proper category and subcategory will minimize any delay in routing the report.
Version
   
This is the exact release of the affected product. We ask for version information twice: The first provides a general sense of the release, and the second offers details on the exact release and build. Try to use the most up-to-date version. Sun, like most engineering organizations, focuses on the current and upcoming releases.
Regression version
   
If the bug is a regression, we want to know in what release the behavior worked before. This information helps to narrow the search for when and where the code changed.
Severity
   
This field indicates the importance of the issue. Note that for high-severity items, the submitter should seek community or paid support, because this reporting channel is not designed to help with individual cases.
 

Test Case

The test case is the meat of the report. An error trace, although helpful, does not provides the richness of information available from a test case. Simply put, if we cannot reproduce the problem on our machines, we will have difficulty locating the cause of the failure and developing a solution.

As long as a report is complete, less is more. We tend to review shorter reports first, because our time is limited. Your spending a little time to make a report more concise often results in a faster response time.

Table 3 provides more detail.
Table 3: The Test Case in a Bug Report
Operating system and hardware
   
The platform (operating system and hardware) indicates which system will reproduce the problem. If the behavior occurs on more than one platform, pick one to remove any ambiguity.
Source code for an executable test case
   
The source code should be a stand-alone application that can compile without any external libraries. Again, shorter is better: Anything longer than 20 lines makes it more difficult for engineers to find the actual cause and increases the probability that the problem is actually in the bug submitter's code. Although extracting a small test case from a gigantic deployed application is a challenge, submit as short a test case as possible. This exercise may reveal that the problem is in one's own code, which means that a solution is immediately available.
Steps to reproduce
   
Include the command-line invocation, even if it seems obvious.
Expected result
   
The expected behavior tells us how the test case should behave if no bug were present. If the test case produces this expected result, we know that a fix works.
Actual result
   
The actual behavior tells us the current results of the test case. If Sun engineering does not see this exact behavior, then the system configuration may be causing the problem.
 

Contact Information

The email address allows us to communicate with the report submitter in case we want to clarify anything or send an update. The other information helps us to be more intelligent and polite in our responses.

Table 4 provides more detail.
Table 4: Contact Information in a Bug Report
Your name
   
This field tells us how to politely address the submitter.
Company
   
The company name provides us with more background about the submitter.
Email
   
An email address is critical because it is the only way to contact the submitter. The willingness to communicate indicates the submitter's seriousness, so we delete bug reports in which the submitter has not provided a valid email address.
Sun online account ID
   
This information identifies submitters who are part of the Sun Developer Network.
 
Examples

Here are links to two reports for the same issue: imprecision when multiplying two floating point numbers. Note how the second version makes analysis much easier.

    * This long report is vague yet complicated. In addition, the submitter is rude. What reasonable engineer would spend time sifting through the report?
    * This model report is excellent. Because it is complete and compact, identifying the ultimate fault will be easier.

By the way, the imprecision that both reports point to is expected. The Java programming language, like most programming languages, uses binary numbers to represent our base-10 numbering system. As a result, standard floating-point arithmetic in the Java language is not exact. For precise arithmetic, see the java.math package.
Conclusion

Writing a complete, compact report takes skill and craftsmanship. We hope that our discussion of report elements and sample reports has made this task easier for you. The resulting improvements in reports will mean faster fixes and a higher-quality final product. Ultimately, we all will benefit.



Edited by moderator - 06May2007 at 11:02pm