Topic: Identifying performance bottlenecks

This comes from extrapolation of your test results and watching your transactions under load. For example, you can monitor a database to determine if queries are queuing up and which queries are doing so. Memory leaks can be found by monitoring the paging of the system as well as the actual memory being used and not freed up. (You can also check for things that use memory, like threads that do not die.)

There are many ways to do this. The way I like is the mathematical approach: use queuing theory. This allows you to apply various performance laws to the results of the system. For example, you can use the utilization law to determine the fraction of time that given resources are busy. Or, in the case of your database as just one example, use a forced flow law to show the average number of transactions/processes to visit a given queue (i.e., a database) over a given period of time.

You can also get a little more into it by looking at simultaneous resource possession but then you are getting into analytical-statistical models that are best used for capacity planning. These do, however, home in on the actual problem by helping to show conditions under burst traffic and heavy-tail distributions.

Rather than me sitting here boring you to tears with more detail, two excellent books on this subject:

Capacity Planning for Web Performance: Metrics, Models, and Methods

Scaling for E-Business: Technologies, Models, Performance, and Capacity Planning

Both are by Daniel A. Menasce and Virgilio A.F. Almeida. These have become my canonical references for all things performance related.