When you want to know about performance of your application, you need to base the performance on the overall experience of the consumer. That means you must include all the time from the request being sent from the client until the response has been returned. Depending on your system, there may be many different process interactions, maybe on different servers or services, making up the complete round trip from request to response.
I recently had a conversation with a developer addressing just such an issue. They did not have resources allowing them to get data automatically captured in SQL server for query performance. Regardless, that only presented part of the picture when it came to total execution time. They came up with their own solution for monitoring. They created an interface and monitoring system, where any process could be monitored. The results were captured and serialized on a different thread to reduce the amount of overhead when performing the work.
This final product took some effort to implement. Now the entire application is instrumented with performance monitoring. The net result is that they capture performance statistics at all layers, and retain the data over time. They now mine that data on a regular basis, and are able to identify areas where performance is falling below previous trends, regardless of layer. They can also monitor performance from a complete user experience.
Using this tool they have been able to identify queries with poor performance. They also found areas where the application required a different approach. Sometimes there was not enough filtering criteria, and the system was returning more data than was necessary for the user to consume. They identified areas where data was static, and computing results at run time was not efficient, replacing the computed results with pre-computed data marts.
This just the initial value available from this monitoring implementation. There are other gold nuggets yet to be retrieved. You could identify if a server was not processing any load where you have load balancing. You could compare the performance of one server to another when they each perform the same application logic. You could simply go crazy with the data stored in this kind of monitoring.
What would you do with all of this data? Do you monitoring tools with the same kind of capabilities? Drop a comment with your thoughts.
Cheers,
Ben