Evaluating IT Performance Metrics, or There's More to Life Than Page Loads
NetQoS Performance Experts

When it comes to IT performance metrics for web applications, the current metric of choice is page load time; that is, how long it takes to load an average page in a user's browser. For most webmasters, anything less than a 10-second page load is acceptable while anything over that mark is deemed too slow for prime time. It is probably a good idea for your web site to load pages in fewer than 10 seconds, but page load time is not the only way users experience your website. How about the impact of a poor design that forces users to navigate to 30 pages before placing something in a shopping basket? How many users will click away from the site even if those 30 pages load in fewer than 10 seconds?

If your application is not web based, you must still look at metrics. IT performance Metrics help you discover emerging trends, often early on, and can help you proactively respond to performance issues. How do you trust the effectiveness of the metrics you consult? How do you know which metrics to track? When you combine this confusion with office politics where different groups want to track different metrics for different purposes, it can be nearly impossible to find a simple set of meaningful metrics. This does not mean that you should not try.

IT performance metrics for an application can be one of the most quantitative, critical measurements you track. Today's applications are complex. Data moves from databases to websites through reporting, CRM tools, messaging servers, email servers, data warehouses, and other places. It can be difficult to track performance bottlenecks and spot issues. This is where IT performance metrics can really help you.

The problem with metrics is that the people interpreting them seem to boil them down into something they can understand. Page load time is easily understandable; therefore, we leap to that metric as a web server metric standard. If you write enterprise software, similar metrics emerge. In our quest to quantify things, metrics are easy prey, but they do not always paint the entire picture. Someone still needs to interpret the results.

To make the most of metrics, the results should depict the user experience in some way. In my experience, the most effective metrics are the ones that can repeatedly measure a task that a typical user would perform. I think it is a mistake to try to "boil down" metrics into one summary number. It is a better idea to use metrics in combination with other factors to rate the overall effectiveness of your applications. You might choose to combine metrics with usability for a powerful punch.

Online retailer Eddie Bauer, an established merchant with an emerging online presence, uses a combination of statistics, usability, and customer focus groups to help chart site contents, what should be on their site, and their site's future direction. Metrics give them a basis or foundation on which they can build a user-focused experience. The people behind the site still need to make the decisions based in part on data gathered from the metrics.

Another way to combine usability and metrics is to list 10 or 20 common tasks your users must complete to use your application and then generate metrics around those tasks. Logging in might be a task and a metric might be the time it takes the user to log in, create a session object, and download a cookie.

Metrics should be easy to gather and interpret for those trying to use them. You want metrics to work for you, not you work for metrics. If you find that you spend too much time tracking metrics and not enough time using them, you probably need to re-think your metrics. Try simplifying or consolidating metrics to come up with a set that you can live with, but one that still allows you to spot trends and issues before they occur.

Many large organizations are so enthralled with metrics that they form separate labs, divisions, or even corporate sites to generate, track, and measure performance metrics. Performance labs are great because they allow larger companies to focus on performance, acquire experts in the optimization field, and gather them to solve more complex problems. A smaller development shop tracking the right statistics while watching the correct trends can often trump the big company in the quest for better performance. One person with a stopwatch who says, "Hey, wait a minute." is often faster than a committee.

You might not have a fancy lab or division, but you can still put metrics to work for you. Even the best metrics are only as good as the way in which you interpret them. Establishing metrics that are easy to generate and interpret and that allow you to predict trends or emerging issues can help a development organization. Sometimes, even tracking the dreaded page load time in a simple spreadsheet can be a valuable tool in the quest for IT performance.

Read more about IT performance metrics in this white paper on the metrics that matter for performance management