So…you are responsible for the computer security of your organization. You probably have many great ideas on how to do this. You start looking around for products and services to implement those plans of yours and figure out quickly there are no commercial solutions that fit into your budget. Now what do you do?
Enter Metrics—the solution to all your problems (well, not really, but this is an article about Metrics). Back in 2008, I wrote an article published in the Educause magazine called Security Metrics: A Solution in Search of a Problem. Rather than rehash that article since you can read it yourself, I will start with the final tag line and proceed from there: “Metrics are the answer, now go and find your questions.”
One problem that metrics can help with is procuring resources to build your security program. Many organizations have either a small security department, or they view security as a “second job” for the members of the network group. I am confident that anyone involved in security, full or part time, is fully aware that, in many cases, we are losing the war. You will find yourself in trouble at some point unless your security program is growing and changing.
Security metrics will reveal the maturity of your security program. Here are the four steps to organizations process improvement, from James Harrington’s book The Improvement Process:
Measure: If you can’t measure it, you can’t understand it.
Understand: If you can’t understand it, you can’t control it.
Control: If you can’t control it, you can’t improve it.
Improve: The final step: Now go back to the first step—Measure.
Metrics or measurement is the first step needed to improve your security. There is a saying “if you can’t measure it, it didn’t happen.” I have a saying about security: “If it didn’t happen, you probably have a job tomorrow.” In either case, metrics will be the key to unlock the funds needed to improve your security program.
When starting off down the security road, the metrics that you collect will be what I refer to as “counting metrics” – they usually start with the idea of “how many”:
- We had 40 compromised systems last month.
- We had 25 compromised user accounts last week.
- We met with 200 users and did training last year.
These metrics are a great place to start. They will demonstrate that your security program is producing measureable results and allow you to grow your program by using them as justification for additional resources. For example:
- With better protection, we should be able to reduce the number of compromised systems.
- With multi factor authentication, we will be able to reduce the number of compromised accounts.
- With more people or an online training program, we should be able to train more people and make that training more effective.
I believe that all metrics that are collected should allow you to enhance the program being measured. I do not believe in using scare metrics. Stating that 500,000 spam messages were blocked is a useless piece of information since you really cannot do anything to change that number. The amount of spam (or viruses) that you receive is a function of the internet, not something that you have any control over.
In my mind, I measure the value of a metric by evaluating if it can be used to improve security. I like to think that we (the good guys) are at war with the bad guys. There is a phrase “OODA loop” that applies very well to using metrics to improve security:
“The phrase OODA loop refers to the decision cycle of observe, orient, decide, and act, developed by military strategist and United States Air Force Colonel John Boyd. Boyd applied the concept to the combat operations process, often at the strategic level in military operations. It is now also often applied to understand commercial operations and learning processes. The approach favors agility over raw power in dealing with human opponents in any endeavor.” (Source: Wikipedia)
I liken it to a thermostat: metrics measure the current security environment and tell the security team how the controls are working.
Once you have built your security program to the point where you are entering the measurement phase, an additional piece of the security puzzle is governance—the political structure of your organization, and to whom the security team reports. It is my opinion that many organizations have a broken governance structure. The security team reports somewhere in the IT organization and therefore is in direct competition with other IT functions for resources. This may mean that if there is a choice between fixing payroll and fixing security, you can bet that payroll will always win. From my perspective, security needs to report as high up in the organization as possible, preferably to the Board.
At Columbia University, we have a hybrid reporting structure; we work both for the VP of IT (CIO) and also report quarterly to the Board of Trustees. This means that security needs to produce metrics that will be of interest to the board as well as in a format that is meaningful.
You should make sure that the reports you create are designed for the audience that will be consuming them. As the presenter, you will need to know all the details behind the pictures. When a high-level view is needed, you should make sure that you are not including all the facts and figures.
As time goes on, your metrics should show that you are building a more secure environment. This should trigger a change to add a risk-based security component to your security program. Looking at risk-based metrics will show how your programs are preventing incidents, rather than counting the number of fires that you have extinguished.
Doing risk management in the IT world is complicated and time consuming. It involves knowing the value of information (data classification), where the data is being stored (servers, desktops, under peoples desks, etc.), how the data is being accessed (web or fat clients) and many other details that can be used to calculate a risk score.
Risk metrics will typically show assets and the associated risk score by group, often in the form of a heat map:
“A heat map is a two-dimensional representation of data in which values are represented by colors. A simple heat map provides an immediate visual summary of information. More elaborate heat maps allow the viewer to understand complex data sets.” (Source: Whatis.com)
In conclusion, I believe that metrics should be considered a tool for process improvement, not an end in themselves. Generating pretty pictures should not replace understanding the underlying facts that the pictures represent. Make sure that the metrics are a true and meaningful picture of the process being studied. Remember to always sanity check your findings. Only you can prevent metrics abuse.
About the author: Joel Rosenblatt is Director, Computer & Network Security at Columbia University. He has been working in IT for over 39 years, the last 17 in Computer security.