My last post spawned some questions, which I responded to in turn. Here was my response:

[Adapters]
Adapters for assessment results can take a few forms, but let’s address three specific scenarios that fan-in to an assessment results/presentation step and a few that fan-out.

[Fan in]
Fan in typically comes from three sources: 1) static tools, 2) testing tools, and 3) manual analysis.

Adapters that deal with fan-in have three challenges to surmount:
A)Technically trans-code results from #1, #2, or #3 into a single tool’s format for roll-up, or a tool-independent alternative

B) Normalize results between #1, #2, and #3 so that ‘apples’ and ‘oranges’ get reported rather than ‘apples’ and ‘cars’

C) Code results with organization-specific “whathaveyou”

[Challenge A - Trans-code]
The good news about the tools is that they nearly all export to some XML format you can manipulate. This tackles output from #1 and #2. Organizations that adopted any tool early have struggled with keeping up with format changes but have indicated to me that they’re willing to pay this small price for the ability to ‘plug’ that many results into their developers’ bug-tracking systems.

A bigger challenge (effort and therefore cost-wise) is getting manual results (#3) into either the format driven by the tool responsible for reporting or that independent format I described. Smart consulting firms have built ‘emselves workflow to manage this. In the ’90’s we at Cigital used to pine for the report-generating automation of our then-competitor @Stake and waffle about whether our client-base was too disparate and our assessments too varied in scope to support a similar trick. I see no reason not to, within an organization, code up something that helps-technically-integrate your manual review findings with those found by tools.

[Challenge B - Normalize]
Manual reviewers report to me their discomfort with the rigidity of systems like “The Seven Kingdoms” and its competitors. Mind you, I don’t dislike these taxonomies, but people get crotchety and might complain if they’re forced to cram their findings down into a SAST or DAST report. They perceive, it would seem, the vulnerabilities they find to be like snowflakes ;-)

An independent format can be successful as well. One can roll their own (for their organization), or lean heavily on the community and go with something like Mitre’s CWE. At Cigital, we’re involved with CWE and follow-on work but have also helped people organize their own. A critical discussion of this trade off is possible, but beyond the scope of this email. That leads us into…The cost of normalizing results between #1 and #2 eclipses the technical challenges of building the adapter.

[Challenge C - 'Whathaveyou']
I would say this though: if your organization has progressed to the point where it possesses security standards (prescriptive or otherwise), they should absolutely be referenced by appropriate findings in the report. This goes for both violations of the standards and the applicability of standards that would serve to prevent or mitigate a particular risk.

References to policy/standards can be augmented with best practices, if your organization makes that distinction. I’ve seen organizations link to internal/external resources for training and/or further information as well. If you’re going to try to transition from “the bug hunt” to “building security in”, one great way is to provide developers immediately-available information as they’re making the mistake.

[Fan out]
Now, you have to ‘fan out’ into support for bug-tracking systems. Most organizations’ security groups have at least as much battle to fight here as a security consultancy, actually. Why? Because the organization hasn’t mandated a single development toolkit. The good news here is that while there may be more fan-out than there was fan-in, bug-tracking tools were built to be supplied data. Writing this portion of the adapter–a conduit between what normalized findings you’ve compiled and the offending team’s bug tracking system is fairly (technically) straightforward.

[Industry Std. 'Schema']
Sean Barnum has done a flotilla of work on this topic with Bob Martin at Mitre. Though it can be cumbersome or uninteresting to practitioners, I think the work they’re doing is important because whether its admitted or not, work on audit, testing, and verification methodologies/standards must implicitly take a stand on defining the words you listed (finding, root cause, vuln., etc.). Where such efforts take a stand, one’s organization can find alignment with their own notions quite challenging. Where it’s implicit (or worse, ambiguously and poorly defined) you get lots of wasted time as assessors argue with development over semantics, next steps, and responsibilities.

Each methodology has its own limitations in this department, resulting from its focus and perspective, IMO. If you look at OSSTM, there’s a wealth of definition around activities, which really helps those implementing it differentiate what techniques they could apply in testing their system. Their template reporting form falls short on defining constructs such as root cause and finding and ’speaks’ like an auditor’s report. This doesn’t do the depth and breadth of their assessing techniques justice which means, ultimately, adopting it will take a lot of work in the realm of that normalization task we treated earlier. NIST’s methodology formalized controls even more producing the 800-53 publication. I need to look at their recent foray into app sec and reconsider ASVS much more closely and for much longer to make judgments in this realm. Currently, I’ve only considered it in the insanely and unfairly narrow context of “a set of stuff to look for.” I’ll follow up with you on t!
his later this week or next.

[Correlating Risk Systems]
Taking your question literally: Risk systems? Most risk management companies wield Powerpoint and Excel, and as such, glue is hard to come by–let alone ‘open glue.’ I don’t have much experience with Archer, but their glue is proprietary but their suite includes the ability to weave together policy, requirements, findings, and change/bug management. It sits outside the MS Office stack, but what little experience I’ve had with it wasn’t necessarily positive ;-)

There are three new books (recently released) that are worth a look. Once is an absolute necessity for any security practitioner. The others may be interesting for some readers of the blog.

The book that you MUST READ RIGHT NOW is the second edition of Ross Anderson’s Security Engineering book. Ross did a complete pass on his classic tome and somehow made it even better. It also comes in handy as a weapon as it is so heavy. Books like Ross’s are a refreshing reality check from the usual pablum published in computer security.

Simply put, this is a must read book for every security professional. I don’t have my real copy yet from the publisher (but they say one is on the way), but I did take a close look through the manuscript. Ross retains his number one slot on my list of top 5 things every software security person should read.

Incidentally, I interviewed Ross for Silver Bullet last year (in April). Ross’s episode is the most popular of all 24 episodes released to date with over 18,000 downloads. You might want to give that a listen as well.

The other two books that are worth a look are Crimeware and The New School of Information Security. Lets cover them in reverse.

The New School of Information Security is a book worth buying for the cover alone. I know of no other computer security book with a Kandinski on the front. Even though I know Adam Shostack from way back (and never could have predicted that he would become a Microsoft guy), I saw his book at RSA, bought it for the cover, and only then discovered that he was the author! My plan was to give the book to a good friend who I know is a huge Kandinski fan. On the way to complete that errand, I had a chance to look though the book and now I need a copy of my own! If you’re a follower of the economics of security school (which Ross and Bruce Schneier have helped spearhead), you’ll like this book.

Crimeware is an academic tome written by my friend Markus Jakobsson. I contributed a chapter on software security bug taxonomy. My copy showed up last night, and I have earmarked more time to read it thoroughly. The enemy has changed over the last decade, and criminals are bringing the game to a new level.

Spring may not be the best reading time, but it does appear to be the best time for a crop of interesting new security books!

My previous post sparked comments from Mike Rothman, Alex, Christofer Hoff, Arthur, and perhaps others I haven’t seen. I sincerely appreciate everyone’s considered feedback. In this case, the feedback was to tell me I’m off-base on terminology, and that’s all good. I’m happy to take lumps when I mess something up.

I really meant it when I said, perhaps unclearly, that I was talking about the meaning of information security “risk management” as the activity occurs in most organizations today, not any textbook definition of risk management and certainly not any definition that uses words like “exact” or “right.”

Is that better definition and more precise activity a good North Star? Absolutely. Do you tell an organization that is just learning and applying a subjective skill that if they don’t do it perfectly, they’re doing it wrong? Do you tell an organization that is doing something that works for them that they’re doing it wrong because it doesn’t match the textbook? Not if you actually want to be helpful.

The crux of the following screed is that purist “risk management” and the information security “risk management” that is (not could or should be, but is) practiced in the day-to-day business world are currently distant cousins that may meet in the future. I also feel this is the perfect time to trot out the old Yogiism that, “In theory, theory and practice are the same. In practice, they’re not.”

Many of the fundamental information sources that organizations are turning to today give no prescriptive guidance at all on risk management. The information security drivers don’t help much either. By way of example, the PCI audit procedures recommend that reviewers “Verify that the information security policy includes an annual risk assessment process that identifies threats, vulnerabilities, and results in a formal risk assessment.” FFIEC guidance for financial institutions desires a “multidisciplinary and knowledge-based approach” that identifies “all reasonably foreseeable threats.” This gives organizations no guidance at all and, after they have caused problems, all threats are reasonably foreseeable.

In my opinion, the common definition of information security risk management in the world today is the set of activities that is preventing me from failing compliance and examiner audits, that is streamlining my security operations (for better or worse; I don’t know because I don’t know how to measure it), and that appears to be reason I am not suffering even more security breaches. Again, there are always enlightened organizations that are doing more and better, but they are the minority. There is no common yardstick, no well of enlightenment. If FFIEC, Basel II, SOX, PCI, and assorted other examiners and auditors see a recognizable security program that is producing good results, then by definition whatever these organizations are doing is good “risk management.” Likewise, if I’m not suffering breaches, then I have good “risk management” in the eyes of those whose opinion I really care about (e.g., people who buy my stuff, can fine me, or can put me in jail). Call that “information security” if you want (and so it is), but saying it’s not risk management is just po-tay-to, po-tah-to.

I have my understanding of the textbook and “perfect world” definitions of risk management. I usually cast risk management as the process that starts with risk assessment. This, of course, is a catch-all term for analyses of business goals, threats, assets, event frequencies, downsides, and so on. We also need information on vulnerability, risk appetite, internal and external mandates, and so on. Some good risk models applicable to the target environment help, too. Risk management as a process then works through decision-making and continues with activities directed at risk avoidance, risk reduction (whether through reduction of loss frequency or loss severity), risk transfer, and explicit risk retention. Lather, rinse, and repeat in an ongoing “it’s a journey, not a destination” sort of way.

“Proper risk analysis,” as Alex put it, “can’t mean unnecessary controls.” I interpret this the same as I would interpret that “proper” medical care can’t produce incorrect diagnoses and “proper” maintenance can’t ever damage a vehicle. In other words, there’s no such thing in any practical sense. Smart people trying to do the right thing with the best of intentions and the best of education mess these things up all the time. Even at Harvard, someone graduates last in each and every class.

“Proper” rarely happens in the field. Proper risk management costs too much and requires to many propeller-heads to make it go. Anyone doing “proper” (full, complete, accurate, perfect) risk management would know enough not to spend the money it takes to do “proper” risk management. While accurate conclusions are crucial, very few, if any, information security decisions require that much effort or that much precision. With all due respect, I’ll wait while that sinks in.

Yes, I suppose I am dealing with some aspect of “issue management,” to throw yet another term at those activities organizations engage in to ensure their networks and applications are appropriately secure. The “issue” is “I don’t want to overspend, but I also don’t want to be the last one to do some security thing because I don’t want to explain to a jury why I’m the only organization not doing something that might have prevented a breach.” Further, the “risk tolerance of the data owner” is often of little consequence since data aggregation that increases sensitivity, public perception, and a variety of other business problems often take “risk” decisions out of the data owner’s hands.

With respect to “If your definition of risk is correct, and if your analysis is good, then all that is left is for the decision maker to figure out how willing they are to lose money,” this doesn’t happen. Even with terabytes of actuarial data, insurance companies can’t make perfect longevity predictions and odds-makers can’t reliably predict the Super Bowl winner in October. And these are really smart people, with really cool technology, with millions of dollars in backing, and the opportunity to make billions off a good decision. In other words, they’re *really* motivated. Does anyone really believe the average CIO and CISO stand a chance of having a “correct” risk definition and “good” analysis?

As to the idea that you can’t overspend because “the data owner expends only the amount of resources they are willing to allocate to reduce possibilities to their acceptable level,” well, again, I’m stymied. There’s no response for such Pollyanna thinking. It’s like a having a doctor from a world-class hospital telling a M*A*S*H surgeon to just “do it clean, quiet environment and it’ll be okay.” For example, what if the data owner postulated above is willing to spend $1B on a $1M problem? Exactly.

Activities correlating to “done correctly” and “can’t overspend” and “perfect data” and so on only happen in textbooks and fabricated business school case studies.

Information security risk management in the field is an ugly business. It’s not a matter of “when done properly” — it isn’t. It uses gut instinct for fuel and eats academics for lunch. Sometimes the process is too ethereal and we get bad decisions and too few controls. Sometimes the process is too grounded in activity and we get bad decisions and too many controls instead of more thoughtful analysis. Sometimes, bad things like public data breaches never happen even to the goofiest organizations and, sometimes, bad things seem to happen over and over even to organizations with thoughtful, resourced, and active risk management processes. It happens.

To address Hoff’s contention, I’m not “making excuses” for it; I’m documenting it. Clearly information security and risk management are not the same thing. And, to a geologist, sculptor, or someone refinishing a kitchen, marble and granite are also very different things. To the average person needing to prop up a wobbly business process, however, they’re both just rocks.

“Risk aware” organizations do spend better, but they still end up buying the same basic information security technologies as everyone else because that’s all there is. Where I see them spending more pragmatically (the perfect word here) is in training and in configuration and use of technology, which is truly a good thing. These are everyday blocking and tackling activities at most organizations. Where risk awareness truly benefits an organization is in creative areas such as software development. Thoughtfully applied security in the SDLC of an application-driven organization (think ISV or large brokerage) will result in much more overall net goodness than another internal firewall or blocking ZIP files at the Internet boundary.

“True risk management” is indeed a wonderful tool and I will always push organizations in that direction. The progression from raw art to even a glimmer of science is still a long road for information security risk management.

As to a “proper” definition of “risk management,” I spent years bemoaning the change in the definition of “hacker.” It was once a really useful word that described a true artist with code and equipment. Now, it describes any many manner of computer-based criminal activity. I wish you better luck preserving a classic definition of “risk management.”

Technorati Tags: risk management

IT controls. Corporate governance. Decision support. Right-sized spending (another phrase I thought I coined, but I see it gets three hits in Google). These are all part of the all-too-nebulous activity often referred to as data security risk management.

Let’s put a stake in the ground on what risk management means. I’m not referring to how it’s defined so much as what it actually means to business. Risk management means there is a thought process that includes ensuring the right people with adequate skills are given useful information and actually decide whether to do this or that to more effectively achieve security goals. Something like, “The available data indicate that path A at price B mitigates problems C, D, and E, but causes problem F, while path Z at price Y, mitigates problems C, E, and X, but causes problem W. What’s your decision?” Or, as happens much, much too often: “Hackers, phishers, and insiders…oh my, what are we going to do?”

Everyone makes dozens, hundreds, or perhaps thousands of risk management decisions every day. These are parents, doctors, lawyers, truck drivers, and everyone else, including CEOs, CIOs, IT security personnel, software architects, developers, and so on. Some people have good gut instincts, shoot from the hip, and end up with decisions that only occasionally burst into flames. For my risk appetite, that’s too little risk management. Others wait for every possible scrap of data, agonize over the possibilities, and end up with decisions that only occasionally aren’t completely overcome by events. That’s too much risk management.

The impact of too little risk management is usually too few security controls and, therefore, too much unpredicted expense in a variety of areas: incident response, litigation, and recovery, to name a few. These are often the result of public things that can have lasting effects on brand. This is easy to understand.

The impact of too much risk management is usually too many security controls and, therefore, too much predicted expense in a variety of areas: hardware, software, tools, people, processes, and so on. These are all internal things that can setiously impair agility, efficiency, and overhead, and this is usually much harder to understand.

Which is worse?

Let me clarify that I’m being a little fast and loose with “too much risk management” above. In my experience, the problem is almost never too much “risk management,” it’s almost always too much security fabric resulting from a fixation on or over-thinking of each and every security issue, whether applicable or not, combined with a natural tendency to equate activity with progress. As a consultant, I’ve heard some form of the following dialog hundreds of times: “What are we doing about the security problem I’ve heard about?” followed by a confident “We have people choosing A as we speak.” More security controls, especially generic plug-n-play things, does not automatically mean less risk, but it sure is highly demonstrable activity (to managers, to auditors, to examiners).

I think we’ve seen the impact of too few controls all too vividly recently. Just type “data breaches” into your favorite search engine and you’ll be inundated with examples. And that’s just one example of one kind of bad outcome associated with too few security controls.

But, too many controls can also have a direct impact on the very core of an organization. To illustrate, let’s turn to the great triad of operational excellence, product leadership, and customer intimacy.

A company whose success is based on operational excellence requires quality goods at reasonable prices. This requires very efficient operations, excellent supply chain management, and practically flawless execution. Superfluous security controls, whether in people, process, technology or embedded in software, may not prevent flawless execution, but it certainly seems like a good way to slow down a just-in-time supply chain (e.g., we quarantine all email attachments for 12 hours) and disrupt operations (e.g., by implementing ultra-fine-grained entitlements where it just isn’t necessary).

When your success is based on product leadership, you are probably innovating constantly and spending like mad to build brand. Staying ahead of everyone else requires great design and development, a fantastic knowledge of your customers, and quick time to market. Security controls that have no material relationship to any practical attack can easily derail any product process that relies on agility (e.g., SDLC security tools and processes that check for things that have no matching threat slowing down product cycles).

If customer intimacy is your key to success, then you are likely to be extraordinarily focused on customer service, with lots of choices that can be tailored to customer desires. Here you will have to excel in customer management, have product and service delivery that exceeds expectations, and engender client trust. Too few security controls (”Hey, I’m this person you’ve never met, please reset my password.” “Sure thing!”) can certainly end a client relationship, but so can unnecessary security controls (e.g., three personal questions, identify an image, and last four of your SSN just to activate the Forgot My Password page) as it also clearly indicates that you just don’t know what you’re doing. Where it’s obvious that you don’t understand the appropriate intersection between your risk management, your customer’s risk appetite, and the actual risk, a trusted partnership is pretty much out of the question.

All in all, too few security controls is probably the greater of the two evils. On the other hand, it’s probably the easiest to remedy. Even if you do no risk management at all, if you have the money to purchase and correctly install most of the major security technologies out there, the shotgun approach will in fact reduce security risk. You’ll never know if you’ve done enough or if you’ve overspent, but you’ll have a story to tell to the masses. On the other hand, a thoughtful security approach based on sound risk management will give you a story to tell to savvy customers and increasingly well-educated auditors and examiners.

Technorati Tags: risk management

As I get back into the risk management arena after a sojourn in knowledge management (mainly designing knowledge-driven offerings and monetizing the associated intellectual property), I find yet another example of “the more things change, the more they stay the same.” I think the executive view of information security risk management techniques as viable decision support tools has come a long way, but the complaint I hear from most executives is that even the simplest risk modeling appears to take a very long time.

I find that information security risk management still has the unfortunate combination of being a lightning rod for snake oil in the marketplace, being a real polarizing and dogmatic topic that often defies reasoned discourse, and being something that the average person handles by gut and not by numbers. Practitioners, academics, and managers alike bemoan the scarcity of actuarial data on which to base decisions. And, just to round out the issues list, any given group of people will quickly fall into its own set of terms and meanings, making it very difficult for any two groups to have short, interesting arguments that actually advance the knowledge base.

In my recent reviews of what’s going on in the world, risk modeling exercises related to application security seem to stretch on for two primary reasons:

1. An obsession with knowing every “threat”
2. Not having a good rule for deciding when a threat-vulnerability-control coupling deserves no more scrutiny

What I’ve evolved over the past couple of decades to reduce this work is something I’ve called “Looking for zeros” and “Looking for ones.”

In my experience, knowing the exact threat (i.e., combination of attacker, attack, attack path, resources, and some intangible things such as motive) is often irrelevant. I call this “Looking for ones.” For example, if a particular attack always works (e.g., cross-site scripting in a particular web form), then it likely matters not whether the attacker is a national government, a terrorist, a criminal, a script kiddie, or someone who accidentally pastes HTML into the field — the “success value” for this attack-vulnerability tuple will always be ‘1′. Knowledge of the attacker might give us a bit more information about what he or she might be ultimately trying to accomplish, but the decision to fix the problem should already have been made.

Similarly, this is why we decompose threat, vulnerability, assets, and other things into constituent components. I call this part “Looking for zeros” and it works really well. As soon we find a ‘0′ for a material aspect of attacker, attack, attack path, resources, motivation, threat frequency, feasibility, accessibility, susceptibility, vulnerability prevalence, asset value, event cost, downside, caring, and so on for the various dimensions of “bad stuff happens,” then that scenario is over and we can move on. Again, even given a national government with lots of resources and a zero-day electronic attack that really works, we just don’t care if the attack works only on WebServerA and we’ve deployed WebServerB, or if the attack path required (e.g., site must be using JSessionID), or whatever else, simply doesn’t apply to us.

When we bring into the mix controls (for which we reason about preventive, detective, corrective, and deterrent values), then we try to kill off entire attacks (e.g., timeouts that prevents script kiddies doing half-open TCP stuff), attack paths (e.g., Internet routers that allow incoming packets with RFC 1918 addresses), motivations (e.g., dye packs that render stolen cash unspendable), and so on all at once; seldom should the IT or software development world try to work off “threats,” and specifically threat agents (attackers), one at a time. That just takes too long.

A comprehensive and holistic view of what makes “risk” happen will always make risk management much more efficient and make the results of risk management much more usable in day-to-day decision support.

Technorati Tags: risk management, threat modeling