Maturity Models vs. Top 10 Lists

by jOHN on Monday, March 30, 2009

A few back, I wrote about Maturity Models vs. ASVS. On SC-L, a ‘discussion’ broke out regarding Maturity Models (MM) vs. Top N lists. Like ASVS, Top 10 lists target a different problem than MMs. In particular, the discussion focused around how one should enhance their assessment practices.

I’ve edited and reproduced my SC-L post here for those of you who don’t read the list. My objective? provide specifics on what one can do to enhance assessment practices. I provide a commentary on the on-going discussion throughout. The central theme of my approach?

security vulnerabilities don’t count unless they affect development

Now the approach:

Integrate Assessment Practices

[What?]
Wrap the assessment activities (both tool-based and manual techniques) in a process that:

  • Normalizes findings under a common reporting vocabulary and demonstrates impact
  • Include SAST, DAST, scanning, manual, out-sourced, & ALL findings producers in this framework
  • Anchors findings in either a developmental root cause or other software artifact:
  • Use Case, reqs, design, spec, etc.
  • Builds adaptors so that bugs are automatically entered in tracking systems
  • Adaptors should include both tool-based and manual findings
  • Calculates impact with an agreed-upon mechanism that rates security risk with other factors:
  • Functional release criteria
  • Other non-security non-functional requirements

[Realistic?]
I believe so. Cigital’s more junior consultants work on these very tasks, and they don’t require an early-adopter to fund or agree to them. There’s plenty of tooling out there to help with the adapters and plenty of presentations/papers on risk (www.riskanalys.is), normalizing findings (cwe.mitre.org/) , and assessment methodology (www.cigital.com/papers/download/j15bsi.pdf).

[Panacea?]
No. I’ve done research and consulting in functional testing. If you think security is powerless against development, try spending a few years in a tester’s shoes! Functionality may be king for development and PMs, but I’ve found that functional testing has little to no power. While a lack of features may prevent software from going out the door, very rarely do I find that functional testing can implement an effective “go/no-go” gate from their seat in the org. That’s why testing efforts seek muscle from their friend Security (and its distant cousins under quality “Load and Performance”) to stop releases from going out the door.

There’s no reason NOT to integrate with testing efforts, reporting, and groups: we should. There’s every reason security should adhere to the same interface everyone else does with developers (let them produce code and let them consume bugs)… I think the steps I outlined under ‘what’ bring us closer. I don’t think we need to (or can expect to) flip organizational precedent and behavior on its head to make progress.

[Steering]
The above scenario doesn’t allow explicitly for two key input/outputs from the software security ecosystem:

  1. Handling ultra-high-priority issues in real time
  2. Adjusting and evolving to changing threat landscapes

Establish a Steering Committee

[What?]
Establish a steering committee on which a software security, dev, architecture, operations, and corporate risk sit. These folk should manage the risk-model, scoring, security standards that drive the assessment verification standard, and the definition of both short-term and longer-term mitigating strategies. I’d argue that you’d like Industry representation too. That organization could come in written form (like the Top N lists) or in the form of consulting or a panel.

When incidents or firefights come into play, absolutely allow them to be handled out of band (albeit through a documented process), but! Not until they’ve been rated with the agreed-upon model.

[Realistic?]
Yes. I have several clients that use this structure. I speak with non-clients that do the same. Data gathering for scoring and prioritization is easy if you’ve done the steps in the previous section. The operations guys help you grade the pertinence of your efforts to what they’re seeing ‘in the wild’ too.

[Panacea?]
Does a steering committee help you respond with agility to a high-priority threat in real time? Not explicitly. But, it does help if your organizational stakeholders already have a working relationship and a mutual respect. Also: I think one root cause of the underlying discomfort (or dislike) with people’s perspectives on this thread has been:

“OK… you don’t like Top N lists… So what do you do?”

In my mind… The above answers that question.

[Assessment and Tools]
Do I believe that the normalized findings will emerge only from static analysis (or any other kind of vulnerability detection tool)? Absolutely not. People who follow my writings know I expect dramatic (ally high and low) numbers to be associated with tools.

Let’s summarize my data. Organizations can expect:

  1. Static analysis tools to account for 15-20% of their total findings, out of the box
  2. An initial false positive rate as high as 75-99% from a static analysis tool, without tuning
  3. Less than 09% code coverage (by even shallow coverage metrics) from pen-testing tools

Qualitatively, I can tell you that I expect an overwhelming majority of static analysis results produced in an organization to come from customization of their adopted product.

Simply: if you base your world view on only those things a tool (any tool) produces, you’re world view is too narrow and will prove ineffective. The same is true of those who narrow their scope to the OWASP Top-10 or the SANS Top 25.

[Top N Redux]
Some have left the impression that starting with a Top N list is of no use. Please don’t think I’m in this camp. In my last two public presentations I’ve indicated, “If you’re starting from scratch these lists (or lists intrinsically baked into a tool’s capabilities for detection) are a great place to start.” Yes, one of these presentations was entitled “Why Top N lists are bad” ;-) Also:

If you can’t afford frequent industry interaction-use Top N lists as a proxy for it. They’re valuable, but like anything, only to a point.

For me, this discussion will remain circular until we think about it in terms of measured, iterative organizational improvement. Why? Because when an organization focuses on getting beyond a “Top N” list it will just create their own organization-specific “Top N” list :-) If they’re smart though, they’ll call it a dash board and vie for a promotion ;-)

From the other side? People building Top N lists know they’re not a panacea, but also know that a lot of organizations simply can’t stomach the kind of emotional investment that BSIMM (and the ilk) come with.

This leaves me with the following:

[Conclusions]

  1. Top N lists are neither necessary nor sufficient for organization success
  2. Top N lists are necessary but not sufficient for industry success
  3. Maturity models are neither necessary nor sufficient for organizational success
  4. Maturity models are necessary but not sufficient for industry success
  5. Always avail yourself of what the industry produces;
  6. Never confine yourself to a single industry artifact dogmatically;
  7. Whatever you consume from industry, improve it by making it your own; and
  8. Where-ever you are in your journey, continue to improve iteratively.

[Related Perennial Rabbit Holes] (bonus)