My last post spawned some questions, which I responded to in turn. Here was my response:

[Adapters]
Adapters for assessment results can take a few forms, but let’s address three specific scenarios that fan-in to an assessment results/presentation step and a few that fan-out.

[Fan in]
Fan in typically comes from three sources: 1) static tools, 2) testing tools, and 3) manual analysis.

Adapters that deal with fan-in have three challenges to surmount:
A)Technically trans-code results from #1, #2, or #3 into a single tool’s format for roll-up, or a tool-independent alternative

B) Normalize results between #1, #2, and #3 so that ‘apples’ and ‘oranges’ get reported rather than ‘apples’ and ‘cars’

C) Code results with organization-specific “whathaveyou”

[Challenge A - Trans-code]
The good news about the tools is that they nearly all export to some XML format you can manipulate. This tackles output from #1 and #2. Organizations that adopted any tool early have struggled with keeping up with format changes but have indicated to me that they’re willing to pay this small price for the ability to ‘plug’ that many results into their developers’ bug-tracking systems.

A bigger challenge (effort and therefore cost-wise) is getting manual results (#3) into either the format driven by the tool responsible for reporting or that independent format I described. Smart consulting firms have built ‘emselves workflow to manage this. In the ’90’s we at Cigital used to pine for the report-generating automation of our then-competitor @Stake and waffle about whether our client-base was too disparate and our assessments too varied in scope to support a similar trick. I see no reason not to, within an organization, code up something that helps-technically-integrate your manual review findings with those found by tools.

[Challenge B - Normalize]
Manual reviewers report to me their discomfort with the rigidity of systems like “The Seven Kingdoms” and its competitors. Mind you, I don’t dislike these taxonomies, but people get crotchety and might complain if they’re forced to cram their findings down into a SAST or DAST report. They perceive, it would seem, the vulnerabilities they find to be like snowflakes ;-)

An independent format can be successful as well. One can roll their own (for their organization), or lean heavily on the community and go with something like Mitre’s CWE. At Cigital, we’re involved with CWE and follow-on work but have also helped people organize their own. A critical discussion of this trade off is possible, but beyond the scope of this email. That leads us into…The cost of normalizing results between #1 and #2 eclipses the technical challenges of building the adapter.

[Challenge C - 'Whathaveyou']
I would say this though: if your organization has progressed to the point where it possesses security standards (prescriptive or otherwise), they should absolutely be referenced by appropriate findings in the report. This goes for both violations of the standards and the applicability of standards that would serve to prevent or mitigate a particular risk.

References to policy/standards can be augmented with best practices, if your organization makes that distinction. I’ve seen organizations link to internal/external resources for training and/or further information as well. If you’re going to try to transition from “the bug hunt” to “building security in”, one great way is to provide developers immediately-available information as they’re making the mistake.

[Fan out]
Now, you have to ‘fan out’ into support for bug-tracking systems. Most organizations’ security groups have at least as much battle to fight here as a security consultancy, actually. Why? Because the organization hasn’t mandated a single development toolkit. The good news here is that while there may be more fan-out than there was fan-in, bug-tracking tools were built to be supplied data. Writing this portion of the adapter–a conduit between what normalized findings you’ve compiled and the offending team’s bug tracking system is fairly (technically) straightforward.

[Industry Std. 'Schema']
Sean Barnum has done a flotilla of work on this topic with Bob Martin at Mitre. Though it can be cumbersome or uninteresting to practitioners, I think the work they’re doing is important because whether its admitted or not, work on audit, testing, and verification methodologies/standards must implicitly take a stand on defining the words you listed (finding, root cause, vuln., etc.). Where such efforts take a stand, one’s organization can find alignment with their own notions quite challenging. Where it’s implicit (or worse, ambiguously and poorly defined) you get lots of wasted time as assessors argue with development over semantics, next steps, and responsibilities.

Each methodology has its own limitations in this department, resulting from its focus and perspective, IMO. If you look at OSSTM, there’s a wealth of definition around activities, which really helps those implementing it differentiate what techniques they could apply in testing their system. Their template reporting form falls short on defining constructs such as root cause and finding and ’speaks’ like an auditor’s report. This doesn’t do the depth and breadth of their assessing techniques justice which means, ultimately, adopting it will take a lot of work in the realm of that normalization task we treated earlier. NIST’s methodology formalized controls even more producing the 800-53 publication. I need to look at their recent foray into app sec and reconsider ASVS much more closely and for much longer to make judgments in this realm. Currently, I’ve only considered it in the insanely and unfairly narrow context of “a set of stuff to look for.” I’ll follow up with you on t!
his later this week or next.

[Correlating Risk Systems]
Taking your question literally: Risk systems? Most risk management companies wield Powerpoint and Excel, and as such, glue is hard to come by–let alone ‘open glue.’ I don’t have much experience with Archer, but their glue is proprietary but their suite includes the ability to weave together policy, requirements, findings, and change/bug management. It sits outside the MS Office stack, but what little experience I’ve had with it wasn’t necessarily positive ;-)

A few back, I wrote about Maturity Models vs. ASVS. On SC-L, a ‘discussion’ broke out regarding Maturity Models (MM) vs. Top N lists. Like ASVS, Top 10 lists target a different problem than MMs. In particular, the discussion focused around how one should enhance their assessment practices.

I’ve edited and reproduced my SC-L post here for those of you who don’t read the list. My objective? provide specifics on what one can do to enhance assessment practices. I provide a commentary on the on-going discussion throughout. The central theme of my approach?

security vulnerabilities don’t count unless they affect development

Now the approach:

Integrate Assessment Practices

[What?]
Wrap the assessment activities (both tool-based and manual techniques) in a process that:

• Normalizes findings under a common reporting vocabulary and demonstrates impact
• Include SAST, DAST, scanning, manual, out-sourced, & ALL findings producers in this framework
• Anchors findings in either a developmental root cause or other software artifact:
• Use Case, reqs, design, spec, etc.
• Builds adaptors so that bugs are automatically entered in tracking systems
• Adaptors should include both tool-based and manual findings
• Calculates impact with an agreed-upon mechanism that rates security risk with other factors:
• Functional release criteria
• Other non-security non-functional requirements

[Realistic?]
I believe so. Cigital’s more junior consultants work on these very tasks, and they don’t require an early-adopter to fund or agree to them. There’s plenty of tooling out there to help with the adapters and plenty of presentations/papers on risk (www.riskanalys.is), normalizing findings (cwe.mitre.org/) , and assessment methodology (www.cigital.com/papers/download/j15bsi.pdf).

[Panacea?]
No. I’ve done research and consulting in functional testing. If you think security is powerless against development, try spending a few years in a tester’s shoes! Functionality may be king for development and PMs, but I’ve found that functional testing has little to no power. While a lack of features may prevent software from going out the door, very rarely do I find that functional testing can implement an effective “go/no-go” gate from their seat in the org. That’s why testing efforts seek muscle from their friend Security (and its distant cousins under quality “Load and Performance”) to stop releases from going out the door.

There’s no reason NOT to integrate with testing efforts, reporting, and groups: we should. There’s every reason security should adhere to the same interface everyone else does with developers (let them produce code and let them consume bugs)… I think the steps I outlined under ‘what’ bring us closer. I don’t think we need to (or can expect to) flip organizational precedent and behavior on its head to make progress.

[Steering]
The above scenario doesn’t allow explicitly for two key input/outputs from the software security ecosystem:

1. Handling ultra-high-priority issues in real time
2. Adjusting and evolving to changing threat landscapes

Establish a Steering Committee

[What?]
Establish a steering committee on which a software security, dev, architecture, operations, and corporate risk sit. These folk should manage the risk-model, scoring, security standards that drive the assessment verification standard, and the definition of both short-term and longer-term mitigating strategies. I’d argue that you’d like Industry representation too. That organization could come in written form (like the Top N lists) or in the form of consulting or a panel.

When incidents or firefights come into play, absolutely allow them to be handled out of band (albeit through a documented process), but! Not until they’ve been rated with the agreed-upon model.

[Realistic?]
Yes. I have several clients that use this structure. I speak with non-clients that do the same. Data gathering for scoring and prioritization is easy if you’ve done the steps in the previous section. The operations guys help you grade the pertinence of your efforts to what they’re seeing ‘in the wild’ too.

[Panacea?]
Does a steering committee help you respond with agility to a high-priority threat in real time? Not explicitly. But, it does help if your organizational stakeholders already have a working relationship and a mutual respect. Also: I think one root cause of the underlying discomfort (or dislike) with people’s perspectives on this thread has been:

“OK… you don’t like Top N lists… So what do you do?”

In my mind… The above answers that question.

[Assessment and Tools]
Do I believe that the normalized findings will emerge only from static analysis (or any other kind of vulnerability detection tool)? Absolutely not. People who follow my writings know I expect dramatic (ally high and low) numbers to be associated with tools.

Let’s summarize my data. Organizations can expect:

1. Static analysis tools to account for 15-20% of their total findings, out of the box
2. An initial false positive rate as high as 75-99% from a static analysis tool, without tuning
3. Less than 09% code coverage (by even shallow coverage metrics) from pen-testing tools

Qualitatively, I can tell you that I expect an overwhelming majority of static analysis results produced in an organization to come from customization of their adopted product.

Simply: if you base your world view on only those things a tool (any tool) produces, you’re world view is too narrow and will prove ineffective. The same is true of those who narrow their scope to the OWASP Top-10 or the SANS Top 25.

[Top N Redux]
Some have left the impression that starting with a Top N list is of no use. Please don’t think I’m in this camp. In my last two public presentations I’ve indicated, “If you’re starting from scratch these lists (or lists intrinsically baked into a tool’s capabilities for detection) are a great place to start.” Yes, one of these presentations was entitled “Why Top N lists are bad” ;-) Also:

If you can’t afford frequent industry interaction-use Top N lists as a proxy for it. They’re valuable, but like anything, only to a point.

For me, this discussion will remain circular until we think about it in terms of measured, iterative organizational improvement. Why? Because when an organization focuses on getting beyond a “Top N” list it will just create their own organization-specific “Top N” list :-) If they’re smart though, they’ll call it a dash board and vie for a promotion ;-)

From the other side? People building Top N lists know they’re not a panacea, but also know that a lot of organizations simply can’t stomach the kind of emotional investment that BSIMM (and the ilk) come with.

This leaves me with the following:

[Conclusions]

1. Top N lists are neither necessary nor sufficient for organization success
2. Top N lists are necessary but not sufficient for industry success
3. Maturity models are neither necessary nor sufficient for organizational success
4. Maturity models are necessary but not sufficient for industry success
5. Always avail yourself of what the industry produces;
6. Never confine yourself to a single industry artifact dogmatically;
7. Whatever you consume from industry, improve it by making it your own; and
8. Where-ever you are in your journey, continue to improve iteratively.

[Related Perennial Rabbit Holes] (bonus)

• Bugs vs. Flaws: John Steven’06 – www.mail-archive.com/sc-l@securecoding.org/msg00888.html
• Security Vs. Quality: Cowan ‘02 – www.securityfocus.com/archive/98/304766

Warning: a fair amount of cynicism occurs in this post.

Some of my buddies have been exchanging ideas of what keeps us interested and one friend was thinking about how he could use a user’s Facebook login on his site. This nudge along with some work I’m doing with federated identity and Amazon SSO all have brought this federated identity stuff onto the foreground thread in my brain.

It’s all very interesting stuff and I think there’s some great technology behind all of this. I’m not worried about the technology part. It’s whether the technology can ever get implemented that worries me.

Why? Well on the internet audience and the audience demographics are the currency of the realm. If there’s federated identity, then providing all of my identity information to the relying part is redundant. There’s no way marketing is going to let THEIR site be the relying party. Marketing will want THEIR site to be the IdP. That’s because they want the users to sign up and provide all of the contact and demographic information since that’s the only business model that has been proven to work last time I checked.

I can imagine a conversation going like this:

Me: We should implement federated identity so our users don’t have to log in a gazillion times.
Marketing: Good idea.
Me: Whose identity should we use? LiveID? Amazon?
Marketing: Huh? What do you mean? Ours of course. We need the user to sign up to give us their email address.
Me: Well, we can get that. It’s part of the claim that we’ll get as part of SAML.
Marketing: Sam who? When does the user give us his email?
Me: They don’t give us the email directly. They give it to the identity provider and then…
Marketing: No, no, no, no (just like your mom used to do) – this doesn’t sound like a good idea…

So maybe all we really need is an identity selector and we’ll be the digital equivalents of the janitor with the massive key ring on our belts.

Technorati Tags: federated identity, cloud

The Geekonomics blog is often good. A new post indicates Apple’s veneer of more secure than Microsoft is cracking.

It was only a matter of time. I wanted to clarify that though you see a lot of security consultants carrying Macs, in Cigital’s case, it’s not an endorsement. Again, in the interest of disclosure: though I own and operate many platforms I operate more OS X at home and office than the others.

I attribute Mac adoption amongst security folk to two reasons: the platform combines a Unix-like environment with the ability to interact through email and MS Office and that the machines, for the most part, are a snappy bit of hardware, cobbled together into a “shiny object” (desirable) form. This second aspect goes a long way to explain the recent jump in ownership in the security community: “fan-boys.”

Any claims that it’s because “they’re more secure” should be considered with a fair amount of skepticism.

Yes, historically, the platform has suffered less pain of viruses and malware. Yes, certain aspects of their OS/platform and design did make improvements over XP.

The truth is this:

Apple doesn’t “Build Security In” very well at all. You don’t have to be an insider to understand why. Redmond is exporting security blogs, books, and value like never before. You don’t see a lot of Apple security people in the community though. You don’t see good solid standards-based support for authentication or web-services that would help you interact securely with your enterprise (Apple hides behind their ‘vision’ on this one). You don’t see a lot of support for Objective-C in the static analysis tool realm.

I believe that though Apple paid great lip service to security as a differentiator initially, (they even talked about phones like the iPhone becoming the basis of identity moving forward) but yet they abandoned it when they realized the cost of a real enterprise-level program. They also abandoned what was probably their best protection: the PPC processor. Their security proposition, IMO, is based on obscurity.

I’ve said before, “Woe to Apple when market share (and thus Economics) garners the attention necessary to motivate attackers to focus on the platform—attackers may find their task easier than with Vista.”

Ben Worthen broke the BSIMM story on wsj.com as was posted earlier.

I was shocked when someone said, “Oh and ASVS is also available, great” on an OWASP list. Super, I thought, but I don’t understand the connection. When I looked at the WSJ site, I noticed Jim Manico (of OWASP, Aspect, and ASVS fame) wrote, “But for those of your programming web applications, consider looking at http://www.owasp.org/index.php/ASVS – it is focused specifically on web application security evaluation.” I know Jim, he’s a good guy, and my curiosity about where the link between maturity models and verification standards was sated, but I thought I’d spend some time here quickly disambiguating them:

Verification standards, like ASVS, enumerate techniques with which an application’s correct use of security controls can be verified. It also posits (what it calls) verification requirements that serve as specifically enumerated tests of particular security controls (such as input validation). Such a standard is best operationally deployed by an organization’s application security group. In some cases, the audit group may own verification and reporting.

Auditors especially love NIST’s 800-53 standard, which recommends security controls for Federal Information Systems and Organizations. This document has frustrated me historically because of its focus on OS configuration, topology, and patch level. Largely, it ignores the application layer of an organizational stack. As such, organizations applying it have found little impact in preventing the web’s most common application security vulnerabilities. I reckon one aim of ASVS was closing this particular gap.

Regardless of what you think of ASVS or NIST 800-53, one applies the bulk of their guidance to an application and systems.

A maturity model such as BSIMM measures what an organization is doing to secure their software. Organizations will be graded on whether they’ve defined and adopted assessment (verification) techniques, but also as to the state of other organizational constructs such as a training program, post-deployment incident response capabilities, and specific application security management. A maturity model helps organizations place their personnel, use of tools, and practices against industry best practices in a broader context (a software security framework). Verification of applications is only a small piece of that governance activity. As such, BSIMM data is produced, consumed, and managed at the CISO/CIO level, rather than within the application security group (as ASVS).

Organizations will need an ability to consistently and comprehensively verify the security of their applications but this is only a piece of what those same organizations will also need to do, in a more broad context, to make sure they reduce, other-wise manage, or transfer software-induced business risk.

To this extent, comparison of BSIMM to ASVS is even a poor fit than comparing the Rational Unified Process (RUP) to Capability Maturity Model (CMM). -shiver-

The first phase in our endeavor to bring some science to software security is at a close. Our science-y approach started with some anthropology several months ago. We asked nine firms to tell us about their software security group (SSG), its inception, its activities, and the success it has achieved. The result is the Building Security In Maturity Model authored by Gary McGraw, Brian Chess (Fortify), and me, which is out for public use at http://bsi-mm.com. We’re very pleased that the Wall Street Journal broke the news of the BSIMM release.

Please take a look at BSIMM. If you run or are active in a software security group, look at it like a yardstick. Consider the activities listed versus what your organization is doing. Look especially closely at the things all or most organizations do (e.g., http://www.informit.com/articles/article.aspx?p=1326511). If your SSG is not doing those things, you might want to consider them. If your organization needs an SSG, you should be able to use the same activities to get one started.

I want to emphasize that we could not have done this without active participation by the nine firms we interviewed. The data in BSIMM is their data. Data from the interviews we conducted were used to build the model from scratch. The examples included with the activities are real examples. After building BSIMM, we scored each organization using it. The individual scorecards, although unreleasable, are fascinating. They provide a unique glimpse into how local culture, perhaps as much or more than business imperatives, drive the approach to software security. Suffice it to say, for now, that the carrot is once again shown to be mightier than the stick.

I’ll talk more about the BSIMM and individual topics over the coming weeks.

As a final note, BSIMM is a data-driven model. The model will improve when more real-world data is added. If you’d like to discuss how to get your organization’s data into the model–and the comparison of you against others back out–please let me know at smigues -at- cigital.com.

Technorati Tags: BSIMM