A short one ‘real quick’:

I get simultaneously nostalgic and aspirational as holidays and year-end planning bear down on me. Wondering how to innovate and how to get that innovation into use takes a fair amount of my attention. I wrote a blog post in ‘07 on how to get some of that innovation stuff in your own security group.

McGraw collaborated with Routh recently on an article (“Lifestyle Hackers”) for CSO Online. While the article focuses on what a CSO must do to more intelligently deal with social-media savvy employees, it also elucidates what we all know implicitly: consumers and those building sites to cater to them directly are driving innovation faster than the big guys (who used to do the bulk of this driving out of their research labs) are.

This was driven home by a recent “Daily Chart” from the Economist. Microsoft is #2 in spend. I might argue we’ve gotten a lot of value as an industry out of their security initiative too. CAS has always seemed dead-on-arrival to me but, I don’t see progress as a result of research that’s taking us in a fundamentally different direction in Software Security (look at their stated areas of interest for both “Security and Privacy” and “Software Development”). IBM made the list too, but is last (see my last post, in which I discuss my impression of how quickly IBM will adopt innovations like O2).

Other than IBM and Microsoft, you’ll not find software companies on the chart at all. And, while communities might bring together experts and provide progress, I fear it will be all-too incremental. Security is plainly in the hands of consumers. Yet, as the bevy of Facebook security/privacy concerns indicate, their demands too leave us well short of the goal line.

As I drove Dinis to the final day of AppSecDC he (as often is the case) had his laptop open. We traded ideas regarding the future of O2, support, and other broader issues about the future of software security. As we discussed or machinated over word choice, I found myself in near-complete agreement with him: not an unusual circumstance.

In his post RSnake muses:

I’ll be curious to see if any big companies step up to the plate here and takes ownership. It’s a bit unclear about Dinis’ fate within IBM – I think he’s a bit on the fence.

I characterize O2 as a platform that facilitates a highly-experienced or expert-reviewer in understanding software. While Dinis has taken a few runs at automation and work-flow support (before with his WCF stuff and now with his XRules) I think the principal benefit of his current state of development remains unshackling the reviewer from limitations of a SA tool often in terms of data-flow across language boundaries and through framework / generated code. So important is this concept to myself and Cigital that we’ve built our own framework which we call ‘The Factory‘. We use it for a similar purpose as one might use O2. As Dinis consistently reminds me though it is not open source. And, yes, there’s a lot of other wicked-cool stuff in O2 (the Visual Studio debugger integration is my favorite).

Cigital believes in O2 enough that we’ve conducted hands-on O2 training with a bunch of our guys even after Ounce training. I personally believe in the technical value to code reviewers of O2 enough that I put a modicum of code towards it when Dinis needed it in a pinch. I’ve also agreed to build and publish O2 training for the masses; ‘training that makes it seem less scary.

Taking a step back for a second, there’s a large leap between where the world is and the world Dinis describes in his recent blog post. Unfortunately, I see a lot of organizations doing software assessment driven by (and in pursuit of) compliance only.

So, it doesn’t shock me that IBM hasn’t dived head-first into the O2 pool, regardless of the opportunity it may represent. I believe they will fully embrace it when the market can support it. In the meantime, O2 can continue to find hospitality and support in the welcome arms of assessment experts like Cigital.

I’ve been thinking about this for a while and the tone of this year’s OWASP Global Summit has brought the topic to the forefront. OWASP, as many of you know, is a fiercely open source community. At times, participants defend its open and freeness a bit aggressively for my taste. Sure, open and free are founding principals of the community. I think these principals are essential, valuable, and worth protecting. However, I also believe the community-more broadly-would benefit from an evolved perspective.

Specifically, I believe OWASP should welcome branded security vendors and named individual practitioners into its arms openly. There are three reasons and as I outline them, think to yourself about what vendors like RedHat did for the Linux community.

First – Commercial entities can provide professional and enterprise-level support for OWASP projects to willing commercial entities. Code-based projects (like AppSensor, ESAPI, or other) are easier to imagine the impact of than others.

Second – Large entities seeking to participate within OWASP need assurances which the OWASP community hasn’t itself provided. Things I’ve heard loud-and-clear include:

1. Anonymous participation for industry players working for sensitive organizations
2. Structured feedback, steering, and funding for OWASP projects

Vendors do not uniquely possess the ability to provide these capabilities. The community could provide this value but has not prioritized it nor has it been able to convince industry it could appropriately address their security/anonymity concerns or provide tangible value. Vendors have much better luck in these regards.

Third, finally, and most Importantly – vendors desiring to enter the space should be seen as a welcome sign of maturity to the space. Maturity, to me, will mean key advancements:

1. Larger and less ad-hoc budgets within organizations for application security
2. The emergence of higher and more explicit standards for quality for the community’s free and open software/tools
3. Convergence of the security community’s message, which will allow it to be taken more seriously

To facilitate this, I suggest the OWASP board do the following things:

1. Explicitly endorse vendor participation, as long as it meets the community’s code of ethics and conduct
2. Stop ‘the crank’ over people’s personal / corporate emails being used on OWASP lists
3. Protect a commitment to technical quality by avoiding vendor pitches at conferences in chapter meetings, and in posting

I really don’t mind when people use their corporate email addresses when they mail public lists (OWASP or otherwise). As a chapter leader, I don’t (personally) mind when presenters show up with their company’s slide stock though I push them to use the chapter template. To me, corporate emails and slide stock help audience members identify and appropriately couch bias. Given my own profession and employer, my own biases should be evident.

On the community front, my roles spanning the gamut between OWASP Member, Chapter Leader, and invited industry advisor. I see my professional life and my community involvement as being mutually reinforcing and beneficial, rather than conflicts of interest. I enjoy having two outlets for my time and work. And, while, Yes there’s bad individual behavior out there, I’d like to see people more comfortable with their dual-roles. Again, I think their professional career, their volunteer community, and the industry as a whole will benefit.

After what must have been an incredible amount of leg-work a cabal of folk from the DC OWASP chapter are putting on the AppSec DC conference. The conference will also play host to the ‘09 OWASP Global Summit. I hope to see you there. Especially those of you practitioners from within organizations’ security groups–I feel like you provide essential perspective from the trenches of our security war.

Elections
Elections will be held to add another board member to OWASP and I’m anxious to see how the process plays out. Knowing all four announced candidates, I imagine different outcomes based on who receives the nod. In an odd turn of events, I actually like all the candidates; I think they’re great guys. In particular, I’ve known Pravir for many years, I’ve worked with him off-and-on, and respect him deeply.

I’d like to point out Eoin Keary’s bid in particular, because I like his focus on quality and governance. I perceive OWASP be at an inflection point in its development and growing pains are already evident. Selecting particular projects on which to focus, placing them under more rigorous quality control, and working towards maturity criteria others have begun to define can really increase the reach and impact of OWASP. This idea is essential to Mr. Keary’s platform.

Tesauro and Chandra, contributors to project assessment criteria, appear to place importance on this as well. Consider the draft criteria their committee is working on.

Again, I think quality is an ever-more-important imperative as the OWASP community grows and I’d like to see the assessment criteria expand to contain some more explicit and rigorous technical quality gates for a project. As I look at popular existing projects, I am beginning to feel a pressing need for outside review/revision.

Talks
As the Java EE persona of the ESAPI project nears release, I’m anxious to see a more hands-on, more technical, and more developer-focused presentation on the project at AppSec DC. Recent presentations/commentary has felt a bit more like cheerleading to me.

Of course, I’ll be dying to know what Dinis has added to O2 recently and it appears he’ll be presenting on this topic.

Threat Modeling
I’ll be presenting on Threat modeling on Wednesday but I’m also very interested in discussing the topic with the guys from SecurityCompass, who will be giving all-day training on the topic. Rohit in particular, has made what I consider to be top-notch start on his Java EE Security Patterns document and I’m anxious to see the methodology that back-ended their work.

Ben Tomhave wrote a decent post musing “Which came first: The Software or The Security?”.

In particular, Ben asks whether the response an organization has to its security problems should possess a time component. “Yes”, he answers his own question emphatically. I agree, and for a few reasons worth expounding on.

…the only difference between today and yesterday
If you’re effective at assessment, you’ll often leave an organization’s apps in shambles. Somewhat older data I have indicate an expectation for 10 critical vulnerabilities / 100KLoC, with 30-40 high-risk vulnerabilities following suit(*1). Organizations holding their broken app at arms length panic. “We’ve got to pull it from production!”, they exclaim. Others take a different approach, “We’ll stop everything until those vulnerabilities are fixed–no more features.”

This behavior is irrational and needs to be quelled. The only difference between “today” and “yesterday” in this situation is that the organization knows how screwed they are (*2).

Ben indicates that visible risks should be prioritized based on a risk management framework. Ostensibly, we’re very much in agreement on that one. There may be vulnerabilities that deserve such “stop the press” kind of treatment, but there is often a normalization process that must occur before that set is culled from an assessment (*3).

After that triage, the rest (of the vulnerabilities), and I’ve said this before, should enter the change-management/bug-tracking process like everything else: features, customer complaints, and so forth. Dealing with security vulnerabilities in-band both sobers security analysts’ dreams of the all-important ’sploit and raises the respectability of security requirements within the normal development process by treating them as first-class citizens of a release process.

Depending on whether an organization has rapid-turn around (as happens when development engages in weekly sprints, organized through SCRUM) or more monolithic develop cycles, these change requests can be weighed from the perspective of whether or not they should be:

1. Released out-of-band as a patch
2. Included in the next planned release, potentially:
   
   • At the expense of other functions/features
   • At increased expense,

   or:
3. Deferred until a future release

Sometimes, the ‘correct’ answer is to conduct a usability study, or revisit the SLA promised by a system/service, and get feedback as to what form the fix will take before any of the above can happen.

In summary, time continues to elapse at a constant rate. In this case visibility is increasing. Visibility should allow for a better risk management decision, not a knee-jerk reaction.

Moreover, it’s important to discern the difference between 1) the organization’s increased visibility into vulnerabilities, not 2) an increased discoverability of the vulnerability itself. Yes, if the assessor used an automated pen-testing tool to find the vuln., it is probably pretty discoverable, but the organization’s knowledge of the vuln. does not increase it’s discoverability(*4).

Onto the next element of time…

0 to 60mph in?
Ben also rightly indicates (paraphrasing) knowing there’s a gap between you and the security posture you’d like to have, you’re not going to get there instantaneously. Very true. This, in my mind, is where organizations could improve the most. Most organizations I talk to have dashboards showing quantities like

• Percentage of applications in compliance with assessment policies;
• Number of outstanding critical + high vulnerabilities;
• Time to remediate critical + high vulnerabilities;
• <Metric du jour>

I’d like to see organizations begin managing to security goals like:

…Next year, we expect no application providing access to [asset X] to be vulnerable to discoverable by an external Threat, be he/she authorized or not, accessing our systems only through the web;

…in three years, we expect no vulnerability that would provide access to [asset X] with any Internet-facing software, regardless of immediate discoverability or exploitability (by an external threat);

…in that same three-year time frame, we also expect no Internet-facing apps. to fall prey to [common web attacks, listed in the SANS Top 25(*5) (aka: CSRF et. al.)] that would result in impersonation and therefore allow for inappropriate access to [asset X] on behalf of an unwitting user.

I’m attempting to extend Ben’s call for a time-based (I’ll call it iterative) approach to responding to security gaps. The previous paragraph proxies for what a more advanced risk management framework would incorporate in the form of probability (discoverability x exploitability x [other factors]). Remember, however, that the [asset X] clauses of these statements proxy for different but important factors in your risk model: intrinsic value of the information asset.

So, whether you’re using a formal risk management framework, or doing it more informally, you can respond iteratively and get time back on your side: having a story about how you’re measurably reducing organizational risk in a meaningful fashion.

Compare this with the alternatives:

1. A straight linear approach – fixing 80% of XSS bugs within an application has what effect on the overall security posture?
2. The bug-of-the-month approach – ‘fixing’ 100% of the CSRF problems within an app, but leaving all the XSS has what effect on the overall security posture?
3. Stopping the presses – fixing 100% of the bugs, and not allowing business functionality to evolve in the balance does what to your business opportunity/revenue stream?

Finally,

Within a single vulnerability’s remediation, I’ve got trade offs
Yes, within the scope of a single killer finding, organizations have to decide how to trade off the time it takes to field a mitigation and the effectiveness of that mitigation on the security posture (queue the “dynamic patching” vs. “fix it in the code and re-deploy” flame).

This, just like our previous two topics, is a matter of risk management. I implore organizations to consider what the capabilities and motivations of their opponent are in this case. If you’re considering protecting against a web-vulnerability delivered through a URL parameter, a WAF rule will prevent a vulnerability scan from finding the ‘fixed’ problem but is unlikely to thwart a skilled penetration tester from manipulating the order of the innocuous and malicious parameters. WAF rules, then, are likely to provide a rapid response to failed scans and simpleton attackers but not concerted attacks of this sort. This same sort of analysis can be done on code-based fixes. When assessments find input validation problems, the development team will often respond by doing a simple (arbitrary) length check, or by black-listing particular SQL characters.

When I was quoted in a Darkreading article as saying costs to fix were [so low], it was these spot-fixes to which I referred. These fixes don’t (often) fully remediate that particular vulnerability (even in that narrow locale) let alone improve the application’s overall security posture. But, I’d say they’re the most common response to an assessment finding. Like McGraw says regarding the BSIMM data, “we observed monkeys eat bananas; is that good? I don’t know.”

Having considered an attacker’s capabilities and motivation, and the value of what you’re protecting… an organization might in fact choose to engage in such cat-and-mouse play as the trivial fixes above imply, engineer a real fix, employ some combination of both, or off-load risk in an entirely different way.

Just like before, however, it’s unlikely the vulnerability was created yesterday, so the software has been vulnerable for some time. When risk analysis has occurred, and the full cost of a complete solution is resolved, complete with its own residual risk calculation, it’s very likely that a practical organization will tier its response, iterating in time.

Thanks for the post, Ben.
-jOHN

(*1) – Within a Java EE app found through tool-assisted code review with no preference for having been reviewed prior or not (be it prior code review or penetration test). Critical and High in these cases varied somewhat, but always having been seen by the client as exploitable and accepted for remediation by development.

(*2) – This is somewhat glib. More accurately, the organization has increased its visibility into what vulnerabilities its software possesses. It can not say these are the only vulnerabilities the software possesses, and, using purely code review, it can not always say with 100% confidence that detected vulnerabilities are exploitable; some statically determined findings are obviously exploitable whereas others will require dynamic verification.

(*3) – This normalization is even more important if the assessment was delivered by a 3rd party tool or assessor conducting their first run/engagement with an organization. Internal assessment teams often have it easier; they should know/conform to the business unit’s risk management measurement memes.

(*4) – Unless they have a really terrible vulnerability management process or a shady security vendor ;-)

(*5) – Indeed, I have suggested use of a Top N list. Here, I’m using it to indicate a priority implicit in industry opinion. Better to augment that clause with [the most common types of attacks observed in our production environment and the #1 emerging attack our security research team advises us on].

My last post spawned some questions, which I responded to in turn. Here was my response:

[Adapters]
Adapters for assessment results can take a few forms, but let’s address three specific scenarios that fan-in to an assessment results/presentation step and a few that fan-out.

[Fan in]
Fan in typically comes from three sources: 1) static tools, 2) testing tools, and 3) manual analysis.

Adapters that deal with fan-in have three challenges to surmount:
A)Technically trans-code results from #1, #2, or #3 into a single tool’s format for roll-up, or a tool-independent alternative

B) Normalize results between #1, #2, and #3 so that ‘apples’ and ‘oranges’ get reported rather than ‘apples’ and ‘cars’

C) Code results with organization-specific “whathaveyou”

[Challenge A - Trans-code]
The good news about the tools is that they nearly all export to some XML format you can manipulate. This tackles output from #1 and #2. Organizations that adopted any tool early have struggled with keeping up with format changes but have indicated to me that they’re willing to pay this small price for the ability to ‘plug’ that many results into their developers’ bug-tracking systems.

A bigger challenge (effort and therefore cost-wise) is getting manual results (#3) into either the format driven by the tool responsible for reporting or that independent format I described. Smart consulting firms have built ‘emselves workflow to manage this. In the ’90’s we at Cigital used to pine for the report-generating automation of our then-competitor @Stake and waffle about whether our client-base was too disparate and our assessments too varied in scope to support a similar trick. I see no reason not to, within an organization, code up something that helps-technically-integrate your manual review findings with those found by tools.

[Challenge B - Normalize]
Manual reviewers report to me their discomfort with the rigidity of systems like “The Seven Kingdoms” and its competitors. Mind you, I don’t dislike these taxonomies, but people get crotchety and might complain if they’re forced to cram their findings down into a SAST or DAST report. They perceive, it would seem, the vulnerabilities they find to be like snowflakes ;-)

An independent format can be successful as well. One can roll their own (for their organization), or lean heavily on the community and go with something like Mitre’s CWE. At Cigital, we’re involved with CWE and follow-on work but have also helped people organize their own. A critical discussion of this trade off is possible, but beyond the scope of this email. That leads us into…The cost of normalizing results between #1 and #2 eclipses the technical challenges of building the adapter.

[Challenge C - 'Whathaveyou']
I would say this though: if your organization has progressed to the point where it possesses security standards (prescriptive or otherwise), they should absolutely be referenced by appropriate findings in the report. This goes for both violations of the standards and the applicability of standards that would serve to prevent or mitigate a particular risk.

References to policy/standards can be augmented with best practices, if your organization makes that distinction. I’ve seen organizations link to internal/external resources for training and/or further information as well. If you’re going to try to transition from “the bug hunt” to “building security in”, one great way is to provide developers immediately-available information as they’re making the mistake.

[Fan out]
Now, you have to ‘fan out’ into support for bug-tracking systems. Most organizations’ security groups have at least as much battle to fight here as a security consultancy, actually. Why? Because the organization hasn’t mandated a single development toolkit. The good news here is that while there may be more fan-out than there was fan-in, bug-tracking tools were built to be supplied data. Writing this portion of the adapter–a conduit between what normalized findings you’ve compiled and the offending team’s bug tracking system is fairly (technically) straightforward.

[Industry Std. 'Schema']
Sean Barnum has done a flotilla of work on this topic with Bob Martin at Mitre. Though it can be cumbersome or uninteresting to practitioners, I think the work they’re doing is important because whether its admitted or not, work on audit, testing, and verification methodologies/standards must implicitly take a stand on defining the words you listed (finding, root cause, vuln., etc.). Where such efforts take a stand, one’s organization can find alignment with their own notions quite challenging. Where it’s implicit (or worse, ambiguously and poorly defined) you get lots of wasted time as assessors argue with development over semantics, next steps, and responsibilities.

Each methodology has its own limitations in this department, resulting from its focus and perspective, IMO. If you look at OSSTM, there’s a wealth of definition around activities, which really helps those implementing it differentiate what techniques they could apply in testing their system. Their template reporting form falls short on defining constructs such as root cause and finding and ’speaks’ like an auditor’s report. This doesn’t do the depth and breadth of their assessing techniques justice which means, ultimately, adopting it will take a lot of work in the realm of that normalization task we treated earlier. NIST’s methodology formalized controls even more producing the 800-53 publication. I need to look at their recent foray into app sec and reconsider ASVS much more closely and for much longer to make judgments in this realm. Currently, I’ve only considered it in the insanely and unfairly narrow context of “a set of stuff to look for.” I’ll follow up with you on t!
his later this week or next.

[Correlating Risk Systems]
Taking your question literally: Risk systems? Most risk management companies wield Powerpoint and Excel, and as such, glue is hard to come by–let alone ‘open glue.’ I don’t have much experience with Archer, but their glue is proprietary but their suite includes the ability to weave together policy, requirements, findings, and change/bug management. It sits outside the MS Office stack, but what little experience I’ve had with it wasn’t necessarily positive ;-)

A few back, I wrote about Maturity Models vs. ASVS. On SC-L, a ‘discussion’ broke out regarding Maturity Models (MM) vs. Top N lists. Like ASVS, Top 10 lists target a different problem than MMs. In particular, the discussion focused around how one should enhance their assessment practices.

I’ve edited and reproduced my SC-L post here for those of you who don’t read the list. My objective? provide specifics on what one can do to enhance assessment practices. I provide a commentary on the on-going discussion throughout. The central theme of my approach?

security vulnerabilities don’t count unless they affect development

Now the approach:

Integrate Assessment Practices

[What?]
Wrap the assessment activities (both tool-based and manual techniques) in a process that:

• Normalizes findings under a common reporting vocabulary and demonstrates impact
• Include SAST, DAST, scanning, manual, out-sourced, & ALL findings producers in this framework
• Anchors findings in either a developmental root cause or other software artifact:
• Use Case, reqs, design, spec, etc.
• Builds adaptors so that bugs are automatically entered in tracking systems
• Adaptors should include both tool-based and manual findings
• Calculates impact with an agreed-upon mechanism that rates security risk with other factors:
• Functional release criteria
• Other non-security non-functional requirements

[Realistic?]
I believe so. Cigital’s more junior consultants work on these very tasks, and they don’t require an early-adopter to fund or agree to them. There’s plenty of tooling out there to help with the adapters and plenty of presentations/papers on risk (www.riskanalys.is), normalizing findings (cwe.mitre.org/) , and assessment methodology (www.cigital.com/papers/download/j15bsi.pdf).

[Panacea?]
No. I’ve done research and consulting in functional testing. If you think security is powerless against development, try spending a few years in a tester’s shoes! Functionality may be king for development and PMs, but I’ve found that functional testing has little to no power. While a lack of features may prevent software from going out the door, very rarely do I find that functional testing can implement an effective “go/no-go” gate from their seat in the org. That’s why testing efforts seek muscle from their friend Security (and its distant cousins under quality “Load and Performance”) to stop releases from going out the door.

There’s no reason NOT to integrate with testing efforts, reporting, and groups: we should. There’s every reason security should adhere to the same interface everyone else does with developers (let them produce code and let them consume bugs)… I think the steps I outlined under ‘what’ bring us closer. I don’t think we need to (or can expect to) flip organizational precedent and behavior on its head to make progress.

[Steering]
The above scenario doesn’t allow explicitly for two key input/outputs from the software security ecosystem:

1. Handling ultra-high-priority issues in real time
2. Adjusting and evolving to changing threat landscapes

Establish a Steering Committee

[What?]
Establish a steering committee on which a software security, dev, architecture, operations, and corporate risk sit. These folk should manage the risk-model, scoring, security standards that drive the assessment verification standard, and the definition of both short-term and longer-term mitigating strategies. I’d argue that you’d like Industry representation too. That organization could come in written form (like the Top N lists) or in the form of consulting or a panel.

When incidents or firefights come into play, absolutely allow them to be handled out of band (albeit through a documented process), but! Not until they’ve been rated with the agreed-upon model.

[Realistic?]
Yes. I have several clients that use this structure. I speak with non-clients that do the same. Data gathering for scoring and prioritization is easy if you’ve done the steps in the previous section. The operations guys help you grade the pertinence of your efforts to what they’re seeing ‘in the wild’ too.

[Panacea?]
Does a steering committee help you respond with agility to a high-priority threat in real time? Not explicitly. But, it does help if your organizational stakeholders already have a working relationship and a mutual respect. Also: I think one root cause of the underlying discomfort (or dislike) with people’s perspectives on this thread has been:

“OK… you don’t like Top N lists… So what do you do?”

In my mind… The above answers that question.

[Assessment and Tools]
Do I believe that the normalized findings will emerge only from static analysis (or any other kind of vulnerability detection tool)? Absolutely not. People who follow my writings know I expect dramatic (ally high and low) numbers to be associated with tools.

Let’s summarize my data. Organizations can expect:

1. Static analysis tools to account for 15-20% of their total findings, out of the box
2. An initial false positive rate as high as 75-99% from a static analysis tool, without tuning
3. Less than 09% code coverage (by even shallow coverage metrics) from pen-testing tools

Qualitatively, I can tell you that I expect an overwhelming majority of static analysis results produced in an organization to come from customization of their adopted product.

Simply: if you base your world view on only those things a tool (any tool) produces, you’re world view is too narrow and will prove ineffective. The same is true of those who narrow their scope to the OWASP Top-10 or the SANS Top 25.

[Top N Redux]
Some have left the impression that starting with a Top N list is of no use. Please don’t think I’m in this camp. In my last two public presentations I’ve indicated, “If you’re starting from scratch these lists (or lists intrinsically baked into a tool’s capabilities for detection) are a great place to start.” Yes, one of these presentations was entitled “Why Top N lists are bad” ;-) Also:

If you can’t afford frequent industry interaction-use Top N lists as a proxy for it. They’re valuable, but like anything, only to a point.

For me, this discussion will remain circular until we think about it in terms of measured, iterative organizational improvement. Why? Because when an organization focuses on getting beyond a “Top N” list it will just create their own organization-specific “Top N” list :-) If they’re smart though, they’ll call it a dash board and vie for a promotion ;-)

From the other side? People building Top N lists know they’re not a panacea, but also know that a lot of organizations simply can’t stomach the kind of emotional investment that BSIMM (and the ilk) come with.

This leaves me with the following:

[Conclusions]

1. Top N lists are neither necessary nor sufficient for organization success
2. Top N lists are necessary but not sufficient for industry success
3. Maturity models are neither necessary nor sufficient for organizational success
4. Maturity models are necessary but not sufficient for industry success
5. Always avail yourself of what the industry produces;
6. Never confine yourself to a single industry artifact dogmatically;
7. Whatever you consume from industry, improve it by making it your own; and
8. Where-ever you are in your journey, continue to improve iteratively.

[Related Perennial Rabbit Holes] (bonus)

• Bugs vs. Flaws: John Steven’06 – www.mail-archive.com/sc-l@securecoding.org/msg00888.html
• Security Vs. Quality: Cowan ‘02 – www.securityfocus.com/archive/98/304766

The Geekonomics blog is often good. A new post indicates Apple’s veneer of more secure than Microsoft is cracking.

It was only a matter of time. I wanted to clarify that though you see a lot of security consultants carrying Macs, in Cigital’s case, it’s not an endorsement. Again, in the interest of disclosure: though I own and operate many platforms I operate more OS X at home and office than the others.

I attribute Mac adoption amongst security folk to two reasons: the platform combines a Unix-like environment with the ability to interact through email and MS Office and that the machines, for the most part, are a snappy bit of hardware, cobbled together into a “shiny object” (desirable) form. This second aspect goes a long way to explain the recent jump in ownership in the security community: “fan-boys.”

Any claims that it’s because “they’re more secure” should be considered with a fair amount of skepticism.

Yes, historically, the platform has suffered less pain of viruses and malware. Yes, certain aspects of their OS/platform and design did make improvements over XP.

The truth is this:

Apple doesn’t “Build Security In” very well at all. You don’t have to be an insider to understand why. Redmond is exporting security blogs, books, and value like never before. You don’t see a lot of Apple security people in the community though. You don’t see good solid standards-based support for authentication or web-services that would help you interact securely with your enterprise (Apple hides behind their ‘vision’ on this one). You don’t see a lot of support for Objective-C in the static analysis tool realm.

I believe that though Apple paid great lip service to security as a differentiator initially, (they even talked about phones like the iPhone becoming the basis of identity moving forward) but yet they abandoned it when they realized the cost of a real enterprise-level program. They also abandoned what was probably their best protection: the PPC processor. Their security proposition, IMO, is based on obscurity.

I’ve said before, “Woe to Apple when market share (and thus Economics) garners the attention necessary to motivate attackers to focus on the platform—attackers may find their task easier than with Vista.”

Ben Worthen broke the BSIMM story on wsj.com as was posted earlier.

I was shocked when someone said, “Oh and ASVS is also available, great” on an OWASP list. Super, I thought, but I don’t understand the connection. When I looked at the WSJ site, I noticed Jim Manico (of OWASP, Aspect, and ASVS fame) wrote, “But for those of your programming web applications, consider looking at http://www.owasp.org/index.php/ASVS – it is focused specifically on web application security evaluation.” I know Jim, he’s a good guy, and my curiosity about where the link between maturity models and verification standards was sated, but I thought I’d spend some time here quickly disambiguating them:

Verification standards, like ASVS, enumerate techniques with which an application’s correct use of security controls can be verified. It also posits (what it calls) verification requirements that serve as specifically enumerated tests of particular security controls (such as input validation). Such a standard is best operationally deployed by an organization’s application security group. In some cases, the audit group may own verification and reporting.

Auditors especially love NIST’s 800-53 standard, which recommends security controls for Federal Information Systems and Organizations. This document has frustrated me historically because of its focus on OS configuration, topology, and patch level. Largely, it ignores the application layer of an organizational stack. As such, organizations applying it have found little impact in preventing the web’s most common application security vulnerabilities. I reckon one aim of ASVS was closing this particular gap.

Regardless of what you think of ASVS or NIST 800-53, one applies the bulk of their guidance to an application and systems.

A maturity model such as BSIMM measures what an organization is doing to secure their software. Organizations will be graded on whether they’ve defined and adopted assessment (verification) techniques, but also as to the state of other organizational constructs such as a training program, post-deployment incident response capabilities, and specific application security management. A maturity model helps organizations place their personnel, use of tools, and practices against industry best practices in a broader context (a software security framework). Verification of applications is only a small piece of that governance activity. As such, BSIMM data is produced, consumed, and managed at the CISO/CIO level, rather than within the application security group (as ASVS).

Organizations will need an ability to consistently and comprehensively verify the security of their applications but this is only a piece of what those same organizations will also need to do, in a more broad context, to make sure they reduce, other-wise manage, or transfer software-induced business risk.

To this extent, comparison of BSIMM to ASVS is even a poor fit than comparing the Rational Unified Process (RUP) to Capability Maturity Model (CMM). -shiver-

James McGovern recently wrote a post on Gartner’s static analysis (SA) report. Among other things, he lamented the lack of actionable guidance within the report. A lack of implementation guidance doesn’t shock me from Gartner, I can’t say I expect that from them.

I can help James and community out by giving some of that guidance myself. I’ll try to do so in a tool-independent way. This topic deserves more than a “blog entry” format (admitting Cigital’s propensity for longer entries ;-)) so if people want more information on the topic, they can ping me.

In essence, you want to know the following about static analysis tools:

1. Who is going to run it (how will owners shepherd it)?
2. How much effort is it going to take to run (what budget will I need to support it)?
3. When do I run it and how often (Where does it fit within my SDL?)
4. What it’s going to find (How will the results enter/impact my organization)?
5. What won’t it find (How does it play with the rest of the assurance ecosystem)?

[Market Adoption and Making Your Choice]
Gartner seems underwhelmed by Microsoft’s ability to effect the commercial static analysis market. Cigital has always believed these technologies would make great features of a compiler/IDE and so we’re also disappointed. They’ve got great technologies but don’t exploit them commercially. I still like their freely available FXCop but this opinion isn’t popular. Perhaps that’s because I see a lot of value to FXCop as a unit testing framework that enables security test and other people are comparing it to glossy commercial tools like Fortify and Ounce run by security analysts in a very hands-off fashion. Who’s considering the tool really does make a huge difference in how it will fare.

From my perspective, the most successful SA tool vendors have made answering the “who should adopt the tool” question more difficult as they’ve waffled on licensing models. By the seat? By the core? By the KLoC? I’ve gotten complaints from prospective tool buyers indicating that a leading tool vendor’s price was 10X another vendor’s. Before you ask “Which one?”: I’ve heard specific examples in opposite directions!

Certain tools will look better to different potential buyers. Experience has shown positive response to Coverity’s C/C++ results from developers. Application security groups like Fortify and Ounce’s products, and several years ago, I saw QA Managers in two very different firms go ga-ga over Klockwork’s tool. I attribute people’s impressions not to a product vendor focusing on quality or security but to how closely the tool vendor’s conception of how roles interface with their tools throughout the vulnerability management and software development life cycles matches the organization’s own structure. Up until a few years ago I don’t believe SA vendors had fully conceived the “developer”, “analyst”, “security manager” roles in their products. Ounce was the one exception: they’ve always staunchly believed themselves to be principally a Security Analyst’s tool. Some vendors attempted to unify roles into use of a single interface while others attempted to split the product up into different configurations for each. This created confusion and friction between some adopting organizations and the tool vendor’s license model.

In the end you want SA tools to affect as many developers as possible and to get low-latency scan results as early in software’s life cycle as possible. However, having successfully deployed tools a variety of ways, I’m convinced central deployment is the way to go. Application Security should run the tool. This will help manage rule/configuration update, as well as central measurement collection, risk management, and issue tracking. How do we get low-latency and early-lifecycle centrally? Treat the tool as a service. More on that later.

Notable exceptions to the central deployment model include business units possessing one or more of the following characteristics:

• Acquisitions of previously successful, culturally-independent companies
• High-quality product development teams
• Fundamentally different SDL, development platform, language, and toolkits

For in-unit deployment one can “run locally” but must “report centrally” supporting security goals involving measurement and risk management. And, let’s be clear: if a business unit has a wicked-good continuous integration/build-management group, they-by all means-should be tapped to integrate with the SA service. Cigital and its clients have had success in integrating both the front (code submission) and back-end (bug/issue tracking) with Fortify and a variety of open source build/CI/bug-tracking products. This maintains a central model, but reduces latency dramatically.

[How Much Effort]
I’m sorely disappointed in the honesty of tool vendor sales-folk regarding level of effort. One of my first analogies for SA tools is this: Did your company buy Mercury products for load and performance testing? Think of your SA tool like Mercury–you’ll need at least as much money to deploy and implement the tool organization-wide as the licenses cost you. Sorry.

Organizations with more than 2-300 developers should plan on the following:

• 3-4 man weeks to do initial tuning and pilot implementation
• 4-16 man hours / large application to integrate with the static analysis tool and assure appropriate configuration
• 4-8 man hours / 50-100KLoC (language-/complexity-dependent) to triage results on a new-application scan
• 1 person / year as tool shepherd (app integration, custom-rule creation, rule-pack maintenance and release, support)

Organizations get into trouble when their SA tool enjoys broad acceptance by the organization before they effectively manage the submission and results triage/documentation process. My data indicates code submission can burn 8-24mhrs / application if security groups don’t ‘remember’ (or automate) the submission and scan setup process. Security groups doing scans centrally and hand-writing code review results can waste another 16hrs documenting their findings / application.

…think about it: if your organization allots a week / tool-assisted code review and wastes 24 hours getting the tool’s first result set and 16hrs documenting findings… that only left 10 hrs to actually review the code! Since I believe that 4-8 hrs / 50-100KLoC is necessary to triage, you can imagine how deep I think such code reviews are :-P At Cigital, we’ve got dramatic benefits to depth by decreasing cost of submission and reporting. This is key to scaling SA efforts. Those adopting tools will feel overwhelmed if they don’t plan to solve submission and reporting problems before wider roll-out.

[How Often]
As often as possible. If you’ve got a Continuous Integration initiative (CI), engage those folk. If you deploy your tool purely centrally, plan to scan each high-risk application at least once / release. No, to my knowledge, there isn’t a good answer to the “can these tools do incremental (partial) scans”. You should always keep old results around though, as they’ll (along with other measures) help you understand whether more or less vulnerabilities are being caught during each scan. Remember though, every time you add a rule to the scan, your vulnerability-finding bar just moved and your metrics might be thrown off.

Ounce, Fortify, and Klocwork seem pretty good about determining whether or not they’ve found a particular issue in a previous scan or not. This includes situations where code blocks move around a bit. This means that running regression scans to determine whether or not an issue has been fixed is viable. I can’t comment on Coverity’s capabilities here.

[What's it going to find]
I’m impressed that Fortify has published its vulnerability taxonomy and a bit shocked that others haven’t produced as public a resource. I’m excited too that vendors have agreed to comply with CWE labels when they report findings. I will say what I always do on this topic though: test the tool on your own code though. You simply don’t know what it will find until you throw all your idiosyncratic build, language, toolkit, and platform nonsense at the particular tool you’ve selected.

[What won't it find]
I’m continually shocked as to how much one can find by running tool Y after tool X on the same piece of code (replace X and Y as you see fit). As I’ve written time-and-time-again, the corner cases and exceptions missed by each engine are baffling. Let me put things in perspective for you: three of my clients have reported that their SA tool deployment find 17, 33, and 50% of the total number of issues found by assessment efforts respectively. If these percentages hold for your organization, inside a coding bug realm then your job is at best 50% done having run the tool and triaged its results.

Considering that Microsoft and Cigital believe that “bugs” account for only 50% of the vulnerability space, that leaves you unaware of 75% of your application’s vulnerability at the end of triage. As I always say, “use the tool to facilitate code review and increase a reviewer’s understanding of the code–not as a code reviewer that finds bugs automatically.”

Good luck out there, I’d love to hear about your particular experiences.
-jOHN