We can’t tackle the “process improvement” problem with our open source provides like we can with those who build our software in-house, or those vendors from whom we acquire code.

Understanding, based on this, we lack as many ‘knobs and dials’ for improving the cleanliness of the pipes through which open source software flows… we may have a flow rate problem as well. Though I lack hard evidence, I’d bet this represents a proverbial iceberg’s tip:

An organization may deploy as much or more open source code as their own in-house developed code.

It’s interesting to think about the money organizations spend to secure the code they build vs. the amount of open source they consume in this light… (participants indicated they didn’t track this spend separate from their development efforts).

Harumph. Our breakout group generated great ideas worth sharing. First, we unearthed a lot of things attending organizations already do. Next, we brainstormed valuable next steps. Here’s categories of activities we came up with:

• Inventory & Inventory Control


• Vulnerability Identification (Assessment)


• Vulnerability Management


• Ownership of Open Source


• Policy (use)


• Policy (Contribution)

What did each entail?

Inventory & Inventory Control

• Identification – All Participating organizations had a manual discovery process for open source usage. Many wanted better automated schemes, despite some existing tool usage.
• Identification of masked open source – Many participating organizations realize that not only do they adopt open source software directly but that the 3rd party code (and open source) they absorb also contain open source software. Discovering this ‘masked open source’ software represents as big a problem as identifying the open source software used directly.
• Centralized open source distribution – Some organizations allow development & deployment of open source software from the web directly whereas others only allow access to and use of open source from a centrally managed repository. Centralizing deployment usage may provide only improved integrity, or may be used to implement an ‘approved package list’.

Vulnerability Identification (Assessment)

• Using assessment tools – About three-quarters of workshop respondents used the same tools they use to assess their application security posture on their open source assets. This, to me, seems worth its own blog entry. I had to wonder aloud, “as organizations move from detective assessment schemes (SCR, PT) to so-called preventative (threat modeling, architecture analysis, misuse/abuse cases) how do they consider open source?” I’m finding that organizations blazing trails into security architecture work commonly omit discussion of their open source frameworks.

Ownership

• Maintain a white list of open source – Seemingly related to the “centralized distribution” item (but surprisingly uncorrelated in our survey), this meant that someone, from security, owned assessing the risk and proffering a “thumbs-up” or “thumbs-down” that can inhibit white list membership.


• Revisit white list – Organizations expressed that they found value in pruning the approved list of open source software based on non-use, newly identified risk, and similar factors. About one-quarter of our group engaged in this activity.


• Ownership of identified risk – Some participants avidly encouraged use of open source within their applications. In these organizations, when a developer chooses to include open source (as opposed to writing a widget themselves), they own any newly identified risk when in that open source. This reminded me eerily of Wall St. traders. Equity investment creates risk. Margin calls create leveraged risk. In this metaphor, choosing to adopt open source seems like a margin call. It’s very possible that a developer can absorb more risk into the organization than they themselves could effectively own up to in black-swan scenarios. It’s unclear how to measure this exposure when adopting open source.
• Collaboration – Certainly an organization-specific and an unsolved problem, participants indicated there may be a “third stakeholder” in the process of identifying and managing open source vulnerability. Two examples given were 1) clearing houses from which organizations purchase open source software and 2) support organizations (a la RedHat).

Vulnerability Management

• Root Cause Analysis – When a vulnerability is found (regardless of means or source: internal/external), organizations sometimes can point to a person or group who understands the vulnerable component (notable exceptions exist for purchased software or that software maintained by a development team that’s vanished). In open source, the organization must expend resources in order to “get smart” on vulnerability’s root cause and make trade-offs about mitigation strategies and their impacts. This represents extra cost on which I’d enjoy having greater visibility.


• Vulnerability Impact Analysis – Almost every participant had some regime by which they discovered (through assessment, feeds, or other means) new vulnerabilities within their adopted open source code base. Everybody possessed some ability to figure out which lines-of-business or development teams might be affected by newly discovered vulnerabilities.


• Patch Management – Only about one half of participants had, in their minds, a good strategy–having assessed the impacted teams–for distributing a patch that remediated open source vulnerability in an organization-wide manner. More strategically, several schemes seemed available to organizations beyond the straightforward “penetrate-and-patch” loop here. Alternatives included:
  
  
  1. Wrapping open source
     
  2. Hardening open source (and centrally distributing)
     
  3. Sandboxing/compartmentalizing open source
     
  
  

Policy (Use)

• Security Policy/Standards – About half participants had some kind of security policy or standards addressing how to securely use open source software within the organization.
• Training – About one quarter of participants trained their developers to use some portion of their open source securely. Interestingly enough, the one quarter of our respondents that trained developers did not line up well with the one half that had security standards. Wild.

Policy (Contribution)

• Legal permission to contribute to open source – Some organizations see open source contribution as a key activity. Others did but have suffered clamp-down from their legal departments because legal fears liability. Others never liked the idea.
• Community Notification on Vulnerability – When an organization contributes to open source and later finds (or is notified of) vulnerability in its contributions, it may need a way to notify the broader community. Organizations also complained about very high latency in the community notifying them of vulnerability in their code. This proved a surprising problem in our brainstorming session. Why? Contributors complained that this was because, often, their contributions were either 1) forked or 2) baked into other products that masked use of the contributions. In either case, it wasn’t evident to those disclosing the vulnerability that the (contributing) organization was responsibly for vulnerable code.

I will absolutely not let it go without saying that though this entry contains many of my own thoughts it heavily relies on the work of many in our breakout session, well-lead by HP’s Brian Chess. Thanks all for a great discussion.
-jOHN

A clear evolutionary path for the family of security toolkits lies ahead. In order to achieve broader adoption and greater effect in larger enterprises the project’s participants must focus not just on API-level design but also on what I’ve dubbed enterprise readiness. Enterprise readiness entails:

• Centrally managed development road map
• Predictably periodic release schedule
• Increased modularity and reduced dependencies
• Adopter-centric documentation
• Stated backward compatibility
• Integration with scanning technologies, demonstrating correct usage

Interestingly, the project in current release form already addresses some of the above items. Users’ complaints prescribe an upgrade-even if that entails only more prominent communication of existing resources.

Road Map
Since 2008, the project underwent a multiple management changes. Now firmly in the hands of individuals focused on secure development, it’s time to form a steering committee or other community process that results in the creation of a single shared direction for all involved parties. Organizations need to know not only what ESAPI represents today but also what it wants to address and offer in the next 12-36 months.

Release Schedule
The 2.0 GA release accomplished much but suffered from a changing and delayed schedule. In order to reliably fit into organizations’ own release plans adopters must be able to rely on a release schedule for their own planning purposes.

It would be helpful if development efforts published and adhered to a “software development lifecycle”, regardless of how agile. Why? Such a move would demonstrate maturity and would help individuals from adopting organizations follow and predict progress against schedule releases.

Pushing features to release compels security folk by deflecting criticism and reducing adopters’ risk. However, from the adopting organizations’ perspective, it creates work. Even if they don’t adopt new releases immediately, it forces information-gathering and decision work. Predictability outranks both feature progress and frequency of releases.

Modularity
Tinkerers and serious adopters complain that a large list of dependencies creates undue collisions and conflicts, complicating both adoption and maintenance. Splitting each language’s code base up (call that better componentization, modularization, or <whatever>) should help address that problem. Truly upgrading existing tightly coupled elements into a la carte modules will require comprehensive refactoring, dramatic changes in dependency choice, and clever use of namespaces and dynamic loading (Spring-style dependency-injection has already been discussed).

Because modularity produces not only its intrinsic benefits but also clarifies subsequent road map definition and reduces risk in release schedule planning, I count pursuit of this goal as the task of paramount importance. Chris hopes to schedule a modularization tiger team shortly.

Documentation
When one [successfully] adopts a particular implementation, what security problems should they expect to no longer find in assessments? What steps will adoption require? How much effort does customization, integration, and implementation entail? Increasing enterprise readiness requires user-centric documentation.

• Define a threat model, scoping toolkit objectives/effect
• Outline user stories, indicating workflow support (and addressed misuse/abuse)
• Produce adoption guide, enumerating steps acquiring organizations need to successfully adopt and implement a security API

Summit participants rightly concluded that early adopters can best help close these documentation gaps based on their own experiences. Later, more specific ‘specification’ of toolkit function can follow.

Backward Compatibility
Throughout its evolution, the toolkit’s implementers balanced backwards compatibility with progress. However, one of the first “user comments” at the summit was, “We need backward compatibility.” This represents the prime example of need for better (perhaps more formal?) communication.

Scanning Tool Support
Finally, getting credit drives good behavior. Providing organizations “rule packs” for static and dynamic assessment tools that show alleviated risk from toolkit use should increase adoption. Updating leading commercially-produced SAST products (or SaaS) with rules that remove findings where provided security functionality effectively mitigates risk requires more thought than existing first-pass “ESAPI rule packs” incorporate.

Contributors must carefully consider the level such rules should target. Yes, unit testing shows quality of the toolkit’s implementation itself, but does not demonstrate correct integration of the toolkit with an application. Toolkit contributors, unfortunately, can not produce a generic parametrized system-level security test either demonstrating adopting applications bear no vulnerability.

Conclusion
Summit participants showed support across this broad set of suggestions and discussed each item in relative depth. Now, the objective will be to follow-through on each without loosing momentum.

One possible facilitator could, of course, be private funding. Such funding could facilitate several factors of enterprise readiness simultaneously by 1) defining a road map, 2) proposing, funding, and therefore guaranteeing a work schedule, and 3) narrowing focus to a particular aspect of (increased) modularity (or documentation). Excitement regarding the possibility of both accelerating and focusing progress caused me to blog about the Interaction Model in my last post.

In the meantime, as summit members return home to day jobs and dig themselves out from under inevitable mounds of e-mail and TODOs, I reflect on the possibility of the ESAPI project with renewed interest.

Cigital has always favored delivery of holistic assessments (those assessments that consider architecture/design as well as code, leveraging static & dynamic tools, and using both tool-assisted and manual techniques). During my tenure as Director of Security, we found the following distribution of findings:

• Static tool: 20%


• Dynamic tool: 5% (*1)


• Manual SCR: 15%


• Architecture Risk Analysis: 60%

Those numbers may surprise and demand some explanation. For instance, though Microsoft at one point reported 50% of their findings were “flaws”, Cigital was finding 60% (on average, with as much as 70%) of its issues as ‘flaws’. If anything, our classification of ‘flaw’ was less inclusive though, and at the time I looked into this, I concluded that our larger percentage could be due to technique, focus on architecture in methodology (LoE), or level of customer familiarity/maturity with the tech-stack they developed on.

Likewise, our process started with a “whitebox” triage then shifted to exploratory testing (aka: dynamic) rather than the other way around (which I find common elsewhere) and that order-of-operations probably moved balance from dynamic to static. So, in practice we “found” vulnerabilities statically and then supported those findings with dynamic tests to validate their exploit-ability and impact (though perhaps dissatisfying, static gets the point in our data set here).

Slicing by technique
Our experience made it pretty evident to us: skip assessment techniques and expect to miss some critical vulnerabilities.

Slicing by vulnerability type
As I’ve mentioned in previous blog entries and mailing list posts, security managers need the capability to assess and report on their ability to find particular classes of security vulnerability as well. WASC’s Web Application Security Statistics provide a starting point for both classes of attacks you might expect to find and their probability. There’s no shortage of vulnerability taxonomy work out there, so we’ll leave this topic at this superficial level.

Optimizing a Security Practice
Whether you know what percentage of your reported findings come from each assessment technique, or you’re just starting to build your assessment practice, optimization (measurement, analysis, iterative improvement) should be a goal. To do this we combine our knowledge of the techniques we’re employing and the enumeration of vulnerabilities we want to find and report.

Confined to the context of static analysis, I described the following loop in a previous blog post to do just that:

• Use a representative sample of your applications to observe a static analysis tool’s performance, not a contrived test suite.
• Consider the tool’s findings relative to pen-testing findings on the same app.
• Did new and interesting findings result?
• Did pen-testing findings found by the static tool provide adequate root-cause analysis and remediation advice to fix problems earlier?
• How long did it take to on-board an app? How will this scale to your portfolio of apps?
• How long did it take to triage the results? How will this scale?

The purpose of this trial experimentation was to provide some basic comparison between assessment techniques (though you could easily come up with a more thorough approach). Of course, you’d want to consider findings from the perspective of all techniques, not just static tools.

Internally, we had our assessment lab managers write up a qualitative comparison of assessment techniques we employ, comparing how well each is “suited” to finding vulnerabilities against common vulnerability taxonomies (*2) and came up with the following:

• ARA: 14%


• Static tool: 12%


• Manual SCR: 21%


• Manual Pen: 21%


• Dynamic Tool: 12%


• Sec Testing: 20%

At this point, the reader may be asking why my experience differs from our lab managers’. Specifically, for instance, “why are dynamic techniques over-estimated here as compared to to historical findings rates?” Two reasons (at least) come to mind:

1. The balance of techniques that constitute our assessments are “moving back” towards dynamic as software-under-test and market forces evolve


2. There’s a difference between how suited a technique is to discover something and how we choose to find it (I guarantee that in some circumstances this gap represent ‘optimization opportunities’, while in others it represents explicit technology or methodological choices).
   

The Caveat You’ve Come to Expect From Me
Whether your considering a vendor’s assessment service, an assessment standard, or your own internal assessment practice, it’s key to be able to detect biases. When others supply you assessment data you’ll likely uncover biases (though it may not be apparent to those reporting initially). Be careful, I doubt you’ll be able to compile, combine, or relate numbers you receive from varying sources for this reason.

If you’re compiling assessment data for the purpose of motivating practice improvement, you may want to either 1) extract universal similarities from the data you get and present those lessons learned to management, or 2) just list the others’ experience separately. This can provide those to whom you’re presenting a compelling view to your perspective on differing assessment approaches.

In fact, many factors will muck up assessment comparison. For instance, when people report a percentage (like I did above), did they mean finding instances, findings, by type, or something else (*3)? What was their methodology, level-of-effort, and how did they conduct assessment. For instance, risk-based assessments produce a very different distribution (with wider variation) than time-boxed techniques, which more frequently drive to an explicit (or worse, implicit) checklist.

One thing universally seems true though: the more assessment techniques a software security practice makes use of, the more types of vulnerabilities it can expect to report.

-jOHN

(*1) – Clarification: our use of dynamic tools is human-assisted (both in terms of crawl and in follow-up with manual effort or leveraging internally developed tools).

(*2) – WASC, OWASP Top 10, CWE Top 25

(*3) – My data is comprised of reported vulnerabilities. Answering questions about what risk-level I reported requires some abstraction as clients differ in their requirements there. Likewise, behavior regarding instances vs. buckets of ‘related vulns’ requires addl. explanation.

In his post he discusses the need to consider the cost of things in terms of the opportunity lost by doing them:

“Jim Rogers has a great practice on saving – when you think of buying something today, simply multiply its cost by twenty to see how much it will be worth down the road when you are retired. A dinner out is “only” $75, but would you still go out to eat if you figured the cost at $1,500? Warren Buffett pithily sums this up as – do I really need a $300,000 haircut?”

Indeed.

What about the reverse? As a security manager, please ask yourself,

“What am I not doing incrementally that I’m going to regret having made no progress on in the future?”

Today, as I look back on complaining about the dynamism of Java EE applications, I could kick myself for not spending a few hours each week building scanning technology to address these issues. Bygones.

What about you? Run an assessment practice? Couldn’t you just kick yourself for not spending a few hours each week:

• Measuring what vulnerabilities you found, and what techniques paid off to find them?


• Measuring which finds were fixed and which most frequently resulted in regression?

How about those of you working with developers and security toolkits? Couldn’t you just kick yourself for not spending a few hours each week:

• Collecting the ‘best examples’ of validation code from your reviews?


• Writing a security widget to encode output for a particularly popular context?

Just as the future cost of (even seemingly small) wasted expense today can be staggering, the regret for not addressing daunting problems with an almost negligible amortized effort can be staggering later.

What can you start addressing incrementally now?

So, why talk about false positives in the first place? Well, tool adopters I correspond with unilaterally indicated that “dealing with false positives” was their big challenge. I thought, “Address the pain directly.”

However, this is not strictly how I went about it with more savvy folk. Instead, we focused on positively identifying those vulnerabilities we could easily verify as needing fixed. This may sound like two sides of the same coin–and to some extent it is: reducing false positives reduces the pile of hay in which one must look for needles.

However, other techniques bore just as much (or more) fruit. We successfully tuned through other means such as:

1. Adding scanning capability (sometimes custom rules, sometimes custom scanners) for vulnerabilities that always slipped by source code review but for which testing verified existance.


2. Separating and promoting those vulnerabilities that take more time/effort to verify with testing alone


3. Plainly removing those rules that produced inaccurate results easily found by testing *gasp*


4. Discussing pain-points with operations staff: what weaknesses attributable to source code were they constantly chasing in production?

This tuning scheme is very myopic: fixating on assessment. That is, it focuses on the relative effort associated with various different security assessment activities. It ignores broader risk management factors (discover-ability, probability of exploit, and impact) and remediation factors (effort to fix, regression rate, etc.).

Advanced practices use all these factors to tune. But, organizations without risk management practice or development relationships can use these assessment-driven measures to drive improvement without external dependencies today.

Expect more on the topic of “Focusing on what to fix” from me.

I’m also pleased to announce that CSO online has syndicated Reality Check and will be distributing the podcast to their CSO audience. You can find the first episode with Steve Lipner here.

And Jim’s episode here.

Download the podcast here.

The OWASP interviewer is Jim Manico, and he did a great job. He was a little worried about some of the questions he asked. In fact, off the record he kept saying he was sorry and telling me that I did not have to address certain questions. Personally, I enjoyed the questions he asked immensely. Though some of his questions were loaded, I do hope that my answers may serve to clarify our position and eliminate OWASP concerns.

Here are a few of the many more questions I address in the podcast:

• Why do you insist on use of the term “software security” as opposed to “application security”?
• What is static analysis good for and what is it no good for?
• What is the exact relationship between Cigital and Fortify?
• Why do you think your “top 19” is any better than the OWASP top 10 or the CWE top 25? (Special note, the 19 Sins work is Mike Howard’s and John Viega’s…I was not involved.)
• Why does Cigital have a proprietary approach to IP?
• What makes the Touchpoints any better than the SDL or CLASP?
• What is your relationship with Allan Paller and SANS?
• Who picked the “porn music” theme for Silver Bullet?

As an extra bonus, the theme music for this episode is a song written and recorded by my band Where’s Aubrey.

Anyway, enjoy the podcast, and let me know what you think about my answers!

More links:

• Show notes
• MP3
• OWASP Podcast RSS Feed
• Direct iTunes link

The Reality Check Podcast with Gary McGraw focuses directly on software security practitioners and practical software security. Reality Check’s sister podcast, the Silver Bullet Security Podcast with Gary McGraw, follows a free form interview style tailored highlight the ideas and experience of security gurus. By contrast, Reality Check is concerned with practical questions centered on running large-scale software security initiatives in the real world.

Reality Check targets experienced leaders working to solve software security problems in large organizations every day. We use a standard script to guide each conversation with questions about history, methodology, best practice, and measurement. We plan to interview leaders of mature software security programs and leaders of programs just getting started.

Your feedback is absolutely welcome. Please subscribe to the series through or RSS feed or through iTunes.

I joined Cigital because I’m passionate about software security, and because of the great people at Cigital, many of whom I’ve known for a decade or more. I hope to help improve security at Cigital’s customers, and to raise awareness more broadly of security issues in government and commercial systems. Look for my occasional postings and rants here, on my personal blog at abqordia.blogspot.com, and on the RISKS digest at www.risks.org.

• Who Should Do Security? (October 2004)
• Application Security Testing Tools: Worth the Money? (November 2004)
• How Do Real Bad Guys Break Software? (December 2004)
• Innovative Rootkits: The Ultimate Weapon? (January 2005)
• Are We In a Computer Security Renaissance? (February 2005)
• Where Does Trust Come From? (March 2005)
• Is Your Mac Really More Secure? (April 2005)
• How Does Security Fit With Engineering? (May 2005)
• Are Cell Phones the Next Target? (June 2005)
• Is Penetration Testing a Good Idea? (July 2005)
• Is VoIP Secure Enough For Prime Time? (August 2005)
• Is Cisco Naked? (September 2005)
• How Bad Is Intrusion Detection? (October 2005)
• Is Security Really About Getting Nothing Done? (November 2005)
• When Does Security Cross the Line? (December 2005)
• Is Sony BMG Run By Malicious Hackers? (January 2006)
• Is Application Security Training Worth the Money? (February 2006)
• How Flawed Is Microsoft? (March 2006)

We all know what’s happening to magazines and newspapers, though, don’t we–they’re turning to bits. When CMP killed IT Architect magazine (along with most of the rest of their paper publications), they repurposed much of the content into websites. I started writing for darkreading.com from the very beginning. Here’s a set of pointers to the darkreading articles:

• Microsoft’s Missed Opportunity (May 3, 2006)
• New Terrorist Profile: Phone Users (June 13, 2006)
• If You Build It, They’ll Crash It (July 7, 2006)
• Google is Evil (August 4, 2006)
• Keep Your Laws Off My Security (September 7, 2006)
• Diebold Disses Democracy (October 9, 2006)
• Boarding-Pass Brouhaha (November 2, 2006)
• Foxy Vista Henhouse (December 11, 2006)
• Hurray for Hollywood!? (January 12, 2007)
• Security’s Symbiosis (February 27, 2007)
• Compliance As Kick-Starter (March 12, 2007)
• Want Turns to Need (April 20, 2007)
• Certifiable (May 9, 2007)
• JSON, Ajax & Web 2.0 (June 7, 2007)
• Consolidate This (July 12, 2007)
• The Ultimate Insider (August 14, 2007)
• Mobile Insecurity (September 14, 2007)
• Online Games & the Law (October 11, 2007)
• Beyond the PCI Band-Aid (December 10, 2007)
• Software Security Strategies (January 9, 2008)
• The Truth Behind Code Analysis (February 13, 2008)

Just recently, I decided to move my monthly column to informIT. The readership is much larger, and I like the affiliation with the company who publishes my books. As part of that move, you can also expect to see Silver Bullet syndicated through informIT as well. You can help me make the move a success by keeping up with my column through informIT. (We’re also planning an RSS feed for articles too, so watch for that as well.)

The first column for informIT is just as much about business as it is about technology. One of the issues we constantly face at Cigital is the problem of helping our customers sell the idea of software security best practices up the chain. A common (and misguided) view is that software security best practices increase development time and add cost. As you can see in my first column, that’s simply not true. Here’s a pointer:

Software [In]security: Paying for Secure Software

I’m very much interested in your feedback on my column and any suggestions you have for topics. Feel free to use the forum below to get in touch. Thanks for reading!