BSIMM2 is the 30 firm version of BSIMM. I wrote up an article with Brian Chess and Sammy Migues (my BSIMM co-creators) called “Software [In]security: What Works in Software Security — Fifteen Common Activities from BSIMM2.” In addition to highlighting the fifteen most common BSIMM activities, the article also provides the 30 firm data for all 110 activities in public for the first time.

We’re unveiling some statistical results at RSA this week that will enhance and expand the dataset published in the article. We’ll do an official BSIMM2 launch within the next couple of months.

Apparently the time has come to re-release the SANS/CWE 25 — something that we can expect annually. The good news is that exercises like this do plenty to hype up software security and its importance. In fact, in many ways the target of these lists is “the reporters who cover software security.” So hype = good.

So why am I not a big fan of these lists? Well, I wrote that down a year ago and what I said then still applies. Sure would be nice to see a reasoned response to my criticisms instead of repetition of the same tired ideas. If you haven’t had a chance yet, go read my January 2009 informIT column “Top 11 Reasons Why Top 10 (or Top 25) Lists Don’t Work.”

There are some important improvements in this year’s top25 list that have been discussed on sc-l. But there is also a problem that really bothers me. The SANS guys are trying to tie the top25 list to software liability (?!) and apparently think they can hold developers accountable for their bugs..well, 25 of them at least. I think this is a wrongheaded approach to software security. I would much rather talk about the real progress the field has made than to hype up yet another list and make the list a critical aspect of software contracts?! Can you imagine what such a move (if it succeeded) would do to the price of software and to the hourly rates of developers? Developers would be compensated like lawyers!

Top-n lists do have their place. In the BSIMM we note 10 firms (of 30) who follow activity [CR1.1]. Here is the activity cut from the BSIMM:

Create a top N bugs list (real data preferred). The SSG maintains a list of the most important kinds of bugs that need to be eliminated from the organization’s code. The list helps focus the organization’s attention on the bugs that matter most. A generic list could be culled from public sources, but a list is much more valuable if it is specific to the organization and built from real data gathered from code review, testing, and actual incidents. The SSG can periodically update the list and publish a “most wanted” report. (For another way to use the list, see [T2.2] Create/ use material specific to company history.)

In my view, a tailored top-n bugs list is way more useful than a generic “world list” like the SANS/CWE25. To think about why this is, consider the differences between code bases from Intel, Microsoft, Symantec, and Nokia (not to mention Wells Fargo)…all BSIMM participants. Whose bugs do you want to eradicate? Yours? Or your neighbors?

Press coverage of the “controversy”:

• SANS Institute, MITRE release new top 25 dangerous coding errors list, TechTarget.
• Top 25 Programming Errors: Should Software Developers be Liable?, Bank Info Security.
• Hold vendors liable for buggy software, group says, ComputerWorld.
• 25 ways to better secure software from cyber attacks, Scientific American Observations.
• Security agencies release Top 25 programming errors, Washington Technology.
• Proposal Would Hold Software Developers Accountable For Security Bugs, Dark Reading.
• Group Proposes Suits Over Faulty Code, Gov Info Security.

The BSIMM study data set has more than tripled in size and now includes data from 30 firms. We are busy working with Betsy Nichols to crunch the numbers now that we have a statistically significant data set. The plan is to announce our results at RSA.

One question that comes up in the BSIMM work fairly consistency is the difference between BSIMM and other maturity models for software security. To answer that question, I wrote an article for informIT entitled “Cargo Cult Computer Security: Why we need more description and less prescription.”

Download audio [mp3]

David Rice, the author of Geekonomics (as well as the 46th Silver Bullet Security podcast victim), and I discuss the BSIMM in a webcast about the upcoming SANS software security event in San Francisco.

The time for science is upon us. And the first step in any scientific approach is measurement.

The BSIMM study focuses attention on software security in large organizations and just at the moment covers the work of 1554 full time employees working every day in 26 software security initiatives. One phenomenon we observed consistently in the BSIMM is that every large initiative has a Software Security Group (SSG) to carry out and lead software security activities.

I wrote about our observations around SSGs in my December informIT article.

Simply put, an SSG is a critical part of a software security initiative in all companies with more than 100 developers. (We’re still not sure about SSGs in smaller organizations, but the BSIMM Begin data (now hovering at 75 firms) may be revealing.)

Cigital’s SSG was formed in 1997 (with John Viega, Brad Arkin, and me as founding members). Since its inception, we’ve helped plan, staff, and carry out ten large software security initiatives in customer firms. One of the most important first tasks is establishing an SSG.

Today we officially launch BSIMM Europe, a study of 9 EU firms’ software security initiatives. We continue to focus our inital data gathering on large-scale software security initiatives at major software firms. Firms in the study include: Nokia, Standard Life, SWIFT, Telecom Italia, and Thomson Reuters.

An informIT article can be found here.

The article describes our findings regarding European software security by contrast with the original BSIMM. Overall, we have tripled the size of the BSIMM study to 27 firms with several more under way. We hope to reach 30 firms by year end.

We released BSIMM v1.5 as part of the BSIMM Europe push. The document (released under the Creative Commons) is available for download and now includes an appendix about BSIMM Europe. The original document (v1.0) has been translated into Italian (by Minded Security) and German (by Virtual Forge).

We are very excited about BSIMM progress and look forward to sharing more real data with the community. No more faith based software security!

It really feels like software security, as a discipline, has made great progress over the last decade. To begin measuring what firms are actually doing to make software security happen, Gary McGraw, Brian Chess, and I last year interviewed the executives running nine software security initiatives, using the twelve practices of the Software Security Framework as our guide. We used the resulting data to guide construction of the Building Security In Maturity Model (BSIMM). A maturity model is appropriate because improving software security almost always means changing the way an organization works–people, process, and automation are all required. While not all organizations need to achieve exactly the same set of software security goals, our experience is that all successful software security initiatives share common ideas and approaches. Regardless of the details of your software development lifecycle, there is much to learn from the practical experience of others.

Since the original surveys, we’ve continued to gather data in formal interviews. And, of course, more data is always better.

But, we’d really like lots more data. In that light, I’d like to announce the BSIMM Begin survey sponsored by Cigital. BSIMM Begin is a questionnaire designed to probe a firm’s progress relative to the level one BSIMM activities. It is also an experiment in self-reporting. While we exercise great care when performing in-person formal interviews, we realize that approach doesn’t scale into the hundreds in any reasonable time frame. We’re hoping that self-reported data allows for the level of analysis that will provide meaningful results to everyone in the community and, perhaps more importantly, to those participating in the survey.

If you would like to participate on behalf of your firm, please go to http://bsi-mm.com/begin/.

Thank you very much.

Ben Worthen broke the BSIMM story on wsj.com as was posted earlier.

I was shocked when someone said, “Oh and ASVS is also available, great” on an OWASP list. Super, I thought, but I don’t understand the connection. When I looked at the WSJ site, I noticed Jim Manico (of OWASP, Aspect, and ASVS fame) wrote, “But for those of your programming web applications, consider looking at http://www.owasp.org/index.php/ASVS – it is focused specifically on web application security evaluation.” I know Jim, he’s a good guy, and my curiosity about where the link between maturity models and verification standards was sated, but I thought I’d spend some time here quickly disambiguating them:

Verification standards, like ASVS, enumerate techniques with which an application’s correct use of security controls can be verified. It also posits (what it calls) verification requirements that serve as specifically enumerated tests of particular security controls (such as input validation). Such a standard is best operationally deployed by an organization’s application security group. In some cases, the audit group may own verification and reporting.

Auditors especially love NIST’s 800-53 standard, which recommends security controls for Federal Information Systems and Organizations. This document has frustrated me historically because of its focus on OS configuration, topology, and patch level. Largely, it ignores the application layer of an organizational stack. As such, organizations applying it have found little impact in preventing the web’s most common application security vulnerabilities. I reckon one aim of ASVS was closing this particular gap.

Regardless of what you think of ASVS or NIST 800-53, one applies the bulk of their guidance to an application and systems.

A maturity model such as BSIMM measures what an organization is doing to secure their software. Organizations will be graded on whether they’ve defined and adopted assessment (verification) techniques, but also as to the state of other organizational constructs such as a training program, post-deployment incident response capabilities, and specific application security management. A maturity model helps organizations place their personnel, use of tools, and practices against industry best practices in a broader context (a software security framework). Verification of applications is only a small piece of that governance activity. As such, BSIMM data is produced, consumed, and managed at the CISO/CIO level, rather than within the application security group (as ASVS).

Organizations will need an ability to consistently and comprehensively verify the security of their applications but this is only a piece of what those same organizations will also need to do, in a more broad context, to make sure they reduce, other-wise manage, or transfer software-induced business risk.

To this extent, comparison of BSIMM to ASVS is even a poor fit than comparing the Rational Unified Process (RUP) to Capability Maturity Model (CMM). -shiver-

The first phase in our endeavor to bring some science to software security is at a close. Our science-y approach started with some anthropology several months ago. We asked nine firms to tell us about their software security group (SSG), its inception, its activities, and the success it has achieved. The result is the Building Security In Maturity Model authored by Gary McGraw, Brian Chess (Fortify), and me, which is out for public use at http://bsi-mm.com. We’re very pleased that the Wall Street Journal broke the news of the BSIMM release.

Please take a look at BSIMM. If you run or are active in a software security group, look at it like a yardstick. Consider the activities listed versus what your organization is doing. Look especially closely at the things all or most organizations do (e.g., http://www.informit.com/articles/article.aspx?p=1326511). If your SSG is not doing those things, you might want to consider them. If your organization needs an SSG, you should be able to use the same activities to get one started.

I want to emphasize that we could not have done this without active participation by the nine firms we interviewed. The data in BSIMM is their data. Data from the interviews we conducted were used to build the model from scratch. The examples included with the activities are real examples. After building BSIMM, we scored each organization using it. The individual scorecards, although unreleasable, are fascinating. They provide a unique glimpse into how local culture, perhaps as much or more than business imperatives, drive the approach to software security. Suffice it to say, for now, that the carrot is once again shown to be mightier than the stick.

I’ll talk more about the BSIMM and individual topics over the coming weeks.

As a final note, BSIMM is a data-driven model. The model will improve when more real-world data is added. If you’d like to discuss how to get your organization’s data into the model–and the comparison of you against others back out–please let me know at smigues -at- cigital.com.

Technorati Tags: BSIMM