The BSIMM project describes and measures the work of 786 SSG members, who together with a satellite of 1750 people, have direct impact on the work of 185,316 developers. (Download a copy today and get your firm involved in the BSIMM Project.)

The BSIMM is mostly about SSDL activities and governance. However, third-party software plays a major role in all of the BSIMM firms and is an important risk factor that must be managed. In addition to talks from member firms, the BSIMM Community Conference also featured a workshop on third-party software and security.

Sammy, Brian, and I wrote up the results in an informIT article that was posted today.

The interesting aspect of our workshop was that it was made up approximately of 50% software vendors and 50% financial services firms. This made for a very interesting conversation around vendor control.

We can’t tackle the “process improvement” problem with our open source provides like we can with those who build our software in-house, or those vendors from whom we acquire code.

Understanding, based on this, we lack as many ‘knobs and dials’ for improving the cleanliness of the pipes through which open source software flows… we may have a flow rate problem as well. Though I lack hard evidence, I’d bet this represents a proverbial iceberg’s tip:

An organization may deploy as much or more open source code as their own in-house developed code.

It’s interesting to think about the money organizations spend to secure the code they build vs. the amount of open source they consume in this light… (participants indicated they didn’t track this spend separate from their development efforts).

Harumph. Our breakout group generated great ideas worth sharing. First, we unearthed a lot of things attending organizations already do. Next, we brainstormed valuable next steps. Here’s categories of activities we came up with:

• Inventory & Inventory Control


• Vulnerability Identification (Assessment)


• Vulnerability Management


• Ownership of Open Source


• Policy (use)


• Policy (Contribution)

What did each entail?

Inventory & Inventory Control

• Identification – All Participating organizations had a manual discovery process for open source usage. Many wanted better automated schemes, despite some existing tool usage.
• Identification of masked open source – Many participating organizations realize that not only do they adopt open source software directly but that the 3rd party code (and open source) they absorb also contain open source software. Discovering this ‘masked open source’ software represents as big a problem as identifying the open source software used directly.
• Centralized open source distribution – Some organizations allow development & deployment of open source software from the web directly whereas others only allow access to and use of open source from a centrally managed repository. Centralizing deployment usage may provide only improved integrity, or may be used to implement an ‘approved package list’.

Vulnerability Identification (Assessment)

• Using assessment tools – About three-quarters of workshop respondents used the same tools they use to assess their application security posture on their open source assets. This, to me, seems worth its own blog entry. I had to wonder aloud, “as organizations move from detective assessment schemes (SCR, PT) to so-called preventative (threat modeling, architecture analysis, misuse/abuse cases) how do they consider open source?” I’m finding that organizations blazing trails into security architecture work commonly omit discussion of their open source frameworks.

Ownership

• Maintain a white list of open source – Seemingly related to the “centralized distribution” item (but surprisingly uncorrelated in our survey), this meant that someone, from security, owned assessing the risk and proffering a “thumbs-up” or “thumbs-down” that can inhibit white list membership.


• Revisit white list – Organizations expressed that they found value in pruning the approved list of open source software based on non-use, newly identified risk, and similar factors. About one-quarter of our group engaged in this activity.


• Ownership of identified risk – Some participants avidly encouraged use of open source within their applications. In these organizations, when a developer chooses to include open source (as opposed to writing a widget themselves), they own any newly identified risk when in that open source. This reminded me eerily of Wall St. traders. Equity investment creates risk. Margin calls create leveraged risk. In this metaphor, choosing to adopt open source seems like a margin call. It’s very possible that a developer can absorb more risk into the organization than they themselves could effectively own up to in black-swan scenarios. It’s unclear how to measure this exposure when adopting open source.
• Collaboration – Certainly an organization-specific and an unsolved problem, participants indicated there may be a “third stakeholder” in the process of identifying and managing open source vulnerability. Two examples given were 1) clearing houses from which organizations purchase open source software and 2) support organizations (a la RedHat).

Vulnerability Management

• Root Cause Analysis – When a vulnerability is found (regardless of means or source: internal/external), organizations sometimes can point to a person or group who understands the vulnerable component (notable exceptions exist for purchased software or that software maintained by a development team that’s vanished). In open source, the organization must expend resources in order to “get smart” on vulnerability’s root cause and make trade-offs about mitigation strategies and their impacts. This represents extra cost on which I’d enjoy having greater visibility.


• Vulnerability Impact Analysis – Almost every participant had some regime by which they discovered (through assessment, feeds, or other means) new vulnerabilities within their adopted open source code base. Everybody possessed some ability to figure out which lines-of-business or development teams might be affected by newly discovered vulnerabilities.


• Patch Management – Only about one half of participants had, in their minds, a good strategy–having assessed the impacted teams–for distributing a patch that remediated open source vulnerability in an organization-wide manner. More strategically, several schemes seemed available to organizations beyond the straightforward “penetrate-and-patch” loop here. Alternatives included:
  
  
  1. Wrapping open source
     
  2. Hardening open source (and centrally distributing)
     
  3. Sandboxing/compartmentalizing open source
     
  
  

Policy (Use)

• Security Policy/Standards – About half participants had some kind of security policy or standards addressing how to securely use open source software within the organization.
• Training – About one quarter of participants trained their developers to use some portion of their open source securely. Interestingly enough, the one quarter of our respondents that trained developers did not line up well with the one half that had security standards. Wild.

Policy (Contribution)

• Legal permission to contribute to open source – Some organizations see open source contribution as a key activity. Others did but have suffered clamp-down from their legal departments because legal fears liability. Others never liked the idea.
• Community Notification on Vulnerability – When an organization contributes to open source and later finds (or is notified of) vulnerability in its contributions, it may need a way to notify the broader community. Organizations also complained about very high latency in the community notifying them of vulnerability in their code. This proved a surprising problem in our brainstorming session. Why? Contributors complained that this was because, often, their contributions were either 1) forked or 2) baked into other products that masked use of the contributions. In either case, it wasn’t evident to those disclosing the vulnerability that the (contributing) organization was responsibly for vulnerable code.

I will absolutely not let it go without saying that though this entry contains many of my own thoughts it heavily relies on the work of many in our breakout session, well-lead by HP’s Brian Chess. Thanks all for a great discussion.
-jOHN

Since the first BSIMM interview in October 2008, we’ve progressed from nine to 30 to 42 firms (and more, at this point). We’ve also measured 11 firms twice—about 19 months between measurements on average—and that has provided the BSIMM community with some unique insight on how software security initiatives change over time. Assessing 42 individual firms and performing 11 re-assessments required 81 sets of interviews in just a shade less than three years.

For my money, that’s not bad for a backyard project.

Of the 42 firms in the data pool, 27 have graciously allowed us to name them as BSIMM participants. They are: Adobe, Aon, Bank of America, Capital One, The Depository Trust & Clearing Corporation (DTCC), EMC, Fannie Mae, Google, Intel, Intuit, McKesson, Microsoft, Nokia, QUALCOMM, Sallie Mae, SAP, Scripps Networks Interactive, Sony Ericsson, Standard Life, SWIFT, Symantec, Telecom Italia, Thomson Reuters, Visa, VMware, Wells Fargo, and Zynga. To these and the other 15 firms, thank you very much for participating. You are directly responsible for advancing the cause of software security.

The BSIMM3 document is freely available under a Creative Commons license. You can get it from http://bsimm.com. Go ahead; it’s a good read. Even if you’re down the road with your software security initiative, you can get a glimpse into the actual software security activities conducted by your peers and competitors. If you’ve yet to get started, BSIMM will give you some great ideas.

As always, we are looking for more people who are interested in participating in the BSIMM study. We’d love to hear from you.

–Sammy.

Dr. Markus Schumacher has served as CEO and Co-Founder of Virtual Forge GmbH since 2006. The company specializes in the security of SAP applications. Dr. Schumacher was previously a representative of the Fraunhofer Institute for Secure Information Technology (SIT) and worked at SAP as a Security Product Manager (NetWeaver). Focus topics were secure development, security testing, security response, product certification (Common Criteria) as well as awareness events for the development crew. Before SAP, Dr. Schumacher, was a member of the scientific staff at the IT Transfer Office (ITO), Department of Computer Science, Darmstadt University of Technology, where he managed projects for customers such as T-Systems Nova, Siemens AG, SAP Corporate Research, and Fujitsu Laboratories. Dr. Schumacher earned his doctorate in computer science field. He has published numerous articles and books (most recently: Secure ABAP Programming at SAP Press) and speaks regularly at international conferences.

Markus met with Gary during his latest stay in Germany. After talking about software security in certain nice places in Heidelberg, the idea came up to capture some insights about software security testing in an interview. Here’s the interview as recorded on Wednesday, April 6, 2011.

Markus: Gary, we talk about software security today, in particular about finding bugs by thorough security testing. How should tests be conducted? Manually or with a tool? And which approach is better?

Gary: This is a little bit like comparing apples and oranges because both approaches can be very useful. But generally speaking, if you can automate a paticular test that means that you’ll be able to apply that test consistently in the future—maybe even across your entire code base. So I’m a big fan of automating as much of testing as you can automate.

Security testing is good, but if people treat it as a ‘security meter’ that can lead to real problems. That is, confused people sometimes think that if they run automated tests and don’t find any problems that the software is free of bugs. But we both know that a result like this just means that you haven’t found anything interesting during a given test. You have to be very careful when you apply automated testing that you know what you are doing and that in the end you know what the results are.

Does that make sense to you?

Markus: That makes perfect sense. You have observed the software security tools market for many years. We see black box scanning tools and code scanning tools out there today; what are the trends that you have observed?

Gary: When it comes to software security, there are basically two kinds of automation. One kind does black box testing and requires that your software be run. We call that dynamic testing. The idea is to test your program automatically while it’s running by providing input and see if you can maliciously break it. There are such tools aimed at Web applications, IBM AppScan for example. The second type of tool is a code scanning tool that does a static analysis. A code scanning tool looks at your code instead of running your program. That is, it looks for bugs that are observable in the code itself.

Both of the two types of tools are the biggest sellers in the software security space today. What happened over the last few years is that the code scanning tools became a lot better and they’ve begun to find widespread use. In fact they accelerated past the black box testing tools in terms of adoption a year or two ago. The reason for that is that black box tools only work for Web applications (they work only over http) while the source code tools work for any kind of software. As you know, there are even specialty white box tools that look over particular languages—like the languages that are built into highly popular systems like SAP.

The ABAP tool that you guys have built is a way of looking at your ABAP code to find bugs and produce security results. In my view it’s really better to focus as early in the lifecycle as you can to find bugs and any static analysis tool can really help to do that. Bottom line: here are many advantages using such static tools over and above dynamic tools.

Markus: Because you build security in from day one.

Gary: That’s part of the idea. Of course you have to think about your design as well. But we haven’t figured out how to automate looking for design flaws yet!

Markus: Don’t write software – then you are good.

Gary: <laughs> Sadly, that’s true.

Markus: Code analysis tools are obviously a good choice. But what are their limitations?

Gary: There are a couple of things that are problematic. One is that people think that the tool will find all possible bugs and then fix the bugs for you. That can be an issue. The thing about these tools is mostly they help you finding possible vulnerabilities and then you have to be smart about determining whether what has been found is a real problem or not. But even more important than that, thinking about how to fix vulnerabilities is a serious problem. If current tools have a limitation it’s that they don’t fix the code, and certainly not automatically! So they are great at finding bugs but it’s up to you to fix them. If you just use these tools to find bugs, pile them up somewhere, and not fix them that doesn’t help security at all.

The other problem with these tools is that because they are doing static analysis they do have the tendency to sometimes find false positives—things that the tool thinks are a problem but it turns out when you think about data flow more carefully (or whatever) they are not a problem. Plenty of people worry about the false positives problem, but I have seen the number of false positives that static tools produce over the last 5 or 7 years drop dramatically. It’s in an acceptable range now I believe.

Markus: I have talked to different clients about false positives. One of them said, ‘tools find issues – some might be false positives, others not – we review them and fix the bugs.’ Others say, ‘for many reasons – I can’t have any false positives even if the tool is sometimes finding real bugs.’ For them it’s better to not see a real bug in favor of a low false positive rate. What would you say to the latter?

Gary: “I’m with the first guys. It’s much better to have a few false positives and find all of your security problems than it is to have no false positives and miss real security problems. This is because security problems are serious and they need to get fixed!

The notion of a code scanning tool sprang from a whole bunch of experience with manual code reviews—digging through code by hand and looking for security bugs. We were doing a lot of that in 1998 and 1999 and we began to figure out a ways to automate parts of that. We created the first code scanning tool for security called ITS4. Things have come a very long way since then, but remember that ITS4 was just using grep-like technology looking for very simple patterns and sometimes you can get simple patterns completely wrong.

Things have improved a huge amount since those days. I think when people talk about false positives in some sense they are using thinking that is about 10 years old (from the ITS4 days). Today the false positive rate has dropped enough that using these tools is something you really just have to do.

Markus: Our strategy of lowering the false positive rate is to apply data-flow analysis consistently, doing many sanity checks like type checking, looking for authority checks, etc. That way we classify the findings – there are certain findings where we are pretty sure will always find real bugs while others are probably not as certain and get a lower rating …

Gary: I think that’s a very good idea.

Markus: … Is this approach a good strategy? That is, starting with the findings that have a very high rating first?

Gary: Yes.

Everyone has a limited amount of time to fix their code. The most important thing is not finding the bugs, but fixing them as I have told you before. If you have a way of helping people prioritize the fixing so that they are fixing stuff that really needs to be fixed, that’s fantastic!

What we see in the field is a lot of people find a lot of bugs but not enough people do enough to fix the bugs. There’s not enough remediation going on. Let’s be clear: it does no good to find bugs if you are not going to fix them. And so I think a focus on telling people ‘this is a bug for sure, and you should fix this one because you won’t waste any of your valuable time’ is a very, very clever strategy.

Markus: We know people who say that such ‘very high’ findings are very likely true positives and consider all others with a lower rating as a false positive because they need to invest too much time on finding out whether they are bugs or not. Accordingly they claim that the false positive rate is too high and a tool might be useless because it doesn’t deliver 100% hits only. Why is it not a good idea to shoot at this false positive thing only?

Gary: If these people are fixing all of the bugs that you are telling them are bugs for sure and have extra time left over, then they can worry about that problem! <laughs> But so far I haven’t seen anybody who has the luxury of that much time. That means their whole point is sort of a moot point. The answer should be: fix the ones that you know are a problem, and when you are done with that we’ll talk.

Markus: Good answer, next question.

Many people get frustrated when they start security testing because of the high amount of findings as result of initial scans. How should people approach this?

Gary: The best way to do this is to turn the things that you are looking for on and off inside the tool. When you try to get people to adopt a tool for the first time, it’s better to have the tool looking for certain categories of bugs (I recommend this be as few as possible). The idea is to make sure that the tool doesn’t just overwhelm the user with a big ‘red screen of death.’

There are a couple of clever ways of doing this. We help many companies adopting such tools wisely throughout their whole development team. One very good trick is to tie the tools to code that the users want to use already. So you have a middleware framework and you want people to use that, then you build some enforcement rules to talk about the use of that particular code, and you focus on that instead of focusing on looking for all bugs at all time throughout the entire code base.

Another way of putting this notion is: tighten the focus of the tool so that it isn’t overwhelming at first, and then loosen that focus up, add more rules, add more kinds of bugs you are looking for over time. Start small. As the code base improves and people get better in using the tool, do more.

Markus: We have a customer following a similar strategy. They did an initial scan with all checks turned on. Then they identified all checks that lead to no findings and made those tests mandatory. Meaning: they are good in this area and they won’t get worse. And then they tightened the focus as you have described it. Like it?

Gary: That’s a good idea, because it’s sort of belts and suspenders approach (so to speak). The idea of working for certain categories of bugs should also be complemented by understanding your code base. If you run a bunch of static analyses, you should amass enough data to determine what your number one bug is. Note that your number one bug may different than somebody else’s number one bug! Then you can set out on a bug eradication mission based on real data from a tool run over your code base, and that’s a very helpful thing.

Remember, if you are finding bugs in your code that means somebody is typing in those bugs— somebody actually wrote that bug. The best thing is to get to that person and teach them not to do it that way. The closer you can get this to the developer’s head (and fingers) the better off you’ll be in my experience.

Markus: But that could be the reason for the resistance. Somebody blames the bug writer for their bad code, their (broken) piece of work. And probably companies do not have a well-developed way of dealing with accidental mistakes.

Gary: That’s right. One problem in security that we have is that developers like their code and treat it like it’s their baby. Then you come along and say, ‘That’s the ugliest baby I’ve ever seen!’ And that makes the developers angry. You really shouldn’t call somebody’s baby ugly, but in security we run around doing that all the time.

We have to understand that people are very sensitive about their code, and we have to be gentle about security problems and teach them that it’s in everybody’s best interest to find and fix these things. The good news is that most developers actually really want to build good stuff. If you say, ‘This is for helping you build better stuff. It’s not something to smack you around and make you look like an idiot, in fact it makes you build better code,’ that fits into the development culture way better.

Markus: Stay away from the ugly-baby guys and support the better developer. I like that.

Another thought on false positives. Sometimes people say that a certain finding is a false positive because there’s no data path to the vulnerability or the code touches non-critical data only. Think of a SQL injection in code that handles temporary data only. A tool cannot make a good decision here. What’s you view on this?

Gary: The answer is a bit convoluted. Because of code reuse and because people will repurpose code in surprising ways, it’s always better to fix those problems. Even if you think that in a particular situation a particular vulnerability might not lead to a security issue. Because odds are high that someone will just cut-and-paste it and use it somewhere else. And then it will be a real problem.

Markus: Cut-and-paste is one thing, another is code that is part of an API, function, or report, that might be used by someone else in a different context.

Gary: Absolutely right. That happens an awful lot.

It’s the same as putting a watchdog in code. I have seen people put a watchdog way at the beginning of code looking for certain kinds of input because there’s a vulnerability way down low in the code and they say, ‘if we strip the input so it never gets down there everything will be fine.’ But then later somebody comes along and creates a new execution path to the same vulnerability with the watchdog so far up there that the flow is no longer controlled by the watchdog anymore. Then you’re screwed. That’s sort of the same idea. Bottom line: if you have a bug in your code, you should fix it.

Markus: Period – nothing to add here, just fix it.

Final question. You’re currently work on BSIMM3. What can we expect in the new version?

Gary: We have continued to grow the size of the BSIMM study. We now have now 33 firms in the study and we have done 60 measurements.

What happened last year was kind of surprising. Many of the firms that were already participating in the BSIMM asked us to measure their major divisions. For example we did six measurements inside of Bank of America. If you know that the Bank of America includes Merrill Lynch, Countrywide, and a bunch of other large financial organizations, that’s not such a big surprise. That meant we spent an awful lot of time doing BSIMM analysis inside firms that were already in the BSIMM.

So we have grown the dataset considerably—doubled it, in fact, since BSIMM2.

The other thing that we have started doing is re-measuring firms that we have already measured in the past. We have measured 10 firms already again. So now we have data that show what happens to a software security initiative over time, and we can talk about what changed between the first and the second measurements. That’s incredibly cool, very powerful data.

Our plan for BSIMM3 is to try to get up to 40 firms and then release the longitudinal data (that is, the data over time) and the new data set with 40 firms all at the same time. I’m hoping to do that in the early summer.

Markus: Is there hope? Are things getting better?

Gary: Things are getting better. 15 years ago nobody really cared about software security. When Viega and I wrote Building Secure Software everybody thought we were crazy. A lot has changed since then. Now, developers are beginning to understand that what they do does have a clear impact on security. And a lot of firms are realizing that their customers are expecting the code to be secure. Customers may not really be explicitly saying ‘this has to be secure’ but they do (implicitly) believe that it already is secure! So it’s really important that firms meet the implicit security expectations of their customers. A lot of firms are realizing that.

As a field we have made a huge amount of progress. The other thing that happened in the past 10 years is the rise of static analysis tools that actually work and can be adopted in large enterprises. And finally the BSIMM project is a relatively new venture—we have only been doing the study for a couple of years. The BSIMM is a scientific approach that relies on effective measurement of a firm and its peer group. That way you can compare and track what different many diverse firms are doing. That’s a very, very powerful thing. So we built a community of like-minded firms who are all working very hard and building up software security and are making great progress. We figured out a way to measure that progress and show it in no-uncertain terms. That’s pretty cool.

Markus: Agreed. And we continue supporting BSIMM by translating it to German.

Next time we will talk about our joint invention, the NoMoRed (No More Red traffic lights) tool that deletes all bugs by just clicking a button. <laughs> I’m looking forward to that. Thank you for your time today.

Transcribed in Heidelberg on April 6, 2011.

Cast (in order of appearance)

• Markus Schumacher, CEO of Virtual Forge GmbH
• Gary McGraw, CTO of Cigital, Inc.
• One Web application scanner: IBM AppScan
• The ABAP tool that you guys have built
• The first code scanning tool for security: its4 (it’s the software stupid)
• The BSIMM
• Building Secure Software, Addison-Wesley Professional, 2001

you will find our observations regarding static analysis shared by others. It’s encouraging to note that Flash Sheridan observes many of the same difficulties and more formally treats them in his ISSRE ’10 publication. It’s worth a read.

A few commentators shared some decent points I wanted to address in a comment. I’m cross-posting my commentary here:

Comments treating the application of static tools as a distinct (and unfavorable) alternative to “fixing your SDLC” disappoint me, especially from the OWASP community (of which I am a contributing paid member). In fact, the community’s OpenSAMM project actively prescribes use of static analysis in not one but four (4) unique activities. So, both BSIMM and SAMM see static analysis as a part of fixing one’s SDL. **Note that in both models, all activities are not compulsory.

“Fix your code” is another red-herring alternative to static analysis. I believe static analysis inseparable from effective usage of a secure programming toolkit (such as OWASP’s ESAPI or an organization’s own toolkit) because static analysis tools can meet the need for organizations to 1) detect consistent and thorough use of secure APIs (and non-use of dangerous ones) as well as 2) detect incorrect usage of such APIs (whether through broken call-order, un-paired functions, or other bugs). We shouldn’t forget, as Mr. Manico indicates, 3) running static tools on these security toolkits themselves finds problems that careful review may not otherwise have found.

So, secure SDL models indicate usage and we–as a community–see use in these tools in helping remind developers how to correctly build secure code. I think we have to agree static analysis tools find exploitable bugs (even though sometimes in an overly expensive or time-consuming way). So, I think we’re stuck with them as a ‘best practice’ in the short term, however little some of us may like them. To dynamic vendors out there, I’d even go so far as saying, “Early adopters have begun to over-rely on static means to find problems more effectively/efficiently found with dynamic tools. Don’t worry, the ones I’m working with will begin to tilt the balance back using assessment data”.

I would like to put a face-and-name on the key notion that almost all commentators have thus-far touched: the static analysis tool that gets used by a majority of application developers out there, in the future, will likely not look like the ones we have today. And that’s fine too.

We just hosted the first ever BSIMM Community Conference in Annapolis, MD this week. I’m proud to say it was a smash hit. The schedule was packed full of interesting talks from leaders among the BSIMM Community including Microsoft, Intel, Salie Mae, JP Morgan Chase, QUALCOMM, Fidelity, Adobe and Cigital, but by far the most important aspect of the conference was the incredible energy generated when a room full of like-minded professionals get together. Twenty of the 32 BSIMM firms were in attendance (each represented by 3 people). My favorite part of the conference was watching attendees eyes light up as they met their peers from other organizations.

The power of the BSIMM was on full display. We discussed the BSIMM’s utility as a measurement tool, as a way to compare progress in software security among firms and vertical markets, as a tool to help position and fund an initiative, and as a set of data to inform strategic software security investment decisions. Science is catching on in software security!

The BSIMM continues to evolve and grow as more measurements are made. Counting major divisions in some of the firms, we have made 57 distinct BSIMM measurements over the last two years. We’re beginning to re-measure some of the original participants and understand how software security initiatives change over time. Expect publications about that work (including lots of actual data) in the Spring.

If you are interested in joining the BSIMM Community (which entails being measured by objective observers, joining the moderated mailing list, and attending BSIMM Community Conferences to come), please don’t hesitate to contact us.

For more about the BSIMM project and to download a copy of the model itself, see the BSIMM website.

There were (at least) two basic approaches to gathering the software security activity data we were looking for. One was to ask what everyone was doing and then deal with the sorting and scoring issues presented by an avalanche of ad hoc data. Alternatively, we could ask about a few of the activities we had already seen in our controlled BSIMM surveys (http://bsimm2.com). We did the latter. On one hand, had we asked general questions about if and how software security was getting done, we might have amassed more data (assuming there were specific software security activities taking place). On the other hand, I believe we would have acquired a lot of data that had little to do with software security and much more to do with security features in software and with IT security.

Because we wanted some insight into a substantial number of data points, we asked a substantial number of questions. Because we didn’t want to ask overly-simple (and transparent) yes/no question about each data point and we wanted to have some confidence (intellectual, not statistical) in the results, we asked two or three questions about each data point. In the end, we asked a bunch of True/False questions, but did not include a Not Applicable option. That was a methodological mistake on our part and several of the respondents let us know that. Also, because we didn’t want the “right” answer to be “True” in every case, we “negated” some of the questions with a strategically placed “not” that really confused some respondents.

Overall, this resulted in a daunting survey that, anecdotally, took a full hour to complete, assuming the person taking the survey actually knew all the answers for his or her organization. If the person didn’t know all the answers, they had to stop and go discover some. Unfortunately, SurveyMonkey doesn’t really lend itself to stopping and starting surveys, so some people simply gave up. A couple of respondents let us know this somewhat colorfully.

We asked 110 questions to probe different aspects of 40 software security activities from the BSIMM—all of the Level 1 activities across 12 BSIMM practices and the few most common activities from Level 2. We received 93 responses, 48 of which were complete enough to allow useful analysis. As an aside, three of the 48 (and 22 of all) respondents declined to provide their email address, preventing us from sending their survey results.

Here are some general statistics on the data pool from the 48 respondents:

These are some fairly large numbers. Even though we went well out of our way to get smaller firms to respond to the survey, we had very little success in that area. One possible knee-jerk reaction is that there simply isn’t any organizational software security initiative in most of the smaller development organizations, but I don’t believe there’s enough data to make this a defensible conclusion.

The following chart shows the average software security activity level determined via the 48 responses. Within this small data set Penetration Testing and Software Environment saw the most activity, closely followed by Code Review, Standards and Requirements, Training, and Compliance and Policy. Security Testing and Configuration Management and Vulnerability Management were noticeably less populated.

In defense of this survey, I’ll note that a handful of participating full BSIMM firms graciously took the time to complete the questions as a scientific control. There was indeed some correlation between their self-reported data and the data we had meticulously gathered through in-person interviews. Informally, however, the results did seem to indicate that self-reported data result in consistently higher scores than results gathered in-person. That is, and as I’m sure is applicable in most walks of life, our own view of whether we are getting something done won’t always match that of an unbiased outsider.

The upshot of all this is that we will step up our efforts to do more in-person interviews in a full BSIMM capacity. The consistency in the data gathering and scoring will make the results more useful to everyone as a benchmark and a yardstick. If you’d like an understanding of where your software security initiative stacks up compared to others by participating the the BSIMM, we’d like to hear from you.

As a last point, we noted through personal knowledge or bounce messages that approximately 20% (10 or so) of the 48 respondents in the usable data set no longer worked at the same place they had just a few months earlier.

As always, feedback and suggestions are welcome.

BSIMM2 is available for free under the creative commons license from bsimm2.com. Download your copy today.

The BSIMM2 document itself is 53 pages. A concise treatment of the results can be found in this month’s informIT column in an article titled “BSIMM2: Measuring the Emergence of a Software Security Community.”

Our study represents the work of 635 people who are members of the 30 firms’ SSGs. Together, the firms have a collective 130 years of experience planning and executing 30 software security initiatives. Among other results, we have identified 15 core BSIMM activities.

We think the descriptive nature of the BSIMM study is an important characteristic of the work. We describe not what you should do for software security, but what successful software security initiatives are actually doing. Use BSIMM2 to measure your own software security initiative and compare it to others.

We’re unveiling some statistical results at RSA this week that will enhance and expand the dataset published in the article. We’ll do an official BSIMM2 launch within the next couple of months.

So why am I not a big fan of these lists? Well, I wrote that down a year ago and what I said then still applies. Sure would be nice to see a reasoned response to my criticisms instead of repetition of the same tired ideas. If you haven’t had a chance yet, go read my January 2009 informIT column “Top 11 Reasons Why Top 10 (or Top 25) Lists Don’t Work.”

There are some important improvements in this year’s top25 list that have been discussed on sc-l. But there is also a problem that really bothers me. The SANS guys are trying to tie the top25 list to software liability (?!) and apparently think they can hold developers accountable for their bugs..well, 25 of them at least. I think this is a wrongheaded approach to software security. I would much rather talk about the real progress the field has made than to hype up yet another list and make the list a critical aspect of software contracts?! Can you imagine what such a move (if it succeeded) would do to the price of software and to the hourly rates of developers? Developers would be compensated like lawyers!

Top-n lists do have their place. In the BSIMM we note 10 firms (of 30) who follow activity [CR1.1]. Here is the activity cut from the BSIMM:

Create a top N bugs list (real data preferred). The SSG maintains a list of the most important kinds of bugs that need to be eliminated from the organization’s code. The list helps focus the organization’s attention on the bugs that matter most. A generic list could be culled from public sources, but a list is much more valuable if it is specific to the organization and built from real data gathered from code review, testing, and actual incidents. The SSG can periodically update the list and publish a “most wanted” report. (For another way to use the list, see [T2.2] Create/ use material specific to company history.)

In my view, a tailored top-n bugs list is way more useful than a generic “world list” like the SANS/CWE25. To think about why this is, consider the differences between code bases from Intel, Microsoft, Symantec, and Nokia (not to mention Wells Fargo)…all BSIMM participants. Whose bugs do you want to eradicate? Yours? Or your neighbors?

Press coverage of the “controversy”:

• SANS Institute, MITRE release new top 25 dangerous coding errors list, TechTarget.
• Top 25 Programming Errors: Should Software Developers be Liable?, Bank Info Security.
• Hold vendors liable for buggy software, group says, ComputerWorld.
• 25 ways to better secure software from cyber attacks, Scientific American Observations.
• Security agencies release Top 25 programming errors, Washington Technology.
• Proposal Would Hold Software Developers Accountable For Security Bugs, Dark Reading.
• Group Proposes Suits Over Faulty Code, Gov Info Security.