The first phase in our endeavor to bring some science to software security is at a close. Our science-y approach started with some anthropology several months ago. We asked nine firms to tell us about their software security group (SSG), its inception, its activities, and the success it has achieved. The result is the Building Security In Maturity Model authored by Gary McGraw, Brian Chess (Fortify), and me, which is out for public use at http://bsi-mm.com. We’re very pleased that the Wall Street Journal broke the news of the BSIMM release.

Please take a look at BSIMM. If you run or are active in a software security group, look at it like a yardstick. Consider the activities listed versus what your organization is doing. Look especially closely at the things all or most organizations do (e.g., http://www.informit.com/articles/article.aspx?p=1326511). If your SSG is not doing those things, you might want to consider them. If your organization needs an SSG, you should be able to use the same activities to get one started.

I want to emphasize that we could not have done this without active participation by the nine firms we interviewed. The data in BSIMM is their data. Data from the interviews we conducted were used to build the model from scratch. The examples included with the activities are real examples. After building BSIMM, we scored each organization using it. The individual scorecards, although unreleasable, are fascinating. They provide a unique glimpse into how local culture, perhaps as much or more than business imperatives, drive the approach to software security. Suffice it to say, for now, that the carrot is once again shown to be mightier than the stick.

I’ll talk more about the BSIMM and individual topics over the coming weeks.

As a final note, BSIMM is a data-driven model. The model will improve when more real-world data is added. If you’d like to discuss how to get your organization’s data into the model–and the comparison of you against others back out–please let me know at smigues -at- cigital.com.

Technorati Tags: BSIMM

James McGovern recently wrote a post on Gartner’s static analysis (SA) report. Among other things, he lamented the lack of actionable guidance within the report. A lack of implementation guidance doesn’t shock me from Gartner, I can’t say I expect that from them.

I can help James and community out by giving some of that guidance myself. I’ll try to do so in a tool-independent way. This topic deserves more than a “blog entry” format (admitting Cigital’s propensity for longer entries ;-)) so if people want more information on the topic, they can ping me.

In essence, you want to know the following about static analysis tools:

1. Who is going to run it (how will owners shepherd it)?
2. How much effort is it going to take to run (what budget will I need to support it)?
3. When do I run it and how often (Where does it fit within my SDL?)
4. What it’s going to find (How will the results enter/impact my organization)?
5. What won’t it find (How does it play with the rest of the assurance ecosystem)?

[Market Adoption and Making Your Choice]
Gartner seems underwhelmed by Microsoft’s ability to effect the commercial static analysis market. Cigital has always believed these technologies would make great features of a compiler/IDE and so we’re also disappointed. They’ve got great technologies but don’t exploit them commercially. I still like their freely available FXCop but this opinion isn’t popular. Perhaps that’s because I see a lot of value to FXCop as a unit testing framework that enables security test and other people are comparing it to glossy commercial tools like Fortify and Ounce run by security analysts in a very hands-off fashion. Who’s considering the tool really does make a huge difference in how it will fare.

From my perspective, the most successful SA tool vendors have made answering the “who should adopt the tool” question more difficult as they’ve waffled on licensing models. By the seat? By the core? By the KLoC? I’ve gotten complaints from prospective tool buyers indicating that a leading tool vendor’s price was 10X another vendor’s. Before you ask “Which one?”: I’ve heard specific examples in opposite directions!

Certain tools will look better to different potential buyers. Experience has shown positive response to Coverity’s C/C++ results from developers. Application security groups like Fortify and Ounce’s products, and several years ago, I saw QA Managers in two very different firms go ga-ga over Klockwork’s tool. I attribute people’s impressions not to a product vendor focusing on quality or security but to how closely the tool vendor’s conception of how roles interface with their tools throughout the vulnerability management and software development life cycles matches the organization’s own structure. Up until a few years ago I don’t believe SA vendors had fully conceived the “developer”, “analyst”, “security manager” roles in their products. Ounce was the one exception: they’ve always staunchly believed themselves to be principally a Security Analyst’s tool. Some vendors attempted to unify roles into use of a single interface while others attempted to split the product up into different configurations for each. This created confusion and friction between some adopting organizations and the tool vendor’s license model.

In the end you want SA tools to affect as many developers as possible and to get low-latency scan results as early in software’s life cycle as possible. However, having successfully deployed tools a variety of ways, I’m convinced central deployment is the way to go. Application Security should run the tool. This will help manage rule/configuration update, as well as central measurement collection, risk management, and issue tracking. How do we get low-latency and early-lifecycle centrally? Treat the tool as a service. More on that later.

Notable exceptions to the central deployment model include business units possessing one or more of the following characteristics:

• Acquisitions of previously successful, culturally-independent companies
• High-quality product development teams
• Fundamentally different SDL, development platform, language, and toolkits

For in-unit deployment one can “run locally” but must “report centrally” supporting security goals involving measurement and risk management. And, let’s be clear: if a business unit has a wicked-good continuous integration/build-management group, they-by all means-should be tapped to integrate with the SA service. Cigital and its clients have had success in integrating both the front (code submission) and back-end (bug/issue tracking) with Fortify and a variety of open source build/CI/bug-tracking products. This maintains a central model, but reduces latency dramatically.

[How Much Effort]
I’m sorely disappointed in the honesty of tool vendor sales-folk regarding level of effort. One of my first analogies for SA tools is this: Did your company buy Mercury products for load and performance testing? Think of your SA tool like Mercury–you’ll need at least as much money to deploy and implement the tool organization-wide as the licenses cost you. Sorry.

Organizations with more than 2-300 developers should plan on the following:

• 3-4 man weeks to do initial tuning and pilot implementation
• 4-16 man hours / large application to integrate with the static analysis tool and assure appropriate configuration
• 4-8 man hours / 50-100KLoC (language-/complexity-dependent) to triage results on a new-application scan
• 1 person / year as tool shepherd (app integration, custom-rule creation, rule-pack maintenance and release, support)

Organizations get into trouble when their SA tool enjoys broad acceptance by the organization before they effectively manage the submission and results triage/documentation process. My data indicates code submission can burn 8-24mhrs / application if security groups don’t ‘remember’ (or automate) the submission and scan setup process. Security groups doing scans centrally and hand-writing code review results can waste another 16hrs documenting their findings / application.

…think about it: if your organization allots a week / tool-assisted code review and wastes 24 hours getting the tool’s first result set and 16hrs documenting findings… that only left 10 hrs to actually review the code! Since I believe that 4-8 hrs / 50-100KLoC is necessary to triage, you can imagine how deep I think such code reviews are :-P At Cigital, we’ve got dramatic benefits to depth by decreasing cost of submission and reporting. This is key to scaling SA efforts. Those adopting tools will feel overwhelmed if they don’t plan to solve submission and reporting problems before wider roll-out.

[How Often]
As often as possible. If you’ve got a Continuous Integration initiative (CI), engage those folk. If you deploy your tool purely centrally, plan to scan each high-risk application at least once / release. No, to my knowledge, there isn’t a good answer to the “can these tools do incremental (partial) scans”. You should always keep old results around though, as they’ll (along with other measures) help you understand whether more or less vulnerabilities are being caught during each scan. Remember though, every time you add a rule to the scan, your vulnerability-finding bar just moved and your metrics might be thrown off.

Ounce, Fortify, and Klocwork seem pretty good about determining whether or not they’ve found a particular issue in a previous scan or not. This includes situations where code blocks move around a bit. This means that running regression scans to determine whether or not an issue has been fixed is viable. I can’t comment on Coverity’s capabilities here.

[What's it going to find]
I’m impressed that Fortify has published its vulnerability taxonomy and a bit shocked that others haven’t produced as public a resource. I’m excited too that vendors have agreed to comply with CWE labels when they report findings. I will say what I always do on this topic though: test the tool on your own code though. You simply don’t know what it will find until you throw all your idiosyncratic build, language, toolkit, and platform nonsense at the particular tool you’ve selected.

[What won't it find]
I’m continually shocked as to how much one can find by running tool Y after tool X on the same piece of code (replace X and Y as you see fit). As I’ve written time-and-time-again, the corner cases and exceptions missed by each engine are baffling. Let me put things in perspective for you: three of my clients have reported that their SA tool deployment find 17, 33, and 50% of the total number of issues found by assessment efforts respectively. If these percentages hold for your organization, inside a coding bug realm then your job is at best 50% done having run the tool and triaged its results.

Considering that Microsoft and Cigital believe that “bugs” account for only 50% of the vulnerability space, that leaves you unaware of 75% of your application’s vulnerability at the end of triage. As I always say, “use the tool to facilitate code review and increase a reviewer’s understanding of the code–not as a code reviewer that finds bugs automatically.”

Good luck out there, I’d love to hear about your particular experiences.
-jOHN

My colleague, Ben Walther, pointed me at this post about Cloud applications and Web-app security by Rich Mogull. The title is “How the Cloud Destroys Everything I Love (About Web App Security)”. The post talks about running Web apps on a cloud platform like EC2. I’m not sure I buy into everything they say.

First, I’m not sure what Rich means by a “Web App”. To me, the term Web App describes an n-Tier application with a browser front end and some kind of backend SQL database. There are maybe some web service calls thrown into the mix. It’s the kind of applications that everybody’s been writing for the last 10 years. So, what’s going to change if I’m running this same application architecture on infrastructure that I’m buying as a service? Sure, I have to worry about all of the inter-machine communication channels because I don’t have the nice data center supplied network security. But what else?

Now when we move to cloud-based applications (environments like AppEngine or 10gen), ones that take advantage of the highly distributed nature of the code as well as the virtual environments, then we have changed the application architecture. I buy that security for these apps changes, but it’s no longer “web app” security. In these cloud-based applications, there are some different fundamental assumptions about the architecture, like no transaction serializability. But for these legacy web apps running in a virtualized infrastructure, I’m less convinced that there is a drastic change.

There are a couple of specific points made in the article that I don’t agree with:

Secure development (somewhat) breaks because the underlying platform can’t be locked down.
Just because you can’t lock it down yourself doesn’t mean that it can’t be locked down. This seems like an argument for secure deployment breaking and not secure development. Even then, the PAAS or IAAS may actually lock the platform down better than you can. It does shift the problem from looking at technical artifacts (configuration files, patch logs, etc) to looking at legal and audit artifacts (SLAs and certifications).

Static analysis tools (mostly) break.
The contention is that there’s less code you program yourself. I don’t see this as true for IAAS platforms like EC2, how much code is provided that you really need to worry about. Besides, static tools are language based and if it’s the same language, it doesn’t really matter whether it’s running on a virtual OS or a physical one. The change that breaks static analysis is the move to dynamically typed languages.

My take is that the infrastructural changes of a cloud computing have a more drastic effect on an organization’s ability to deploy securely. Being able to develop securely is based on the application architecture. I really see these as independent levers rather than a single “cloud” lever.

Technorati Tags: Cloud Computing, Security

A recent thread about hardware and software requirements for development on the Google cloud forum made me wonder what cloud computing will mean for development, test and production environments. There are a lot of really interesting questions here, but my mind got stuck on the relationship between development and production. I’ve always been amused at the IDE feature that touts instant deployment from the developer’s desktop to the production system. Wearing my security hat, I should use the word “horrified” instead of “amused” because I can think of no worse model for deployment than developer-to-production.

I came to the conclusion that deployment of cloud-based applications will require much more engineering rigor to assure the security of the deployed application. There will, of course, be variation in the amount and type of rigor based on the type of cloud the application is being deployed to. For example, deploying to a platform as a service provider will be different than deploying to a infrastructure as a service provider where the later will require a lot more rigor. But, given the amount of manual steps involved in most of the deployments I see, I’m very worried that deployment automation will be overlooked as a required security control for cloud-based apps.

Large companies will have tools like Opsware and Bladelogic for their existing data centers, so they have a base of automation to work from. It’s the medium-large and medium sized organizations that I see as needing automation help to secure their deployments. I think these companies will be adopting cloud-based applications for their cost advantage over internally owned and managed infrastructure. But these organizations have a much more collegial relationship between development and operations and moving to operations to a service model will break these relationships. It’s the same problem that companies face when moving parts of development offshore. The solution for development is more formal API contracts and specifications. With operations, automation is an excellent way of formalizing the “API contract” between development and operations.

Security will have its part in this equation as well. All of the assumptions about who can do what with which parts of the application, who you can trust and with what credentials will have to be re-evaluated. Some of my colleagues would argue that this is the least of our worries. I would agree, but given the lack of a shared understanding of cloud security, I think that these operational concerns need to be on the list.

This is a guest post from Paco Hope, Technical Manager at Cigital.

I read a blog entry about “re-aligning training expectations for QA.” It has some useful points that both developers and so-called “security people” need to hear. I disagree with some implicit biases, however, and I think we need to get past some stereotypes that sneak out in the article.

Bias #1, obviously, is the focus on the web. Despite its omnipresence, there is more non-web software than web software in the world, and non-web software does more important stuff than all the web software combined. The role of security in software testing is vital, and the presence or absence of web technologies does not change that. Despite writing a book on Web Security Testing, I know my place in the universe. Quality assurance and software testing are disciplines far older than the web, and their mission is so much bigger than finding vulnerabilities.

Bias #2 is vulnerabilities über alles. By talking about weaving vulnerabilities into security test plans, we’ve overlooked the first place where security goes into the QA process: test strategy. Look at any of the prominent folks in QA (James and Jon Bach, Michael Bolton, Rex Black, Cem Kaner), the people I’m privileged to share podiums with at QA conferences like STAR West, STAR East, and Better Software, and you’ll see that security is part of the overall risk-based testing strategy. Risk-based testing has been around for a really long time. Longer than the web.

Before anyone talks about vulnerabilities to test for, we have to figure out what the business cares about and why. What could go wrong? Who cares? What would the impact be? Answers to those questions drive our testing strategy, and ultimately our test plans and test cases.

Bias #3 is that idea that a bunch of web vulnerabilities are equivalent in impact to the business. That is, you just toss as many as you can into your test plan and test for as much as you can. This isn’t how testing is prioritized.

You don’t organize testing based on which top X vulnerabilities are likely to affect your organization (as the blog suggests). Likelihood is one part of the puzzle. Business impact is the part that is missing. You prioritize security tests by risk severity—that marriage of likelihood and impact to the business. If I have a whole pile of very likely attacks that are all low or negligible impact, and I have a few moderately likely attacks that have high impact, I should prioritize my testing effort around the ones with greater impact to my business.

Bias #4 is the treatment of testers like second class citizens. In the blog article, developers are “detail oriented” have a “deep understanding of flows.” Contrast this with QA who merely understand “what is provided to them.” They sound impotent, as if all they can do is what they’re told. Software testing, despite whatever firsthand experience the author may have, is a mature discipline. It is older and more formalized than “security” as a discipline. Software testing is older than the Internet or the web. If software testing as a discipline has adopted security too slowly, given security’s rise to the forefront in the marketplace, that might be a legitimate criticism. But I don’t approve of the slandering QA by implying that they just take what’s given them and execute it. QA is hard and there are some really bright minds working in that field.

As someone who has been training in risk-based security testing for several years now, I totally agree with some points, but very much disagree with others. I agree that the “bug parade” (as we call it) of top X vulnerabilities to find is the wrong way to teach security testing. Risk management, though, has been a fundamental part of mainstream QA for a very long time. Likewise, risk management is the same technique that good “security people” use to prioritize their results. Risk management is certainly how the business is going to make decisions about which issues to remediate and when. Risk management is what ties this all together.

If there’s something that QA needs to learn that they’re not already learning, it’s the weaving of “security” into the risk management techniques they already know how to do. If testers fall short in their ability to apply risk management techniques, then they are falling short against the QA yardstick, there’s nothing particularly security-related in this observation. If they are applying mature QA practices with modern risk management, but are not adequately addressing the software-induced business risks facing their stakeholders, then some security training might be in order. That security training should be built on the foundation of modern QA practice, including risk-based testing.

So, in some ways we agree: speak the lingo of QA. But in other ways we disagree. I think the original article fails to give credit to the decades of substantial research and practice in QA. In other words, it’s a lot more than speaking the language. It is standing on the shoulders of giants, not their toes.

Yesterday we released the second episode of the Reality Check Podcast. This month’s victim is Jim Routh, CISO of Depository Trust Clearing Corporation (DTCC). DTCC has a very advanced software security initiative that is well worth learning about. We talk about that in this interview. Have a listen!

I’m also pleased to announce that CSO online has syndicated Reality Check and will be distributing the podcast to their CSO audience. You can find the first episode with Steve Lipner here.

And Jim’s episode here.

OWASP just posted an interview with me as part of their budding podcast series. It’s nice to have the tables turned after doing all the Silver Bullet (and Reality Check) interviews! It’s also nice to be able to answer some of the questions that OWASP types have about Cigital’s approach to software security.

Download the podcast here.

The OWASP interviewer is Jim Manico, and he did a great job. He was a little worried about some of the questions he asked. In fact, off the record he kept saying he was sorry and telling me that I did not have to address certain questions. Personally, I enjoyed the questions he asked immensely. Though some of his questions were loaded, I do hope that my answers may serve to clarify our position and eliminate OWASP concerns.

Here are a few of the many more questions I address in the podcast:

• Why do you insist on use of the term “software security” as opposed to “application security”?
• What is static analysis good for and what is it no good for?
• What is the exact relationship between Cigital and Fortify?
• Why do you think your “top 19” is any better than the OWASP top 10 or the CWE top 25? (Special note, the 19 Sins work is Mike Howard’s and John Viega’s…I was not involved.)
• Why does Cigital have a proprietary approach to IP?
• What makes the Touchpoints any better than the SDL or CLASP?
• What is your relationship with Allan Paller and SANS?
• Who picked the “porn music” theme for Silver Bullet?

As an extra bonus, the theme music for this episode is a song written and recorded by my band Where’s Aubrey.

Anyway, enjoy the podcast, and let me know what you think about my answers!

More links:

• Show notes
• MP3
• OWASP Podcast RSS Feed
• Direct iTunes link

Myself and others have been getting Webinar invites from IBM’s developerworks regarding Rational’s AppScan Developer Edition.

This is of course part of the re-launch of the re-tooled AppScan product. It now includes a set of analysis types, some static and some dynamic. They’ve got fancy new names for subsets of each, some hair splitting to my ear and I’ve been reading/writing on this topic for 10 years.

The market for static analysis is plumping nicely year over year. Static analysis (SA) vendors, such as Fortify, are one-or-two revisions into producing dynamic analysis tools and suites that leverage hybrid approaches. With the big dynamic analysis tools controlled by IBM and HP a certain amount of this cross-over (and much more I predict) was inevitable.

It was also inevitable that that dynamic shops would take shots at the SA space. The dynamic guys got shot up pretty bad as their tools–sold as application security’s first ’silver bullet’–begun to meet with resistance on how much manual effort was still required and on how many false positives and/or missed vulnerabilities remained. Static analysis swept in promising a better solution. “Look at the source code”, they smiled, “and you can be more complete and accurate than if you just poke the application from the outside.” Well, now it’s SA’s turn in the tank and the dynamic shops are enjoying the reversal.

But, in a hybrid world–where the major tool vendors offer both static and dynamic tooling this is technically absurd. AppScan’s explanation on “String Analysis” is particularly absurd. They’ve chosen to attack the effectiveness of static analysis with their ’solution’, a static analysis technique.

It’s true that both Fortify and Ounce principally find SQL-injections through data flow techniques that propagate ‘taint’ and rely on tagging source, sink, and cleansing logic at the resolution of function calls. But I can say with authority that both tools model the potential values of strings as they propagate their data flow analysis as well. AppScan’s innovation isn’t so innovative.

Yes both SA tools, in my opinion, leave a bit to be desired in the core products’ support for what aspects of string propagation and manipulation they can follow. I won’t get into detail, but I advise security managers that it’s worth it to understand where the tool will fail you in this regard.

Coverity and the now defunct CodeAssure did an exceptionally good job modeling string values throughout code’s ’speculative execution’, as part of their static analysis. When I tested these tools, they surprised me in both what they were able to find and what other tools’ false positives they left out of reports. Alas, these two don’t help today’s security manager much if they’re focusing on Java EE or .NET web applications. CodeAssure is gone and Coverity’s product has (in my opinion) limited language support outside C/C++ (though, again, I can not stress enough how positive my experience with it in C/C++ has been).

The dynamic shops had to level the playing field. But, as near as I can tell, the current situation is this:

• The major vendors believe there’s benefit to static and dynamic analysis
• There’s a lot of room for technical improvement in the market leaders’ SA products, with respect to modeling
• The future’s tool will leverage static and dynamic techniques, because each is suited for find a particular class of problems well.

You, as a security manager, will still need to sift through the marketectures and promises and figure out which tool works best on the kind of code your organization builds/buys. A major component of this will be how customizable the tool is.

Over-reliance on ANY automated tools (static or dynamic) leaves you with un-found vulnerabilities and a false sense of security. Cigital’s assessment services rely on these tools for speed and scale, and so we’ve taken great pains to understand where their modeling bends and where it breaks. In our own practice, we augment static techniques with dynamic tests and have even begun writing some next-generation static techniques to counter these limitations. We’ve spent hundreds of hours helping many many clients wring more out of their preferred suite of tools with the same understanding: opting instead for less invasive customization and tuning.

The bottom line is (and I’ve been saying this since at least ‘03 now), there are strengths to each tool and each type of analysis. NONE will get you to the assessment finish line alone. Anyone who tells you otherwise is sellin’ you a load.

Let the posturing begin.

On January 12th, the CWE/SANS Top 25 Most Dangerous Programming Errors list was released. Sean Barnum (a Principal Consultant) participated in the creation of the list, and I did some off the record review myself (not for attribution).

There are some important good things about top ten lists that are worthy of mention. The notion of knowing your enemy is essential in security (as it is in warfare), and top ten lists can help get software people started thinking about attacks, attackers, and the vulnerabilities they go after. These days almost any attention paid to the problem is good attention, and the fact that the technical media is paying attention to software security at all is a good thing. Top ten lists help in that respect.

But I have some serious concerns about these kinds of lists that I wrote about in my informIT article this month:

Top Eleven Reasons Why Top 10 (or Top 25) Lists Don’t Work

Here are the reasons, stripped of history and commentary which you can find in the article:

1. Executives don’t care about technical bugs.
2. Too much focus on bugs.
3. Vulnerability lists help auditors more than developers.
4. One person’s top bug is another person’s yawner.
5. Using bug parade lists for training leads to awareness but does not educate.
6. Bug lists change with the prevailing technology winds.
7. Top ten lists mix levels.
8. Automated tools can find bugs–let them.
9. Metrics built on top ten lists are misleading.
10. When it comes to testing, security requirements are more important than vulnerability lists.
11. Ten is not enough.

I’m happy to announce the launch of my new podcast, the Reality Check Security Podcast with Gary McGraw:

The Reality Check Podcast with Gary McGraw focuses directly on software security practitioners and practical software security. Reality Check’s sister podcast, the Silver Bullet Security Podcast with Gary McGraw, follows a free form interview style tailored highlight the ideas and experience of security gurus. By contrast, Reality Check is concerned with practical questions centered on running large-scale software security initiatives in the real world.

Reality Check targets experienced leaders working to solve software security problems in large organizations every day. We use a standard script to guide each conversation with questions about history, methodology, best practice, and measurement. We plan to interview leaders of mature software security programs and leaders of programs just getting started.

Your feedback is absolutely welcome. Please subscribe to the series through or RSS feed or through iTunes.