James McGovern recently wrote a post on Gartner’s static analysis (SA) report. Among other things, he lamented the lack of actionable guidance within the report. A lack of implementation guidance doesn’t shock me from Gartner, I can’t say I expect that from them.

I can help James and community out by giving some of that guidance myself. I’ll try to do so in a tool-independent way. This topic deserves more than a “blog entry” format (admitting Cigital’s propensity for longer entries ;-)) so if people want more information on the topic, they can ping me.

In essence, you want to know the following about static analysis tools:

1. Who is going to run it (how will owners shepherd it)?
2. How much effort is it going to take to run (what budget will I need to support it)?
3. When do I run it and how often (Where does it fit within my SDL?)
4. What it’s going to find (How will the results enter/impact my organization)?
5. What won’t it find (How does it play with the rest of the assurance ecosystem)?

[Market Adoption and Making Your Choice]
Gartner seems underwhelmed by Microsoft’s ability to effect the commercial static analysis market. Cigital has always believed these technologies would make great features of a compiler/IDE and so we’re also disappointed. They’ve got great technologies but don’t exploit them commercially. I still like their freely available FXCop but this opinion isn’t popular. Perhaps that’s because I see a lot of value to FXCop as a unit testing framework that enables security test and other people are comparing it to glossy commercial tools like Fortify and Ounce run by security analysts in a very hands-off fashion. Who’s considering the tool really does make a huge difference in how it will fare.

From my perspective, the most successful SA tool vendors have made answering the “who should adopt the tool” question more difficult as they’ve waffled on licensing models. By the seat? By the core? By the KLoC? I’ve gotten complaints from prospective tool buyers indicating that a leading tool vendor’s price was 10X another vendor’s. Before you ask “Which one?”: I’ve heard specific examples in opposite directions!

Certain tools will look better to different potential buyers. Experience has shown positive response to Coverity’s C/C++ results from developers. Application security groups like Fortify and Ounce’s products, and several years ago, I saw QA Managers in two very different firms go ga-ga over Klockwork’s tool. I attribute people’s impressions not to a product vendor focusing on quality or security but to how closely the tool vendor’s conception of how roles interface with their tools throughout the vulnerability management and software development life cycles matches the organization’s own structure. Up until a few years ago I don’t believe SA vendors had fully conceived the “developer”, “analyst”, “security manager” roles in their products. Ounce was the one exception: they’ve always staunchly believed themselves to be principally a Security Analyst’s tool. Some vendors attempted to unify roles into use of a single interface while others attempted to split the product up into different configurations for each. This created confusion and friction between some adopting organizations and the tool vendor’s license model.

In the end you want SA tools to affect as many developers as possible and to get low-latency scan results as early in software’s life cycle as possible. However, having successfully deployed tools a variety of ways, I’m convinced central deployment is the way to go. Application Security should run the tool. This will help manage rule/configuration update, as well as central measurement collection, risk management, and issue tracking. How do we get low-latency and early-lifecycle centrally? Treat the tool as a service. More on that later.

Notable exceptions to the central deployment model include business units possessing one or more of the following characteristics:

• Acquisitions of previously successful, culturally-independent companies
• High-quality product development teams
• Fundamentally different SDL, development platform, language, and toolkits

For in-unit deployment one can “run locally” but must “report centrally” supporting security goals involving measurement and risk management. And, let’s be clear: if a business unit has a wicked-good continuous integration/build-management group, they-by all means-should be tapped to integrate with the SA service. Cigital and its clients have had success in integrating both the front (code submission) and back-end (bug/issue tracking) with Fortify and a variety of open source build/CI/bug-tracking products. This maintains a central model, but reduces latency dramatically.

[How Much Effort]
I’m sorely disappointed in the honesty of tool vendor sales-folk regarding level of effort. One of my first analogies for SA tools is this: Did your company buy Mercury products for load and performance testing? Think of your SA tool like Mercury–you’ll need at least as much money to deploy and implement the tool organization-wide as the licenses cost you. Sorry.

Organizations with more than 2-300 developers should plan on the following:

• 3-4 man weeks to do initial tuning and pilot implementation
• 4-16 man hours / large application to integrate with the static analysis tool and assure appropriate configuration
• 4-8 man hours / 50-100KLoC (language-/complexity-dependent) to triage results on a new-application scan
• 1 person / year as tool shepherd (app integration, custom-rule creation, rule-pack maintenance and release, support)

Organizations get into trouble when their SA tool enjoys broad acceptance by the organization before they effectively manage the submission and results triage/documentation process. My data indicates code submission can burn 8-24mhrs / application if security groups don’t ‘remember’ (or automate) the submission and scan setup process. Security groups doing scans centrally and hand-writing code review results can waste another 16hrs documenting their findings / application.

…think about it: if your organization allots a week / tool-assisted code review and wastes 24 hours getting the tool’s first result set and 16hrs documenting findings… that only left 10 hrs to actually review the code! Since I believe that 4-8 hrs / 50-100KLoC is necessary to triage, you can imagine how deep I think such code reviews are :-P At Cigital, we’ve got dramatic benefits to depth by decreasing cost of submission and reporting. This is key to scaling SA efforts. Those adopting tools will feel overwhelmed if they don’t plan to solve submission and reporting problems before wider roll-out.

[How Often]
As often as possible. If you’ve got a Continuous Integration initiative (CI), engage those folk. If you deploy your tool purely centrally, plan to scan each high-risk application at least once / release. No, to my knowledge, there isn’t a good answer to the “can these tools do incremental (partial) scans”. You should always keep old results around though, as they’ll (along with other measures) help you understand whether more or less vulnerabilities are being caught during each scan. Remember though, every time you add a rule to the scan, your vulnerability-finding bar just moved and your metrics might be thrown off.

Ounce, Fortify, and Klocwork seem pretty good about determining whether or not they’ve found a particular issue in a previous scan or not. This includes situations where code blocks move around a bit. This means that running regression scans to determine whether or not an issue has been fixed is viable. I can’t comment on Coverity’s capabilities here.

[What's it going to find]
I’m impressed that Fortify has published its vulnerability taxonomy and a bit shocked that others haven’t produced as public a resource. I’m excited too that vendors have agreed to comply with CWE labels when they report findings. I will say what I always do on this topic though: test the tool on your own code though. You simply don’t know what it will find until you throw all your idiosyncratic build, language, toolkit, and platform nonsense at the particular tool you’ve selected.

[What won't it find]
I’m continually shocked as to how much one can find by running tool Y after tool X on the same piece of code (replace X and Y as you see fit). As I’ve written time-and-time-again, the corner cases and exceptions missed by each engine are baffling. Let me put things in perspective for you: three of my clients have reported that their SA tool deployment find 17, 33, and 50% of the total number of issues found by assessment efforts respectively. If these percentages hold for your organization, inside a coding bug realm then your job is at best 50% done having run the tool and triaged its results.

Considering that Microsoft and Cigital believe that “bugs” account for only 50% of the vulnerability space, that leaves you unaware of 75% of your application’s vulnerability at the end of triage. As I always say, “use the tool to facilitate code review and increase a reviewer’s understanding of the code–not as a code reviewer that finds bugs automatically.”

Good luck out there, I’d love to hear about your particular experiences.
-jOHN

My colleague, Ben Walther, pointed me at this post about Cloud applications and Web-app security by Rich Mogull. The title is “How the Cloud Destroys Everything I Love (About Web App Security)”. The post talks about running Web apps on a cloud platform like EC2. I’m not sure I buy into everything they say.

First, I’m not sure what Rich means by a “Web App”. To me, the term Web App describes an n-Tier application with a browser front end and some kind of backend SQL database. There are maybe some web service calls thrown into the mix. It’s the kind of applications that everybody’s been writing for the last 10 years. So, what’s going to change if I’m running this same application architecture on infrastructure that I’m buying as a service? Sure, I have to worry about all of the inter-machine communication channels because I don’t have the nice data center supplied network security. But what else?

Now when we move to cloud-based applications (environments like AppEngine or 10gen), ones that take advantage of the highly distributed nature of the code as well as the virtual environments, then we have changed the application architecture. I buy that security for these apps changes, but it’s no longer “web app” security. In these cloud-based applications, there are some different fundamental assumptions about the architecture, like no transaction serializability. But for these legacy web apps running in a virtualized infrastructure, I’m less convinced that there is a drastic change.

There are a couple of specific points made in the article that I don’t agree with:

Secure development (somewhat) breaks because the underlying platform can’t be locked down.
Just because you can’t lock it down yourself doesn’t mean that it can’t be locked down. This seems like an argument for secure deployment breaking and not secure development. Even then, the PAAS or IAAS may actually lock the platform down better than you can. It does shift the problem from looking at technical artifacts (configuration files, patch logs, etc) to looking at legal and audit artifacts (SLAs and certifications).

Static analysis tools (mostly) break.
The contention is that there’s less code you program yourself. I don’t see this as true for IAAS platforms like EC2, how much code is provided that you really need to worry about. Besides, static tools are language based and if it’s the same language, it doesn’t really matter whether it’s running on a virtual OS or a physical one. The change that breaks static analysis is the move to dynamically typed languages.

My take is that the infrastructural changes of a cloud computing have a more drastic effect on an organization’s ability to deploy securely. Being able to develop securely is based on the application architecture. I really see these as independent levers rather than a single “cloud” lever.

Technorati Tags: Cloud Computing, Security

A recent thread about hardware and software requirements for development on the Google cloud forum made me wonder what cloud computing will mean for development, test and production environments. There are a lot of really interesting questions here, but my mind got stuck on the relationship between development and production. I’ve always been amused at the IDE feature that touts instant deployment from the developer’s desktop to the production system. Wearing my security hat, I should use the word “horrified” instead of “amused” because I can think of no worse model for deployment than developer-to-production.

I came to the conclusion that deployment of cloud-based applications will require much more engineering rigor to assure the security of the deployed application. There will, of course, be variation in the amount and type of rigor based on the type of cloud the application is being deployed to. For example, deploying to a platform as a service provider will be different than deploying to a infrastructure as a service provider where the later will require a lot more rigor. But, given the amount of manual steps involved in most of the deployments I see, I’m very worried that deployment automation will be overlooked as a required security control for cloud-based apps.

Large companies will have tools like Opsware and Bladelogic for their existing data centers, so they have a base of automation to work from. It’s the medium-large and medium sized organizations that I see as needing automation help to secure their deployments. I think these companies will be adopting cloud-based applications for their cost advantage over internally owned and managed infrastructure. But these organizations have a much more collegial relationship between development and operations and moving to operations to a service model will break these relationships. It’s the same problem that companies face when moving parts of development offshore. The solution for development is more formal API contracts and specifications. With operations, automation is an excellent way of formalizing the “API contract” between development and operations.

Security will have its part in this equation as well. All of the assumptions about who can do what with which parts of the application, who you can trust and with what credentials will have to be re-evaluated. Some of my colleagues would argue that this is the least of our worries. I would agree, but given the lack of a shared understanding of cloud security, I think that these operational concerns need to be on the list.

This is a guest post from Paco Hope, Technical Manager at Cigital.

I read a blog entry about “re-aligning training expectations for QA.” It has some useful points that both developers and so-called “security people” need to hear. I disagree with some implicit biases, however, and I think we need to get past some stereotypes that sneak out in the article.

Bias #1, obviously, is the focus on the web. Despite its omnipresence, there is more non-web software than web software in the world, and non-web software does more important stuff than all the web software combined. The role of security in software testing is vital, and the presence or absence of web technologies does not change that. Despite writing a book on Web Security Testing, I know my place in the universe. Quality assurance and software testing are disciplines far older than the web, and their mission is so much bigger than finding vulnerabilities.

Bias #2 is vulnerabilities über alles. By talking about weaving vulnerabilities into security test plans, we’ve overlooked the first place where security goes into the QA process: test strategy. Look at any of the prominent folks in QA (James and Jon Bach, Michael Bolton, Rex Black, Cem Kaner), the people I’m privileged to share podiums with at QA conferences like STAR West, STAR East, and Better Software, and you’ll see that security is part of the overall risk-based testing strategy. Risk-based testing has been around for a really long time. Longer than the web.

Before anyone talks about vulnerabilities to test for, we have to figure out what the business cares about and why. What could go wrong? Who cares? What would the impact be? Answers to those questions drive our testing strategy, and ultimately our test plans and test cases.

Bias #3 is that idea that a bunch of web vulnerabilities are equivalent in impact to the business. That is, you just toss as many as you can into your test plan and test for as much as you can. This isn’t how testing is prioritized.

You don’t organize testing based on which top X vulnerabilities are likely to affect your organization (as the blog suggests). Likelihood is one part of the puzzle. Business impact is the part that is missing. You prioritize security tests by risk severity—that marriage of likelihood and impact to the business. If I have a whole pile of very likely attacks that are all low or negligible impact, and I have a few moderately likely attacks that have high impact, I should prioritize my testing effort around the ones with greater impact to my business.

Bias #4 is the treatment of testers like second class citizens. In the blog article, developers are “detail oriented” have a “deep understanding of flows.” Contrast this with QA who merely understand “what is provided to them.” They sound impotent, as if all they can do is what they’re told. Software testing, despite whatever firsthand experience the author may have, is a mature discipline. It is older and more formalized than “security” as a discipline. Software testing is older than the Internet or the web. If software testing as a discipline has adopted security too slowly, given security’s rise to the forefront in the marketplace, that might be a legitimate criticism. But I don’t approve of the slandering QA by implying that they just take what’s given them and execute it. QA is hard and there are some really bright minds working in that field.

As someone who has been training in risk-based security testing for several years now, I totally agree with some points, but very much disagree with others. I agree that the “bug parade” (as we call it) of top X vulnerabilities to find is the wrong way to teach security testing. Risk management, though, has been a fundamental part of mainstream QA for a very long time. Likewise, risk management is the same technique that good “security people” use to prioritize their results. Risk management is certainly how the business is going to make decisions about which issues to remediate and when. Risk management is what ties this all together.

If there’s something that QA needs to learn that they’re not already learning, it’s the weaving of “security” into the risk management techniques they already know how to do. If testers fall short in their ability to apply risk management techniques, then they are falling short against the QA yardstick, there’s nothing particularly security-related in this observation. If they are applying mature QA practices with modern risk management, but are not adequately addressing the software-induced business risks facing their stakeholders, then some security training might be in order. That security training should be built on the foundation of modern QA practice, including risk-based testing.

So, in some ways we agree: speak the lingo of QA. But in other ways we disagree. I think the original article fails to give credit to the decades of substantial research and practice in QA. In other words, it’s a lot more than speaking the language. It is standing on the shoulders of giants, not their toes.

Yesterday we released the second episode of the Reality Check Podcast. This month’s victim is Jim Routh, CISO of Depository Trust Clearing Corporation (DTCC). DTCC has a very advanced software security initiative that is well worth learning about. We talk about that in this interview. Have a listen!

I’m also pleased to announce that CSO online has syndicated Reality Check and will be distributing the podcast to their CSO audience. You can find the first episode with Steve Lipner here.

And Jim’s episode here.