To address this concern in conversations conducted as part of the OWASP Threat Modeling project, I worked with Cigital’s Sammy Migues to construct a Glossary of Threat Modeling Terms. Sammy has long been a proponent of such approaches to glossaries: sticking firmly to nouns as nodes and relating those nodes with action verb edges. See figure below:

Please feel free to download and use Glossary of Threat Modeling Terms stored as a Google Doc. (*1)

As you consider the terms and their relationships in this diagram, you may wonder about how/why we sourced what we did when defining terms. We used the following heuristics:

• Favor the earliest definition known


• Allow a more recent definition to update or color a previous definition, especially with respect to application security context


• Do not allow a more recent definition to change direction without having been journal (or similarly peer-review) accepted


• Favor software development and especially architecture sources over security sources

The definitions provided by the OWASP wiki, in particular, fail that third heuristic a fair amount, while often providing the second’s benefit.

As for “why”, perhaps it’s Sammy’s pedigree combined with my years in research and doing both magazine and journal review for IEEE. You’re instructed for better or worse, that there must be a good reason to replace an existing definition. Comparing the published result with Cigital’s internal docs, source material differs considerably. A lot of our internal material on threat modeling relies on work from decades ago. ISACA material, most commonly used in the diagram above, applies more directly to a context of web-applications (remember, the glossary was done for an OWASP project) than ours, which must face a much broader assessment context. Additionally favoring existing externally available documentation, especially from an architecture context such as ISACA, serves to create a better chance for engaging those who we most want in a application security threat modeling discussion: architects, development teams, and development-centric organizations.

The biggest complaint with the glossary thus far has been with regard to its handling of “risk”. Basically, risk management remains largely out of scope of this glossary of terms, which instead chooses to focus on threat, attack surface, and vulnerability enumeration. Yes, analysis of likelihood (whatever that means) and impact are vitally important aspects of threat modeling. These two concepts, however, tend to be organization-/philosophy-specific. As such, they remain out of scope of this series of blog entries as well.

All that having been said, I meant the glossary to be a “worksheet” rather than “the 10 Commandments”. So, I think we have something to start with, improve, and evolve. Feel free to refer back to this either in your own work or as you read coming entries.

-jOHN

(*1) – If you’d like OS X or Windows “source code” to this diagram, simply email me.

Cigital has always favored delivery of holistic assessments (those assessments that consider architecture/design as well as code, leveraging static & dynamic tools, and using both tool-assisted and manual techniques). During my tenure as Director of Security, we found the following distribution of findings:

• Static tool: 20%


• Dynamic tool: 5% (*1)


• Manual SCR: 15%


• Architecture Risk Analysis: 60%

Those numbers may surprise and demand some explanation. For instance, though Microsoft at one point reported 50% of their findings were “flaws”, Cigital was finding 60% (on average, with as much as 70%) of its issues as ‘flaws’. If anything, our classification of ‘flaw’ was less inclusive though, and at the time I looked into this, I concluded that our larger percentage could be due to technique, focus on architecture in methodology (LoE), or level of customer familiarity/maturity with the tech-stack they developed on.

Likewise, our process started with a “whitebox” triage then shifted to exploratory testing (aka: dynamic) rather than the other way around (which I find common elsewhere) and that order-of-operations probably moved balance from dynamic to static. So, in practice we “found” vulnerabilities statically and then supported those findings with dynamic tests to validate their exploit-ability and impact (though perhaps dissatisfying, static gets the point in our data set here).

Slicing by technique
Our experience made it pretty evident to us: skip assessment techniques and expect to miss some critical vulnerabilities.

Slicing by vulnerability type
As I’ve mentioned in previous blog entries and mailing list posts, security managers need the capability to assess and report on their ability to find particular classes of security vulnerability as well. WASC’s Web Application Security Statistics provide a starting point for both classes of attacks you might expect to find and their probability. There’s no shortage of vulnerability taxonomy work out there, so we’ll leave this topic at this superficial level.

Optimizing a Security Practice
Whether you know what percentage of your reported findings come from each assessment technique, or you’re just starting to build your assessment practice, optimization (measurement, analysis, iterative improvement) should be a goal. To do this we combine our knowledge of the techniques we’re employing and the enumeration of vulnerabilities we want to find and report.

Confined to the context of static analysis, I described the following loop in a previous blog post to do just that:

• Use a representative sample of your applications to observe a static analysis tool’s performance, not a contrived test suite.
• Consider the tool’s findings relative to pen-testing findings on the same app.
• Did new and interesting findings result?
• Did pen-testing findings found by the static tool provide adequate root-cause analysis and remediation advice to fix problems earlier?
• How long did it take to on-board an app? How will this scale to your portfolio of apps?
• How long did it take to triage the results? How will this scale?

The purpose of this trial experimentation was to provide some basic comparison between assessment techniques (though you could easily come up with a more thorough approach). Of course, you’d want to consider findings from the perspective of all techniques, not just static tools.

Internally, we had our assessment lab managers write up a qualitative comparison of assessment techniques we employ, comparing how well each is “suited” to finding vulnerabilities against common vulnerability taxonomies (*2) and came up with the following:

• ARA: 14%


• Static tool: 12%


• Manual SCR: 21%


• Manual Pen: 21%


• Dynamic Tool: 12%


• Sec Testing: 20%

At this point, the reader may be asking why my experience differs from our lab managers’. Specifically, for instance, “why are dynamic techniques over-estimated here as compared to to historical findings rates?” Two reasons (at least) come to mind:

1. The balance of techniques that constitute our assessments are “moving back” towards dynamic as software-under-test and market forces evolve


2. There’s a difference between how suited a technique is to discover something and how we choose to find it (I guarantee that in some circumstances this gap represent ‘optimization opportunities’, while in others it represents explicit technology or methodological choices).
   

The Caveat You’ve Come to Expect From Me
Whether your considering a vendor’s assessment service, an assessment standard, or your own internal assessment practice, it’s key to be able to detect biases. When others supply you assessment data you’ll likely uncover biases (though it may not be apparent to those reporting initially). Be careful, I doubt you’ll be able to compile, combine, or relate numbers you receive from varying sources for this reason.

If you’re compiling assessment data for the purpose of motivating practice improvement, you may want to either 1) extract universal similarities from the data you get and present those lessons learned to management, or 2) just list the others’ experience separately. This can provide those to whom you’re presenting a compelling view to your perspective on differing assessment approaches.

In fact, many factors will muck up assessment comparison. For instance, when people report a percentage (like I did above), did they mean finding instances, findings, by type, or something else (*3)? What was their methodology, level-of-effort, and how did they conduct assessment. For instance, risk-based assessments produce a very different distribution (with wider variation) than time-boxed techniques, which more frequently drive to an explicit (or worse, implicit) checklist.

One thing universally seems true though: the more assessment techniques a software security practice makes use of, the more types of vulnerabilities it can expect to report.

-jOHN

(*1) – Clarification: our use of dynamic tools is human-assisted (both in terms of crawl and in follow-up with manual effort or leveraging internally developed tools).

(*2) – WASC, OWASP Top 10, CWE Top 25

(*3) – My data is comprised of reported vulnerabilities. Answering questions about what risk-level I reported requires some abstraction as clients differ in their requirements there. Likewise, behavior regarding instances vs. buckets of ‘related vulns’ requires addl. explanation.

In his post he discusses the need to consider the cost of things in terms of the opportunity lost by doing them:

“Jim Rogers has a great practice on saving – when you think of buying something today, simply multiply its cost by twenty to see how much it will be worth down the road when you are retired. A dinner out is “only” $75, but would you still go out to eat if you figured the cost at $1,500? Warren Buffett pithily sums this up as – do I really need a $300,000 haircut?”

Indeed.

What about the reverse? As a security manager, please ask yourself,

“What am I not doing incrementally that I’m going to regret having made no progress on in the future?”

Today, as I look back on complaining about the dynamism of Java EE applications, I could kick myself for not spending a few hours each week building scanning technology to address these issues. Bygones.

What about you? Run an assessment practice? Couldn’t you just kick yourself for not spending a few hours each week:

• Measuring what vulnerabilities you found, and what techniques paid off to find them?


• Measuring which finds were fixed and which most frequently resulted in regression?

How about those of you working with developers and security toolkits? Couldn’t you just kick yourself for not spending a few hours each week:

• Collecting the ‘best examples’ of validation code from your reviews?


• Writing a security widget to encode output for a particularly popular context?

Just as the future cost of (even seemingly small) wasted expense today can be staggering, the regret for not addressing daunting problems with an almost negligible amortized effort can be staggering later.

What can you start addressing incrementally now?

you will find our observations regarding static analysis shared by others. It’s encouraging to note that Flash Sheridan observes many of the same difficulties and more formally treats them in his ISSRE ’10 publication. It’s worth a read.

A few commentators shared some decent points I wanted to address in a comment. I’m cross-posting my commentary here:

Comments treating the application of static tools as a distinct (and unfavorable) alternative to “fixing your SDLC” disappoint me, especially from the OWASP community (of which I am a contributing paid member). In fact, the community’s OpenSAMM project actively prescribes use of static analysis in not one but four (4) unique activities. So, both BSIMM and SAMM see static analysis as a part of fixing one’s SDL. **Note that in both models, all activities are not compulsory.

“Fix your code” is another red-herring alternative to static analysis. I believe static analysis inseparable from effective usage of a secure programming toolkit (such as OWASP’s ESAPI or an organization’s own toolkit) because static analysis tools can meet the need for organizations to 1) detect consistent and thorough use of secure APIs (and non-use of dangerous ones) as well as 2) detect incorrect usage of such APIs (whether through broken call-order, un-paired functions, or other bugs). We shouldn’t forget, as Mr. Manico indicates, 3) running static tools on these security toolkits themselves finds problems that careful review may not otherwise have found.

So, secure SDL models indicate usage and we–as a community–see use in these tools in helping remind developers how to correctly build secure code. I think we have to agree static analysis tools find exploitable bugs (even though sometimes in an overly expensive or time-consuming way). So, I think we’re stuck with them as a ‘best practice’ in the short term, however little some of us may like them. To dynamic vendors out there, I’d even go so far as saying, “Early adopters have begun to over-rely on static means to find problems more effectively/efficiently found with dynamic tools. Don’t worry, the ones I’m working with will begin to tilt the balance back using assessment data”.

I would like to put a face-and-name on the key notion that almost all commentators have thus-far touched: the static analysis tool that gets used by a majority of application developers out there, in the future, will likely not look like the ones we have today. And that’s fine too.

Dan Cornell wrote a blog post on static analysis coverage. He observed that while the static tool he used completed its scan, it didn’t find things it should. He implicitly referred to this as coverage.

I’ve often put things as follows:

Your tool can’t find what it can’t see and it can’t see what it doesn’t parse.

This got said often years back because at the time it was quite difficult to integrate tools effectively into a builds so that they would in fact ‘see’ everything and successfully parse it (*1). We referred to this concept of how well we, as operators, had integrated as providing visibility into the code. These days, market leading tools don’t have nearly the problem they used to with finding the code they need to parse… or parsing that code successfully (*2).

These days, the frameworks developers use for persistence, object remoting, application navigation, and view display challenge visibility most. Tools see the code for your controller and parse it successfully but they don’t necessarily understand what it represents. This happens in almost every application I scan, even the painfully simple ones.

We couldn’t have our Consultants going out into the world armed with tools bearing only spotty visibility into applications so I wrote a utility identifyEntryPoints to find entry points on its own. We don’t want Consultants guessing at what their static analysis tool missed or popping the hood open to try to discern what it found and not doing what they’re paid to do: assess the application, so I wrote another utility entryPointCompare to compare what the tool found vs. what I was able to find programmatically. See actual output below (I’ve replaced the static tool’s name with ‘TOOL XXX’):

Sanguine:demo jsteven$ /Users/jsteven/code/demo/working/engine/util/entrypointCompare.py
2011-02-06 09:07:08,158 INFO     bois_1.0        bin.execPythonRobot STARTING: /Users/jsteven/code/demo/working/engine/util/entrypointCompare.py
Factory 31, [TOOL XXX] 0
 TOOL XXX missed set(['Index', 'csrchat', 'accountsList', 'Transfer', 'editProfileFormBean', 'newAccountFormBean', 'fileList', 'Auth', 'newClientFormBean', 'qaFormBean', 'UserProfile', 'SetupProfileAction', 'uploadFile', 'VerifyAnswerAction', 'BeanEditProfileAction', 'existingClientFormBean', 'GetDocuments', 'ChatPoll', 'announcements', 'modifyDocument', 'BeanCreateAccountAction', 'ChangePasswordAction', 'VerifyClientAction', 'ChatSend', 'Help', 'fileUploader', 'BackOfficeAdmin', 'passwordFormBean', 'BeanCreateClientAction', 'action', 'Accounting'])

Here, I ran the test on Cigital’s own internal “Bank of Insecurities”, which is a simple Struts1 application. The utility’s output represents all the actions and forms that account for web-based input in their own respective ways. Sure looks like some pretty important stuff was missed in there huh?

The next step was to write a third program registerEntryPoints which would write rules for the missing entry/exit points as appropriate so that the beefy commercial static analysis tool could do what it does best. These utilities represent just some of the internal workings of Cigital’s ESP platform.

Yes, but my tool finds stuff all the time
Would a tool not find any findings without detecting entry points? Well, it would likely find potential vulnerability but only using more local syntactic analysis. Remember, “you can’t find what you can’t see…”

I’m sensing a theme in your posts jOHN
OK, so, that’s two blog posts (Mr. Cornell and my own) complaining about what I refer to visibility. Now what?

Spend time cataloging your chosen tool’s performance
Cigital experience indicates that as few as 10-13 reasonably chosen applications can serve as a representative sample for as many as 300 applications on the Java platform. When you pilot those 10-13 applications, conduct the following steps to avoid the visibility failures seen above in the portfolio as a whole:

1. On-board the application using whatever tool interface gives the most feedback about missing artifacts (*3)
2. Explore scan logs for identified entry points
3. Manually explore the application’s deployment descriptors and critical configuration files
4. Document controller logic as framework default or developer extended
5. Identify key entities within the DAO/persistence framework

When you’ve done that for 10-13 apps, our representative sample, stop and take stock of what you’ve collected. You’ll have created a very good list of items for which you’ll want to create custom rules. Different tools call the kind of rules you’ll create different things, but suffice it to say you’ll be writing rules to tell the tool that it’s:

• Entry: Taking input from untrusted web sources
• Entry: Taking input from untrusted partner applications
• Exit: Placing data in a untrusted view (browser, service repsonse, etc.)
• Exit: Conducting CRUD operations on entity data

The above list is by no means exhaustive. Indeed, you’ll want to look at data originating from the persistence tier and headed to the web (not listed above) but we often consider this as a second phase of customization. You’ll also want to begin considering data entry/exit from 2nd and 3rd party components within the applications being tests. Again, another good subsequent customization phase.

Gosh, this sounds expensive
Yes; ‘well. Compared to simply running a static analysis tool using its IDE-based GUI, triaging results, and calling it quits this is darned expensive. However, it dramatically raises your visibility into an application’s vulnerability (you should also expect false-positive reduction).

Compare this to the potential difference in quality and depth of a manual, tool-assisted penetration test and an automated vulnerability scan. An engineer running AppScan costs much less than an expert penetration test and organizations have come to not only expect different results, but see these as two very different services. I predict that we’ll see a similar distinction between two services (automated and more expert-driven) in the static space in a few years.

Leveraging Manual Efforts Across Assurance Activities
If your organization doesn’t conduct manual penetration testing and is unwilling to pay your engineers to properly on-board applications into its static analysis tool, then neither static nor dynamic analysis will benefit from high visibility into applications under test. The cost will look “high” and “extra” as well.

Conducting the kind of analysis described by this entry (targeting entry and exit points) can inform both static and dynamic analysis alike. Conduct this exercise for applications built with representative technology stacks. On the static front, write custom rules based on your work. On the dynamic side, furnish this work to your dynamic testers (even if they are external vendors) so that their exploration benefits from it.

(*1) – Coverity’s Prevent was always a notable exception and able to integrate with even challenging build environments without much pain.
(*2) – Limitations still exist with complex (distributed) build systems, code-generation schemes, and even the more mundane JSP compilation process.
(*3) – Ask if you need help here, last time I published tool details, vendors got cranky.

If selecting and adopting a tool is so hard, even for experts, why should I bother?

Good question. The article was not written as indictment or defense of static analysis–it was a cautionary tale whose moral: “organizations can get wrapped around the wrong axle when selecting and adopting a tool” I believe in to the utmost.

If it is worth adopting static analysis, as I believe it is, then the question becomes: How do I avoid doing it poorly? I’ll start by revisiting some suggestions made in the article’s conclusion and continue.

Seek Experience
Expert consulting can dramatically improve the speed and effectiveness of an organization’s definition of and scaling static analysis and SCR practices. But, you don’t need experts’ help to successfully pick and adopt a tool.

Leverage communities you’re already amongst to absorb their experience. Reach out to:

1. Your local OWASP chapter
2. Organizations within your vertical
3. Similarly sized/structured organizations within your geography

Within the communities, others have likely already selected and adopted a tool. Though some will not share their selection or adoption experiences openly, others will. And, I’ve found that human nature can’t resist sharing at least _some_ aspects of a good war story.

War stories hint at underlying data more valuable than the particular selected tool: the aspects of adoption that posed the most challenge or required the most effort. Having your troops pointed in the right direction when fighting breaks out will benefit you more than making the better choice between equipping troops with a carbine or a rifle.

During last year’s BSIMM conference in Annapolis, MD, I saw tremendous inter-organizational knowledge sharing on the static analysis front. Not only was I impressed, but as usual, I learned things. War stories? Those improved with volume imbibed.

Eschew Deep Scientific Comparison for Trial Experience
Unless you graduated with an advanced degree focused on static analysis, or spent the last ten years building/analyzing it, I don’t feel that a comparative study will be satisfying. Many have asked me, “With you experience–come-on–tell me… which is better, A or B?” Individuals asking me almost always hear a variant of the following candor:

Comparing the analysis engines of the two market leaders is a lot like wondering about the Audi S4 vs. the BMW M3. Both take a comparable approach to their flag-ship consumer performance car: both switch between ~3.xL six-cylinder blown engines or ~4.xL naturally-aspirated V8s. In the end, both gobble premium fuel and get to 62mph in 4.x seconds. Both are fine pieces of engineering and each has its own religious following.

Like the previous advice, knowing how to drive it and where the car’s limits are likely provides lower track times than the selection of one car over the other for the average-to-enthusiast driver. It simply doesn’t matter whether the trappings of boost pressure produced by the S4′s supercharger beat out the direct-injection, independent throttles of the M’s high-revving V8 (*1)

I’ve digressed a bit too far, haven’t I? Don’t do the same with your static tool

• Use a representative sample of your applications to observe a static analysis tool’s performance, not a contrived test suite.
• Consider the tool’s findings relative to pen-testing findings on the same app.
• Did new and interesting findings result?
• Did pen-testing findings found by the static tool provide adequate root-cause analysis and remediation advice to fix problems earlier?
• How long did it take to on-board an app? How will this scale to your portfolio of apps?
• How long did it take to triage the results? How will this scale?

As your trial encounters problems, make explicit note of them. As you move from 3-10 apps to the portfolio as a whole, can your organization withstand stresses at scale? If not, you may want to switch tools or approaches.

I’ve seen security managers I respect get incredibly far without any consulting help just by taking an iterative approach and asking questions like those listed above.

Worry about what you can control
You control your organization’s staff size, skill set, scanning policy, and infrastructure. You do not control the architecture, implementation, or bugs associated with the static analysis tool you buy and deploy.

So, as you talk to others about their experiences selecting and adopting a tool… as you conduct trials with a selected tool… and as you plan a larger roll-out at scale of your static practices, think about strengths and weaknesses in a chosen tool not in terms of how its competition might perform relatively but in terms of whether or not they compliment or expose your organization’s staff-based, skill, policy, and infrastructure weaknesses (or play to its strengths in those same areas).

And finally, don’t be afraid to ask for help.
-jOHN

(*1) – In the interest of full-disclosure, I cast my lot with the 2011 M3 over its Audi competition despite BMW’s ‘heretic’ departure from their I6 engine architecture heritage in the 3-series. I can make no such clear claim to my allegiance on the static front.

I can help James and community out by giving some of that guidance myself. I’ll try to do so in a tool-independent way. This topic deserves more than a “blog entry” format (admitting Cigital’s propensity for longer entries ) so if people want more information on the topic, they can ping me.

In essence, you want to know the following about static analysis tools:

1. Who is going to run it (how will owners shepherd it)?
2. How much effort is it going to take to run (what budget will I need to support it)?
3. When do I run it and how often (Where does it fit within my SDL?)
4. What it’s going to find (How will the results enter/impact my organization)?
5. What won’t it find (How does it play with the rest of the assurance ecosystem)?

[Market Adoption and Making Your Choice]
Gartner seems underwhelmed by Microsoft’s ability to effect the commercial static analysis market. Cigital has always believed these technologies would make great features of a compiler/IDE and so we’re also disappointed. They’ve got great technologies but don’t exploit them commercially. I still like their freely available FXCop but this opinion isn’t popular. Perhaps that’s because I see a lot of value to FXCop as a unit testing framework that enables security test and other people are comparing it to glossy commercial tools like Fortify and Ounce run by security analysts in a very hands-off fashion. Who’s considering the tool really does make a huge difference in how it will fare.

From my perspective, the most successful SA tool vendors have made answering the “who should adopt the tool” question more difficult as they’ve waffled on licensing models. By the seat? By the core? By the KLoC? I’ve gotten complaints from prospective tool buyers indicating that a leading tool vendor’s price was 10X another vendor’s. Before you ask “Which one?”: I’ve heard specific examples in opposite directions!

Certain tools will look better to different potential buyers. Experience has shown positive response to Coverity’s C/C++ results from developers. Application security groups like Fortify and Ounce’s products, and several years ago, I saw QA Managers in two very different firms go ga-ga over Klockwork’s tool. I attribute people’s impressions not to a product vendor focusing on quality or security but to how closely the tool vendor’s conception of how roles interface with their tools throughout the vulnerability management and software development life cycles matches the organization’s own structure. Up until a few years ago I don’t believe SA vendors had fully conceived the “developer”, “analyst”, “security manager” roles in their products. Ounce was the one exception: they’ve always staunchly believed themselves to be principally a Security Analyst’s tool. Some vendors attempted to unify roles into use of a single interface while others attempted to split the product up into different configurations for each. This created confusion and friction between some adopting organizations and the tool vendor’s license model.

In the end you want SA tools to affect as many developers as possible and to get low-latency scan results as early in software’s life cycle as possible. However, having successfully deployed tools a variety of ways, I’m convinced central deployment is the way to go. Application Security should run the tool. This will help manage rule/configuration update, as well as central measurement collection, risk management, and issue tracking. How do we get low-latency and early-lifecycle centrally? Treat the tool as a service. More on that later.

Notable exceptions to the central deployment model include business units possessing one or more of the following characteristics:

• Acquisitions of previously successful, culturally-independent companies
• High-quality product development teams
• Fundamentally different SDL, development platform, language, and toolkits

For in-unit deployment one can “run locally” but must “report centrally” supporting security goals involving measurement and risk management. And, let’s be clear: if a business unit has a wicked-good continuous integration/build-management group, they-by all means-should be tapped to integrate with the SA service. Cigital and its clients have had success in integrating both the front (code submission) and back-end (bug/issue tracking) with Fortify and a variety of open source build/CI/bug-tracking products. This maintains a central model, but reduces latency dramatically.

[How Much Effort]
I’m sorely disappointed in the honesty of tool vendor sales-folk regarding level of effort. One of my first analogies for SA tools is this: Did your company buy Mercury products for load and performance testing? Think of your SA tool like Mercury–you’ll need at least as much money to deploy and implement the tool organization-wide as the licenses cost you. Sorry.

Organizations with more than 2-300 developers should plan on the following:

• 3-4 man weeks to do initial tuning and pilot implementation
• 4-16 man hours / large application to integrate with the static analysis tool and assure appropriate configuration
• 4-8 man hours / 50-100KLoC (language-/complexity-dependent) to triage results on a new-application scan
• 1 person / year as tool shepherd (app integration, custom-rule creation, rule-pack maintenance and release, support)

Organizations get into trouble when their SA tool enjoys broad acceptance by the organization before they effectively manage the submission and results triage/documentation process. My data indicates code submission can burn 8-24mhrs / application if security groups don’t ‘remember’ (or automate) the submission and scan setup process. Security groups doing scans centrally and hand-writing code review results can waste another 16hrs documenting their findings / application.

…think about it: if your organization allots a week / tool-assisted code review and wastes 24 hours getting the tool’s first result set and 16hrs documenting findings… that only left 10 hrs to actually review the code! Since I believe that 4-8 hrs / 50-100KLoC is necessary to triage, you can imagine how deep I think such code reviews are At Cigital, we’ve got dramatic benefits to depth by decreasing cost of submission and reporting. This is key to scaling SA efforts. Those adopting tools will feel overwhelmed if they don’t plan to solve submission and reporting problems before wider roll-out.

[How Often]
As often as possible. If you’ve got a Continuous Integration initiative (CI), engage those folk. If you deploy your tool purely centrally, plan to scan each high-risk application at least once / release. No, to my knowledge, there isn’t a good answer to the “can these tools do incremental (partial) scans”. You should always keep old results around though, as they’ll (along with other measures) help you understand whether more or less vulnerabilities are being caught during each scan. Remember though, every time you add a rule to the scan, your vulnerability-finding bar just moved and your metrics might be thrown off.

Ounce, Fortify, and Klocwork seem pretty good about determining whether or not they’ve found a particular issue in a previous scan or not. This includes situations where code blocks move around a bit. This means that running regression scans to determine whether or not an issue has been fixed is viable. I can’t comment on Coverity’s capabilities here.

[What's it going to find]
I’m impressed that Fortify has published its vulnerability taxonomy and a bit shocked that others haven’t produced as public a resource. I’m excited too that vendors have agreed to comply with CWE labels when they report findings. I will say what I always do on this topic though: test the tool on your own code though. You simply don’t know what it will find until you throw all your idiosyncratic build, language, toolkit, and platform nonsense at the particular tool you’ve selected.

[What won't it find]
I’m continually shocked as to how much one can find by running tool Y after tool X on the same piece of code (replace X and Y as you see fit). As I’ve written time-and-time-again, the corner cases and exceptions missed by each engine are baffling. Let me put things in perspective for you: three of my clients have reported that their SA tool deployment find 17, 33, and 50% of the total number of issues found by assessment efforts respectively. If these percentages hold for your organization, inside a coding bug realm then your job is at best 50% done having run the tool and triaged its results.

Considering that Microsoft and Cigital believe that “bugs” account for only 50% of the vulnerability space, that leaves you unaware of 75% of your application’s vulnerability at the end of triage. As I always say, “use the tool to facilitate code review and increase a reviewer’s understanding of the code–not as a code reviewer that finds bugs automatically.”

Good luck out there, I’d love to hear about your particular experiences.
-jOHN

This is of course part of the re-launch of the re-tooled AppScan product. It now includes a set of analysis types, some static and some dynamic. They’ve got fancy new names for subsets of each, some hair splitting to my ear and I’ve been reading/writing on this topic for 10 years.

The market for static analysis is plumping nicely year over year. Static analysis (SA) vendors, such as Fortify, are one-or-two revisions into producing dynamic analysis tools and suites that leverage hybrid approaches. With the big dynamic analysis tools controlled by IBM and HP a certain amount of this cross-over (and much more I predict) was inevitable.

It was also inevitable that that dynamic shops would take shots at the SA space. The dynamic guys got shot up pretty bad as their tools–sold as application security’s first ‘silver bullet’–begun to meet with resistance on how much manual effort was still required and on how many false positives and/or missed vulnerabilities remained. Static analysis swept in promising a better solution. “Look at the source code”, they smiled, “and you can be more complete and accurate than if you just poke the application from the outside.” Well, now it’s SA’s turn in the tank and the dynamic shops are enjoying the reversal.

But, in a hybrid world–where the major tool vendors offer both static and dynamic tooling this is technically absurd. AppScan’s explanation on “String Analysis” is particularly absurd. They’ve chosen to attack the effectiveness of static analysis with their ‘solution’, a static analysis technique.

It’s true that both Fortify and Ounce principally find SQL-injections through data flow techniques that propagate ‘taint’ and rely on tagging source, sink, and cleansing logic at the resolution of function calls. But I can say with authority that both tools model the potential values of strings as they propagate their data flow analysis as well. AppScan’s innovation isn’t so innovative.

Yes both SA tools, in my opinion, leave a bit to be desired in the core products’ support for what aspects of string propagation and manipulation they can follow. I won’t get into detail, but I advise security managers that it’s worth it to understand where the tool will fail you in this regard.

Coverity and the now defunct CodeAssure did an exceptionally good job modeling string values throughout code’s ‘speculative execution’, as part of their static analysis. When I tested these tools, they surprised me in both what they were able to find and what other tools’ false positives they left out of reports. Alas, these two don’t help today’s security manager much if they’re focusing on Java EE or .NET web applications. CodeAssure is gone and Coverity’s product has (in my opinion) limited language support outside C/C++ (though, again, I can not stress enough how positive my experience with it in C/C++ has been).

The dynamic shops had to level the playing field. But, as near as I can tell, the current situation is this:

• The major vendors believe there’s benefit to static and dynamic analysis
• There’s a lot of room for technical improvement in the market leaders’ SA products, with respect to modeling
• The future’s tool will leverage static and dynamic techniques, because each is suited for find a particular class of problems well.

You, as a security manager, will still need to sift through the marketectures and promises and figure out which tool works best on the kind of code your organization builds/buys. A major component of this will be how customizable the tool is.

Over-reliance on ANY automated tools (static or dynamic) leaves you with un-found vulnerabilities and a false sense of security. Cigital’s assessment services rely on these tools for speed and scale, and so we’ve taken great pains to understand where their modeling bends and where it breaks. In our own practice, we augment static techniques with dynamic tests and have even begun writing some next-generation static techniques to counter these limitations. We’ve spent hundreds of hours helping many many clients wring more out of their preferred suite of tools with the same understanding: opting instead for less invasive customization and tuning.

The bottom line is (and I’ve been saying this since at least ’03 now), there are strengths to each tool and each type of analysis. NONE will get you to the assessment finish line alone. Anyone who tells you otherwise is sellin’ you a load.

Let the posturing begin.

I presented a brief summary of the study results at the recent “Making the Business Case for Software Assurance” workshop hosted by Carnegie Mellon’s Software Engineering Institute, and sponsored by the US Department of Homeland Security. I’ll also be presenting an even briefer summary of the results at the 24th Annual Computer Security Applications Conference in December.

In my new role at Cigital, I’m hoping to be able to expand the survey beyond software vendors into e-commerce vendors, embedded software suppliers, financial institutions, etc., as well as to systematize the survey so it can be done by filling out a web form instead of as an interview. I welcome your suggestions as to how to make this project more relevant to vendors and software purchasers – and also welcome your participation in the survey, as well as suggestions on how to fund the ongoing work!

And finally, thanks to the (anonymous) vendors who participated in the first phase of the project. While I can’t thank them by name, I very much appreciate their input.

Some pointers:

From Slashdot
Posted by: kdawson, on 2008-01-09 01:20:00

Stony Stevenson alerts us to a US Department of Homeland Security program in which subcontractors have been examining FOSS source code for security vulnerabilities. InformationWeek.com takes a glass-half-empty approach to reporting the story, saying that for FOSS code [1]on average 1 line in 1000 contains a security bug. From the article: ‘A total of 7,826 open source project defects have been fixed through the Homeland Security review, or one every two hours since it was launched in 2006…’ ZDNet Australia prefers to emphasize those FOSS projects that [2]fixed every reported bug, thus achieving a clean bill of health according to DHS. These include PHP, Perl, Python, Postfix, and Samba.

Firstly, I am a big fan of code scanning and believe that use of static analysis tools should always be one of the basic security steps integrated into every SDLC. However, there are huge problems with declaring security after passing a code scan with an arbitrary tool and a random set of rules. The most obvious issue is that security defects come in two flavors—bugs and flaws—each accounting for roughly 50% of defects in practice. Code scanning tools can only find bugs. Here are two stupid examples for effect: can a code scanning tool determine that no user authentication was performed? How about whether or not a playback attack will work?

The second most obvious problem is that the list of rules enforced by a static analysis engine can never be complete. Discussion about this is left as an exercise for the reader.

Architectural risk analysis (crazily called “threat modeling” by Microsofties) is, like code scanning, an essential software security best practice. Formal methods are one way to go about attacking the flaw problem. In the US we rely on flakier heuristic-based approaches such as the one we use at Cigital. But no matter the approach, we can’t ignore the architecture.

References

1. http://www.informationweek.com/story/showArticle.jhtml?articleID=205600229&cid=RSSfeed_IWK_All
2. http://www.zdnet.com.au/news/security/soa/11-open-source-projects-pass-security-health-check/0,130061744,339284949,00.htm