Let the posturing begin…

by jOHN on Thursday, January 22, 2009

Myself and others have been getting Webinar invites from IBM’s developerworks regarding Rational’s AppScan Developer Edition.

This is of course part of the re-launch of the re-tooled AppScan product. It now includes a set of analysis types, some static and some dynamic. They’ve got fancy new names for subsets of each, some hair splitting to my ear and I’ve been reading/writing on this topic for 10 years.

The market for static analysis is plumping nicely year over year. Static analysis (SA) vendors, such as Fortify, are one-or-two revisions into producing dynamic analysis tools and suites that leverage hybrid approaches. With the big dynamic analysis tools controlled by IBM and HP a certain amount of this cross-over (and much more I predict) was inevitable.

It was also inevitable that that dynamic shops would take shots at the SA space. The dynamic guys got shot up pretty bad as their tools–sold as application security’s first ‘silver bullet’–begun to meet with resistance on how much manual effort was still required and on how many false positives and/or missed vulnerabilities remained. Static analysis swept in promising a better solution. “Look at the source code”, they smiled, “and you can be more complete and accurate than if you just poke the application from the outside.” Well, now it’s SA’s turn in the tank and the dynamic shops are enjoying the reversal.

But, in a hybrid world–where the major tool vendors offer both static and dynamic tooling this is technically absurd. AppScan’s explanation on “String Analysis” is particularly absurd. They’ve chosen to attack the effectiveness of static analysis with their ‘solution’, a static analysis technique.

It’s true that both Fortify and Ounce principally find SQL-injections through data flow techniques that propagate ‘taint’ and rely on tagging source, sink, and cleansing logic at the resolution of function calls. But I can say with authority that both tools model the potential values of strings as they propagate their data flow analysis as well. AppScan’s innovation isn’t so innovative.

Yes both SA tools, in my opinion, leave a bit to be desired in the core products’ support for what aspects of string propagation and manipulation they can follow. I won’t get into detail, but I advise security managers that it’s worth it to understand where the tool will fail you in this regard.

Coverity and the now defunct CodeAssure did an exceptionally good job modeling string values throughout code’s ‘speculative execution’, as part of their static analysis. When I tested these tools, they surprised me in both what they were able to find and what other tools’ false positives they left out of reports. Alas, these two don’t help today’s security manager much if they’re focusing on Java EE or .NET web applications. CodeAssure is gone and Coverity’s product has (in my opinion) limited language support outside C/C++ (though, again, I can not stress enough how positive my experience with it in C/C++ has been).

The dynamic shops had to level the playing field. But, as near as I can tell, the current situation is this:

  • The major vendors believe there’s benefit to static and dynamic analysis
  • There’s a lot of room for technical improvement in the market leaders’ SA products, with respect to modeling
  • The future’s tool will leverage static and dynamic techniques, because each is suited for find a particular class of problems well.

You, as a security manager, will still need to sift through the marketectures and promises and figure out which tool works best on the kind of code your organization builds/buys. A major component of this will be how customizable the tool is.

Over-reliance on ANY automated tools (static or dynamic) leaves you with un-found vulnerabilities and a false sense of security. Cigital’s assessment services rely on these tools for speed and scale, and so we’ve taken great pains to understand where their modeling bends and where it breaks. In our own practice, we augment static techniques with dynamic tests and have even begun writing some next-generation static techniques to counter these limitations. We’ve spent hundreds of hours helping many many clients wring more out of their preferred suite of tools with the same understanding: opting instead for less invasive customization and tuning.

The bottom line is (and I’ve been saying this since at least ’03 now), there are strengths to each tool and each type of analysis. NONE will get you to the assessment finish line alone. Anyone who tells you otherwise is sellin’ you a load.

Let the posturing begin.

5 Responses to “Let the posturing begin…”

  1. If you were worried that your food was poisoned, would you pay a wandering magician to strain the poison out? Good luck.

  2. My thought was that if you don’t trust the cooking process, you are in trouble that post-cooking-processing can’t fix.

    I think finding security holes is a halting problem: that is, provably impossible. Not to say we shouldn’t look for security holes, just that we know we can’t find them all. Security holes are like poison… if you miss finding just a little bit, you are still poisoned.

    (Suppose the wandering magician can look at your food and say, “Yup, no arsenic in this food.” But he’s unaware of cyanide.)

    Static analysis is one leg of the table. Testing is another. Neither is sufficient to trust a piece of software. What is needed in addition is knowledge about WHO built the software and WHAT they did to make it bug-free. If you face programs built by strangers, using methods and tools known to be unsafe, you start with a level of risk. Static and dynamic analysis can reduce part of that risk.

  3. jOHN says:

    Tom,

    I don’t think you’ll get any Cigital takers for an argument that “building security in” is preferable to an attempt to strain vulnerability from your software soup after it’s cooked (to continue your metaphor). One of Gary’s last books was titled “Building Security In”.

    I’m a bit concerned about the inversion you’ve posited in the next paragraph though. Simply put: all soup made today is poisoned. There’s just no one shipping/deploying vulnerability-free software. It’s true that knowing you’re not going to die of arsenic tells you nothing about thallium or anti-coagulants in your brew… but that doesn’t mean we shouldn’t use tools such as static and dynamic analysis before we ship/deploy to detect what we can find.

    If assurance activities reduce to the halting problem (they do), then your solution is equally imperfect: it reduces to the question, “Can one get dirty water from a clean pipe?” Yes. Yes (s)he can. Microsoft and other ‘secure’ SDLs will not guarantee that the “water” poured from your software development “pipe” will be clean. Indeed, every secure SDL mandates assurance activities in addition to constructive steps.

    In short, we need to do both constructive and assurance activities to reach the current state of the art in secure software development. I’d argue that static and dynamic tools will help in this pursuit–both with consistency/scale and with depth (if used correctly).

    In fact there are whole other classes of poison (such as bacterial or viral) which we see as the metaphorical equivalents of deployment misconfiguration or architectural flaws) we’ll want to check for too. Your static and dynamic tool kits might not help you reliably find these problems. This just means that we’re going to have to augment them with constructive or assurance-based manual techniques.

    …as I frequently say in conference presentations, “Software security is still in diapers… any one who tells you otherwise is full of … well… the reason you wear diapers.” :D

  4. Crappy Static Analysis Tools Make Me Appreciate Strong Typing says:

    static analysis tools are great, until the point where you end up outside the comfy predefined and preoptimized cases that are packaged with the product and try using it on particularly custom or odd software.

    Then they are a LOT of hassle, as you have to more or less write and test your own small suite of analysis rules to replicate the same coverage you would be getting if you were doing something more standardized.

    while writing your own rules is certainly possible, the vendors I’ve seen put very little effort into making this not as untested, pseudosupported, and complicated as they possibly can.

  5. jOHN says:

    “Crappy”,

    Oh boy–there’s a lot to deal with here. First off: Yes, I agree static analysis tools (and the companies that sell them) have definitely treated customization as a second-class citizen or worse. I have to hand it to Fortify though, when Cigital sold them our technology, we vehemently stressed the need for extension and they listened. Even still, I agree, documentation and support is alpha-level in maturity or worse for customization.

    Still, for those who can surmount the learning curve for customization, dramatic depth and accuracy gains will be yours. We’ve seen this across the board in both customizing Ounce and Fortify’s tools. We’ve seen similar benefits from ‘tuning’ Coverity’s tools.

    To give you a feel for what you can expect, I think 40% false-positive reduction is a conservative goal in the Java SE/EE rule space. Likewise, I expect that fully 50% of an organization’s corporate standards can be automated reasonably within a single calendar year (with far less than a man-year of effort).

    Eric Dalci (one of my favorite Cigital guys for static analysis) and I talk on this topic at conferences frequently and have created a framework for tool customization:

    http://www.cigital.com/papers/download/Framework%20for%20Custom%20Rules.pdf

    I’m not the only one that can customize these tools either. At least seven Cigital consultants are quite exceptional at it. Most of my clients have one or two individuals who I’d label “quite capable” as well.

    If you’d like additional information on this topic, please email me personally. Tools left on the shelf are of little value ;-)

Leave a Reply