Oh boy–there’s a lot to deal with here. First off: Yes, I agree static analysis tools (and the companies that sell them) have definitely treated customization as a second-class citizen or worse. I have to hand it to Fortify though, when Cigital sold them our technology, we vehemently stressed the need for extension and they listened. Even still, I agree, documentation and support is alpha-level in maturity or worse for customization.

Still, for those who can surmount the learning curve for customization, dramatic depth and accuracy gains will be yours. We’ve seen this across the board in both customizing Ounce and Fortify’s tools. We’ve seen similar benefits from ‘tuning’ Coverity’s tools.

To give you a feel for what you can expect, I think 40% false-positive reduction is a conservative goal in the Java SE/EE rule space. Likewise, I expect that fully 50% of an organization’s corporate standards can be automated reasonably within a single calendar year (with far less than a man-year of effort).

Eric Dalci (one of my favorite Cigital guys for static analysis) and I talk on this topic at conferences frequently and have created a framework for tool customization:

http://www.cigital.com/papers/download/Framework%20for%20Custom%20Rules.pdf

I’m not the only one that can customize these tools either. At least seven Cigital consultants are quite exceptional at it. Most of my clients have one or two individuals who I’d label “quite capable” as well.

If you’d like additional information on this topic, please email me personally. Tools left on the shelf are of little value

Then they are a LOT of hassle, as you have to more or less write and test your own small suite of analysis rules to replicate the same coverage you would be getting if you were doing something more standardized.

while writing your own rules is certainly possible, the vendors I’ve seen put very little effort into making this not as untested, pseudosupported, and complicated as they possibly can.

because arsenic will kill me, I’d like to scan for its existence as cheaply and thoroughly as I’m able to. Heck, I’d like to scan for as many things that might kill me as I can. Static analysis will find _some_ things, and dynamic analysis will find _other_ things. My experience is the set of things each finds overlaps slightly.

My concern, simply put, is that the tool vendors are increasingly claiming they sell the only ‘poison taster’ you’ll need. My experience is that this is never the case.

Based on the kind of software you as an organization ‘cook’, I suggest testing out poison-tasting capabilities of each a bit, and building yourself a pipeline that most inexpensively (and appropriately to your risk aversion) detects poison.

I don’t think you’ll get any Cigital takers for an argument that “building security in” is preferable to an attempt to strain vulnerability from your software soup after it’s cooked (to continue your metaphor). One of Gary’s last books was titled “Building Security In”.

I’m a bit concerned about the inversion you’ve posited in the next paragraph though. Simply put: all soup made today is poisoned. There’s just no one shipping/deploying vulnerability-free software. It’s true that knowing you’re not going to die of arsenic tells you nothing about thallium or anti-coagulants in your brew… but that doesn’t mean we shouldn’t use tools such as static and dynamic analysis before we ship/deploy to detect what we can find.

If assurance activities reduce to the halting problem (they do), then your solution is equally imperfect: it reduces to the question, “Can one get dirty water from a clean pipe?” Yes. Yes (s)he can. Microsoft and other ‘secure’ SDLs will not guarantee that the “water” poured from your software development “pipe” will be clean. Indeed, every secure SDL mandates assurance activities in addition to constructive steps.

In short, we need to do both constructive and assurance activities to reach the current state of the art in secure software development. I’d argue that static and dynamic tools will help in this pursuit–both with consistency/scale and with depth (if used correctly).

In fact there are whole other classes of poison (such as bacterial or viral) which we see as the metaphorical equivalents of deployment misconfiguration or architectural flaws) we’ll want to check for too. Your static and dynamic tool kits might not help you reliably find these problems. This just means that we’re going to have to augment them with constructive or assurance-based manual techniques.

…as I frequently say in conference presentations, “Software security is still in diapers… any one who tells you otherwise is full of … well… the reason you wear diapers.”

I think finding security holes is a halting problem: that is, provably impossible. Not to say we shouldn’t look for security holes, just that we know we can’t find them all. Security holes are like poison… if you miss finding just a little bit, you are still poisoned.

(Suppose the wandering magician can look at your food and say, “Yup, no arsenic in this food.” But he’s unaware of cyanide.)

Static analysis is one leg of the table. Testing is another. Neither is sufficient to trust a piece of software. What is needed in addition is knowledge about WHO built the software and WHAT they did to make it bug-free. If you face programs built by strangers, using methods and tools known to be unsafe, you start with a level of risk. Static and dynamic analysis can reduce part of that risk.

Haha! I might pay such a magician if (s)he came with a provable track record

Allow me to propose another metaphor:

I realize the mid-market will do little else than turn the key and kick the tires before they buy the car (static or dynamic)… my objective is simply to get them to stop believing the commercials and actually take the thing for a test drive. Test driving the car/truck might even give them a better feel for what will meet their needs than the ‘magician’ might discern from his tower–notwithstanding its superior view :-p

And, like most families, those who research beyond the test drive they may find that they can fully satisfy their needs more cheaply with both a minivan and a four-door sport coupe rather than buying into some all-things-to-all-people vehicle, like the ‘sporty’ Mazda CX-7 or the Porsche Cayenne S.