On Open Source
There has been a recent flurry of activity regarding security assurance on a hush-hush open source mailing list I lurk on. The debate recently has to do with formal methods versus code scanning… apples and oranges in my view. However, there’s a new flurry of press over Coverity’s use of their tool to analyze well-known globs of open source. (One poster suggested that passing a scan like this with flying colors means security has been attained… argh!)
Some pointers:
From Slashdot
Posted by: kdawson, on 2008-01-09 01:20:00Stony Stevenson alerts us to a US Department of Homeland Security program in which subcontractors have been examining FOSS source code for security vulnerabilities. InformationWeek.com takes a glass-half-empty approach to reporting the story, saying that for FOSS code [1]on average 1 line in 1000 contains a security bug. From the article: ‘A total of 7,826 open source project defects have been fixed through the Homeland Security review, or one every two hours since it was launched in 2006…’ ZDNet Australia prefers to emphasize those FOSS projects that [2]fixed every reported bug, thus achieving a clean bill of health according to DHS. These include PHP, Perl, Python, Postfix, and Samba.
Firstly, I am a big fan of code scanning and believe that use of static analysis tools should always be one of the basic security steps integrated into every SDLC. However, there are huge problems with declaring security after passing a code scan with an arbitrary tool and a random set of rules. The most obvious issue is that security defects come in two flavors—bugs and flaws—each accounting for roughly 50% of defects in practice. Code scanning tools can only find bugs. Here are two stupid examples for effect: can a code scanning tool determine that no user authentication was performed? How about whether or not a playback attack will work?
The second most obvious problem is that the list of rules enforced by a static analysis engine can never be complete. Discussion about this is left as an exercise for the reader.
Architectural risk analysis (crazily called “threat modeling” by Microsofties) is, like code scanning, an essential software security best practice. Formal methods are one way to go about attacking the flaw problem. In the US we rely on flakier heuristic-based approaches such as the one we use at Cigital. But no matter the approach, we can’t ignore the architecture.
References
January 10th, 2008 at 12:05 pm
Code scanning is one leg of the table.
Code review is another. You could apply
http://www.multicians.org/thvv/nasty.html
and replace “quality” by “security.”
We could write zero defect code that does exactly what we want
but often choose not to: too hard, takes too long.
Even if we do everything right, there will still be bugs
because sometimes we don’t want the right things.
January 10th, 2008 at 1:38 pm
Agreed. I suppose when I think code scanning (and say it) I really mean code review. A fool with a tool is still a fool!
gem
January 19th, 2008 at 1:37 am
Even though I am totally lost reading your comments on online poker, I am so excited and pleased to see that someone is addressing the problem of how the cards are dealt and how far removed the game is from sitting down and playing at a holdem table with a real deck of cards. I have been playing poker almost all my life and knew that the hands being delt online were not following the trend of a real game being dealt by a real person using a real deck of cards. Some of the things I’ve seen online defy all odds. Now that you have enlightened me on at least some of the problems, can you comment on whether or not there are any fair online poker sites in your opinion and also are some poker games less affected than others. By that I mean for example is five card draw less affected than holde or ohmaha. You have been very helpful, I only wish I understood computers a little better.
Thanks again,
Patrick
February 15th, 2008 at 9:58 am
Hi Patrick,
I’m not really sure what you’re talking about…perhaps a different article? But anyway, to learn more about what we did to move online poker in the right direction, see:
http://www.cigital.com/papers/download/developer_gambling.php
Also note that my book “Exploiting Online Games” has some discussion of poker security as well. Hope that helps.
gem
February 15th, 2008 at 10:00 am
To everybody else,
For those of you following this thread for what it really is…
You might be interested in an expanded version of this posting recently published as a darkreading article:
http://www.darkreading.com/document.asp?doc_id=146053&WT.svl=column1_1
gem
February 18th, 2008 at 5:36 pm
Gary said
> Code scanning tools can only find bugs. Can a code scanning
> tool determine that no user authentication was performed? How
> about whether or not a playback attack will work? (Just for
> the record, the answer is “no way” in both cases.)
You have some particular kind of “code scanning” in mind. Rule based tools that have a rule requiring authentication would notice if it was missing. Code review can range across
- hey Gary, do you see anything wrong with this program?
- let’s run BugFinderMax over this module
- let’s apply a Belady-style formal inspection with a project tailored checklist
and many other variations.
Different code review processes will miss different bugs and find others, whether implemented in silicon, rulebook, or experienced person. It is a mistake to think that any method will solve problems that not explicitly targeted.
I bet there is a theorem here: given a bug finding tool, there is a bug it can’t find; given a bug, a tool can be constructed to find it.
February 19th, 2008 at 7:59 am
Hi Tom,
I’m pretty sure that what you’re suggesting can be done to find out whether authentication code is present is an NP-hard problem (not to mention AI-Complete). You’re right that I was addressing automatic code review with a static analysis tool like Fortify or Coverity.
In any case, I am more concerned about our inability to find FLAWS than our ability to find all BUGS.
gem