Justice League Blog

On Open Source

Posted On January 9, 2008 by gem

There has been a recent flurry of activity regarding security assurance on a hush-hush open source mailing list I lurk on. The debate recently has to do with formal methods versus code scanning… apples and oranges in my view. However, there’s a new flurry of press over Coverity’s use of their tool to analyze well-known globs of open source. (One poster suggested that passing a scan like this with flying colors means security has been attained… argh!)

Some pointers:

From Slashdot
Posted by: kdawson, on 2008-01-09 01:20:00

Stony Stevenson alerts us to a US Department of Homeland Security program in which subcontractors have been examining FOSS source code for security vulnerabilities. InformationWeek.com takes a glass-half-empty approach to reporting the story, saying that for FOSS code [1]on average 1 line in 1000 contains a security bug. From the article: ‘A total of 7,826 open source project defects have been fixed through the Homeland Security review, or one every two hours since it was launched in 2006…’ ZDNet Australia prefers to emphasize those FOSS projects that [2]fixed every reported bug, thus achieving a clean bill of health according to DHS. These include PHP, Perl, Python, Postfix, and Samba.

Firstly, I am a big fan of code scanning and believe that use of static analysis tools should always be one of the basic security steps integrated into every SDLC. However, there are huge problems with declaring security after passing a code scan with an arbitrary tool and a random set of rules. The most obvious issue is that security defects come in two flavors—bugs and flaws—each accounting for roughly 50% of defects in practice. Code scanning tools can only find bugs. Here are two stupid examples for effect: can a code scanning tool determine that no user authentication was performed? How about whether or not a playback attack will work?

The second most obvious problem is that the list of rules enforced by a static analysis engine can never be complete. Discussion about this is left as an exercise for the reader.

Architectural risk analysis (crazily called “threat modeling” by Microsofties) is, like code scanning, an essential software security best practice. Formal methods are one way to go about attacking the flaw problem. In the US we rely on flakier heuristic-based approaches such as the one we use at Cigital. But no matter the approach, we can’t ignore the architecture.

References

  1. http://www.informationweek.com/story/showArticle.jhtml?articleID=205600229&cid=RSSfeed_IWK_All
  2. http://www.zdnet.com.au/news/security/soa/11-open-source-projects-pass-security-health-check/0,130061744,339284949,00.htm
This entry was posted in Assurance Software Security Software Security Touchpoints. Bookmark the permalink.
« »
  • http://www.multicians.org/thvv/tvv-home.html Tom Van Vleck

    Code scanning is one leg of the table.
    Code review is another. You could apply
    http://www.multicians.org/thvv/nasty.html
    and replace “quality” by “security.”

    We could write zero defect code that does exactly what we want
    but often choose not to: too hard, takes too long.

    Even if we do everything right, there will still be bugs
    because sometimes we don’t want the right things.

  • http://www.cigital.com/~gem gem

    Agreed. I suppose when I think code scanning (and say it) I really mean code review. A fool with a tool is still a fool!

    gem

  • Patrick Lind

    Even though I am totally lost reading your comments on online poker, I am so excited and pleased to see that someone is addressing the problem of how the cards are dealt and how far removed the game is from sitting down and playing at a holdem table with a real deck of cards. I have been playing poker almost all my life and knew that the hands being delt online were not following the trend of a real game being dealt by a real person using a real deck of cards. Some of the things I’ve seen online defy all odds. Now that you have enlightened me on at least some of the problems, can you comment on whether or not there are any fair online poker sites in your opinion and also are some poker games less affected than others. By that I mean for example is five card draw less affected than holde or ohmaha. You have been very helpful, I only wish I understood computers a little better.
    Thanks again,
    Patrick

  • http://www.cigital.com/~gem gem

    Hi Patrick,

    I’m not really sure what you’re talking about…perhaps a different article? But anyway, to learn more about what we did to move online poker in the right direction, see:

    http://www.cigital.com/papers/download/developer_gambling.php

    Also note that my book “Exploiting Online Games” has some discussion of poker security as well. Hope that helps.

    gem

  • http://www.cigital.com/~gem gem

    To everybody else,

    For those of you following this thread for what it really is…

    You might be interested in an expanded version of this posting recently published as a darkreading article:

    http://www.darkreading.com/document.asp?doc_id=146053&WT.svl=column1_1

    gem

  • http://www.multicians.org/thvv/tvv-home.html Tom Van Vleck

    Gary said
    > Code scanning tools can only find bugs. Can a code scanning
    > tool determine that no user authentication was performed? How
    > about whether or not a playback attack will work? (Just for
    > the record, the answer is “no way??? in both cases.)

    You have some particular kind of “code scanning” in mind. Rule based tools that have a rule requiring authentication would notice if it was missing. Code review can range across
    - hey Gary, do you see anything wrong with this program?
    - let’s run BugFinderMax over this module
    - let’s apply a Belady-style formal inspection with a project tailored checklist
    and many other variations.

    Different code review processes will miss different bugs and find others, whether implemented in silicon, rulebook, or experienced person. It is a mistake to think that any method will solve problems that not explicitly targeted.

    I bet there is a theorem here: given a bug finding tool, there is a bug it can’t find; given a bug, a tool can be constructed to find it.

  • http://www.cigital.com/~gem gem

    Hi Tom,

    I’m pretty sure that what you’re suggesting can be done to find out whether authentication code is present is an NP-hard problem (not to mention AI-Complete). You’re right that I was addressing automatic code review with a static analysis tool like Fortify or Coverity.

    In any case, I am more concerned about our inability to find FLAWS than our ability to find all BUGS.

    gem

Recent Posts

Categories

Recent Comments

Download Our Media Kit