James McGovern recently wrote a post on Gartner’s static analysis (SA) report. Among other things, he lamented the lack of actionable guidance within the report. A lack of implementation guidance doesn’t shock me from Gartner, I can’t say I expect that from them.

I can help James and community out by giving some of that guidance myself. I’ll try to do so in a tool-independent way. This topic deserves more than a “blog entry” format (admitting Cigital’s propensity for longer entries ;-)) so if people want more information on the topic, they can ping me.

In essence, you want to know the following about static analysis tools:

1. Who is going to run it (how will owners shepherd it)?
2. How much effort is it going to take to run (what budget will I need to support it)?
3. When do I run it and how often (Where does it fit within my SDL?)
4. What it’s going to find (How will the results enter/impact my organization)?
5. What won’t it find (How does it play with the rest of the assurance ecosystem)?

[Market Adoption and Making Your Choice]
Gartner seems underwhelmed by Microsoft’s ability to effect the commercial static analysis market. Cigital has always believed these technologies would make great features of a compiler/IDE and so we’re also disappointed. They’ve got great technologies but don’t exploit them commercially. I still like their freely available FXCop but this opinion isn’t popular. Perhaps that’s because I see a lot of value to FXCop as a unit testing framework that enables security test and other people are comparing it to glossy commercial tools like Fortify and Ounce run by security analysts in a very hands-off fashion. Who’s considering the tool really does make a huge difference in how it will fare.

From my perspective, the most successful SA tool vendors have made answering the “who should adopt the tool” question more difficult as they’ve waffled on licensing models. By the seat? By the core? By the KLoC? I’ve gotten complaints from prospective tool buyers indicating that a leading tool vendor’s price was 10X another vendor’s. Before you ask “Which one?”: I’ve heard specific examples in opposite directions!

Certain tools will look better to different potential buyers. Experience has shown positive response to Coverity’s C/C++ results from developers. Application security groups like Fortify and Ounce’s products, and several years ago, I saw QA Managers in two very different firms go ga-ga over Klockwork’s tool. I attribute people’s impressions not to a product vendor focusing on quality or security but to how closely the tool vendor’s conception of how roles interface with their tools throughout the vulnerability management and software development life cycles matches the organization’s own structure. Up until a few years ago I don’t believe SA vendors had fully conceived the “developer”, “analyst”, “security manager” roles in their products. Ounce was the one exception: they’ve always staunchly believed themselves to be principally a Security Analyst’s tool. Some vendors attempted to unify roles into use of a single interface while others attempted to split the product up into different configurations for each. This created confusion and friction between some adopting organizations and the tool vendor’s license model.

In the end you want SA tools to affect as many developers as possible and to get low-latency scan results as early in software’s life cycle as possible. However, having successfully deployed tools a variety of ways, I’m convinced central deployment is the way to go. Application Security should run the tool. This will help manage rule/configuration update, as well as central measurement collection, risk management, and issue tracking. How do we get low-latency and early-lifecycle centrally? Treat the tool as a service. More on that later.

Notable exceptions to the central deployment model include business units possessing one or more of the following characteristics:

• Acquisitions of previously successful, culturally-independent companies
• High-quality product development teams
• Fundamentally different SDL, development platform, language, and toolkits

For in-unit deployment one can “run locally” but must “report centrally” supporting security goals involving measurement and risk management. And, let’s be clear: if a business unit has a wicked-good continuous integration/build-management group, they-by all means-should be tapped to integrate with the SA service. Cigital and its clients have had success in integrating both the front (code submission) and back-end (bug/issue tracking) with Fortify and a variety of open source build/CI/bug-tracking products. This maintains a central model, but reduces latency dramatically.

[How Much Effort]
I’m sorely disappointed in the honesty of tool vendor sales-folk regarding level of effort. One of my first analogies for SA tools is this: Did your company buy Mercury products for load and performance testing? Think of your SA tool like Mercury–you’ll need at least as much money to deploy and implement the tool organization-wide as the licenses cost you. Sorry.

Organizations with more than 2-300 developers should plan on the following:

• 3-4 man weeks to do initial tuning and pilot implementation
• 4-16 man hours / large application to integrate with the static analysis tool and assure appropriate configuration
• 4-8 man hours / 50-100KLoC (language-/complexity-dependent) to triage results on a new-application scan
• 1 person / year as tool shepherd (app integration, custom-rule creation, rule-pack maintenance and release, support)

Organizations get into trouble when their SA tool enjoys broad acceptance by the organization before they effectively manage the submission and results triage/documentation process. My data indicates code submission can burn 8-24mhrs / application if security groups don’t ‘remember’ (or automate) the submission and scan setup process. Security groups doing scans centrally and hand-writing code review results can waste another 16hrs documenting their findings / application.

…think about it: if your organization allots a week / tool-assisted code review and wastes 24 hours getting the tool’s first result set and 16hrs documenting findings… that only left 10 hrs to actually review the code! Since I believe that 4-8 hrs / 50-100KLoC is necessary to triage, you can imagine how deep I think such code reviews are :-P At Cigital, we’ve got dramatic benefits to depth by decreasing cost of submission and reporting. This is key to scaling SA efforts. Those adopting tools will feel overwhelmed if they don’t plan to solve submission and reporting problems before wider roll-out.

[How Often]
As often as possible. If you’ve got a Continuous Integration initiative (CI), engage those folk. If you deploy your tool purely centrally, plan to scan each high-risk application at least once / release. No, to my knowledge, there isn’t a good answer to the “can these tools do incremental (partial) scans”. You should always keep old results around though, as they’ll (along with other measures) help you understand whether more or less vulnerabilities are being caught during each scan. Remember though, every time you add a rule to the scan, your vulnerability-finding bar just moved and your metrics might be thrown off.

Ounce, Fortify, and Klocwork seem pretty good about determining whether or not they’ve found a particular issue in a previous scan or not. This includes situations where code blocks move around a bit. This means that running regression scans to determine whether or not an issue has been fixed is viable. I can’t comment on Coverity’s capabilities here.

[What's it going to find]
I’m impressed that Fortify has published its vulnerability taxonomy and a bit shocked that others haven’t produced as public a resource. I’m excited too that vendors have agreed to comply with CWE labels when they report findings. I will say what I always do on this topic though: test the tool on your own code though. You simply don’t know what it will find until you throw all your idiosyncratic build, language, toolkit, and platform nonsense at the particular tool you’ve selected.

[What won't it find]
I’m continually shocked as to how much one can find by running tool Y after tool X on the same piece of code (replace X and Y as you see fit). As I’ve written time-and-time-again, the corner cases and exceptions missed by each engine are baffling. Let me put things in perspective for you: three of my clients have reported that their SA tool deployment find 17, 33, and 50% of the total number of issues found by assessment efforts respectively. If these percentages hold for your organization, inside a coding bug realm then your job is at best 50% done having run the tool and triaged its results.

Considering that Microsoft and Cigital believe that “bugs” account for only 50% of the vulnerability space, that leaves you unaware of 75% of your application’s vulnerability at the end of triage. As I always say, “use the tool to facilitate code review and increase a reviewer’s understanding of the code–not as a code reviewer that finds bugs automatically.”

Good luck out there, I’d love to hear about your particular experiences.
-jOHN

I was talking with one of my consultants the other day about a common confusion Developers sometimes have regarding a pretty mundane piece of security guidance. Specifically, “What does it mean I have to turn off logging/debug in production?”

In my mind, these two separate issues exist here, intertwined.

Almost every logging framework has an in-place configuration option for increasing/decreasing log-level; most allow you to reconfigure on a running system without restart. The point here being that an application developer, with use of such a package, get to 1) keep a single code base, 2) turn a high level of logging off by default in a production environment, and 3) ratchet up log-level in circumstances requiring it (incident response or in-place debugging).

Often, people mix the above guidance with “don’t roll debug code out to production.” Code Analysis tools, for instance, give this advice. Two things principally concern me here. First, you don’t want a bunch of “main” or other “debug-only” entry points to the code. Attackers capable of compromising host systems could shoe-horn their way into an application this way. The situation becomes worse when said debug interfaces are web-based. Second, especially for thick clients or code running in less trusted environments, debug code tends to ‘express’ things about the design (or leak critical data) in an undisciplined manner. Removing such code from deployments to untrusted zones CAN be advisable, but one must balance complete removal with the need to ‘turn on’ nascent debug code in times of need as described by the previous paragraph. Only protection of the most sensitive information, and most invasive code qualifies for the extreme guidance “remove before deploying” that this paragraph describes.

The best thing one can do is give the developer guidance as to what types of test/debug code might qualify for wholesale removal. I’ll start you a list:

1. Client ‘harness’ code allowed to connect to server interfaces w/o
   authentication
2. Client code allowed to edit/manipulate local data repositories/stores
   directly
3. Command ‘harnesses’, allowed to (re)set or manipulate server state
4. Client code that dumps secret keys, tokens, or stored-but-otherwise-protected credentials
5. Backdoors
6. Code allowing for ‘insecure failure’ or back off, say to an older protocol
7. ‘Re-implementation’ outside a sandbox (native equivalents of a managed binary)

Just a small tidbit, away from the glib towards concrete guidance.

-jOHN

Technorati Tags: logging, debugging

We’re pleased to have a guest blogger for this Justice League entry. Michael Cohen is a Senior Security Consultant at Cigital where he is responsible for leading, assessing, architecting and implementing secure software for Fortune 500 companies. Michael also works with Cigital teams on enterprise-wide security solutions intended to improve an organization’s security posture and help them meet audit and regulatory requirements. Michael just gave a talk to the Washington, DC IASA chapter that was well received, and is now the subject of this entry:

When I hear software architects talk about the architecture they’ve crafted to depict the various structures and behaviors of a system, they point out interesting techniques they’ve applied that best convey how the system should work and be put together. An architect’s enthusiasm for the little details reveals a great sense of accomplishment and creativity, but most of all good architecture conveys sorely needed information in order to help those who develop, test and maintain the system. Sharing the tips and tricks gathered from the field is what helps our community move forward to building better software. A classic example of this is the Design Patterns book written by the Gang of Four.

Not too long ago I was also sharing my ideas on architecture and security at a local IASA chapter here in Washington, DC. The group was a crowd of like-minded architects who work for large Fortune 500 companies, government agencies, and promising local startups. My topic was pragmatic secure architecture. The basic idea was to look at some real ways to incorporate security into architecture using Cigital’s risk management and threat modeling practices. You can find the slides for my presentation here.

For the uninitiated, threat modeling is a way of depicting where threats (think malicious people, attackers, and so on) can touch a software architecture and how they may be able to exploit critical assets using various attack patterns. Threat modeling is valuable for determining an architecture’s security posture. In addition, identifying risks in user requirements and business goals, and tying those risks to a threat model results in a map of how design flaws impact the system, its users and the overall business. Threat models coupled with identified high-level risks are a great way to get other stakeholders involved with security decisions. And mind you, these are stakeholders who would otherwise be unable to contribute and supply critical feedback.

The attendees at the chapter meeting were glad they attended the presentation and heard something worthwhile that they could use in their daily architectural activities. Many people brought up interesting points about how to best protect critical assets, what a real risk to a system is, and what is considered a good enough mitigation. What I found particularly interesting was how threat modeling provided a way for everyone to contribute interesting ideas about where security vulnerabilities may exist and how they could be mitigated (which, if you think about it, is the entire point of threat models).

I encourage any architect to share their ideas on building a better architecture with their peers, whether it be at a local organization or at a conference. As a senior security consultant at Cigital, I like to take shared ideas and mold them into my own unique way of thinking about the world. The best results come when I can apply new ideas to my daily activities as I help our customers assess and create more secure software.

… On an entirely separate note, McGraw still owes me money. I’m watching you, Gary.

Technorati Tags: software architecture, software quality, software security, touchpoints

One of the commonly proposed remedies for SQL Injection is to use SQL stored procedures. Use of stored procedures can greatly reduce the likelihood that you’ll code an SQL injection, but it’s not the stored procedure-ness that’s saving you. Stored procedures let you use Static-SQL instead of forcing you to always use Dynamic-SQL. In Static-SQL the metadata for the query is known at compile-time whereas in Dynamic-SQL the program constructs the query at runtime. Dynamic-SQL is injectable; Static-SQL is not.

For most programs, Static-SQL is sufficient. Only a small subset of features such as user driven filtering interfaces need Dynamic-SQL’s flexibility. Unfortunately, the common database access protocols, ODBC and JDBC, are based on Dynamic-SQL and there’s no provision in the interface for static queries. It’s into this void that SQL stored procedures stepped in.

But SQL stored procedures aren’t limited to Static-SQL. It’s possible to use Dynamic-SQL through the PREPARE, EXECUTE and EXECUTE IMMEDIATE statements (these are the ANSI SQL-92 statement names, your RDBMS have different names). In fact, ODBC and JDBC are just wrappers that ship the SQL text from your program to the database server and make calls to the RDBMS through the runtime support for these Dynamic-SQL statements (actually). So, stored procedures that execute Dynamic-SQL statements are equally vulnerable to SQL injection. It doesn’t matter if it’s Java code concatenating strings and passing them to JDBC or if the strings are concatenated in PL/SQL. Both are equally vulnerable.

By now you’re probably wondering why I’m splitting this hair. I’m splitting it because I want to ensure that we protect ourselves from the programmer we’ve not yet met. It’s the programmer that goes and dutifully reads our coding guideline that says “Use SQL stored procedures to prevent SQL Injection” and now has to extend our application with the feature that introduces Dynamic-SQL. But, because we haven’t properly articulated the guidance, s/he creates an SQL injection vulnerability.

Implementing your queries using Static-SQL will go a long way to protect your application from SQL Injection. It’s also possible to simulate Static-SQL using ODBC/JDBC type interfaces with the religious use of SQL parameter markers. This leaves the use of Dynamic-SQL only for truly dynamic queries like the dynamic user-defined filter I mentioned above. There’s a pattern for securely implementing such a filter that’s floating around in my head. I need to write that down, but you’ll have to wait until next month. Sorry.

Technorati Tags: sql, defects, flaws