We can’t tackle the “process improvement” problem with our open source provides like we can with those who build our software in-house, or those vendors from whom we acquire code.

Understanding, based on this, we lack as many ‘knobs and dials’ for improving the cleanliness of the pipes through which open source software flows… we may have a flow rate problem as well. Though I lack hard evidence, I’d bet this represents a proverbial iceberg’s tip:

An organization may deploy as much or more open source code as their own in-house developed code.

It’s interesting to think about the money organizations spend to secure the code they build vs. the amount of open source they consume in this light… (participants indicated they didn’t track this spend separate from their development efforts).

Harumph. Our breakout group generated great ideas worth sharing. First, we unearthed a lot of things attending organizations already do. Next, we brainstormed valuable next steps. Here’s categories of activities we came up with:

• Inventory & Inventory Control


• Vulnerability Identification (Assessment)


• Vulnerability Management


• Ownership of Open Source


• Policy (use)


• Policy (Contribution)

What did each entail?

Inventory & Inventory Control

• Identification – All Participating organizations had a manual discovery process for open source usage. Many wanted better automated schemes, despite some existing tool usage.
• Identification of masked open source – Many participating organizations realize that not only do they adopt open source software directly but that the 3rd party code (and open source) they absorb also contain open source software. Discovering this ‘masked open source’ software represents as big a problem as identifying the open source software used directly.
• Centralized open source distribution – Some organizations allow development & deployment of open source software from the web directly whereas others only allow access to and use of open source from a centrally managed repository. Centralizing deployment usage may provide only improved integrity, or may be used to implement an ‘approved package list’.

Vulnerability Identification (Assessment)

• Using assessment tools – About three-quarters of workshop respondents used the same tools they use to assess their application security posture on their open source assets. This, to me, seems worth its own blog entry. I had to wonder aloud, “as organizations move from detective assessment schemes (SCR, PT) to so-called preventative (threat modeling, architecture analysis, misuse/abuse cases) how do they consider open source?” I’m finding that organizations blazing trails into security architecture work commonly omit discussion of their open source frameworks.

Ownership

• Maintain a white list of open source – Seemingly related to the “centralized distribution” item (but surprisingly uncorrelated in our survey), this meant that someone, from security, owned assessing the risk and proffering a “thumbs-up” or “thumbs-down” that can inhibit white list membership.


• Revisit white list – Organizations expressed that they found value in pruning the approved list of open source software based on non-use, newly identified risk, and similar factors. About one-quarter of our group engaged in this activity.


• Ownership of identified risk – Some participants avidly encouraged use of open source within their applications. In these organizations, when a developer chooses to include open source (as opposed to writing a widget themselves), they own any newly identified risk when in that open source. This reminded me eerily of Wall St. traders. Equity investment creates risk. Margin calls create leveraged risk. In this metaphor, choosing to adopt open source seems like a margin call. It’s very possible that a developer can absorb more risk into the organization than they themselves could effectively own up to in black-swan scenarios. It’s unclear how to measure this exposure when adopting open source.
• Collaboration – Certainly an organization-specific and an unsolved problem, participants indicated there may be a “third stakeholder” in the process of identifying and managing open source vulnerability. Two examples given were 1) clearing houses from which organizations purchase open source software and 2) support organizations (a la RedHat).

Vulnerability Management

• Root Cause Analysis – When a vulnerability is found (regardless of means or source: internal/external), organizations sometimes can point to a person or group who understands the vulnerable component (notable exceptions exist for purchased software or that software maintained by a development team that’s vanished). In open source, the organization must expend resources in order to “get smart” on vulnerability’s root cause and make trade-offs about mitigation strategies and their impacts. This represents extra cost on which I’d enjoy having greater visibility.


• Vulnerability Impact Analysis – Almost every participant had some regime by which they discovered (through assessment, feeds, or other means) new vulnerabilities within their adopted open source code base. Everybody possessed some ability to figure out which lines-of-business or development teams might be affected by newly discovered vulnerabilities.


• Patch Management – Only about one half of participants had, in their minds, a good strategy–having assessed the impacted teams–for distributing a patch that remediated open source vulnerability in an organization-wide manner. More strategically, several schemes seemed available to organizations beyond the straightforward “penetrate-and-patch” loop here. Alternatives included:
  
  
  1. Wrapping open source
     
  2. Hardening open source (and centrally distributing)
     
  3. Sandboxing/compartmentalizing open source
     
  
  

Policy (Use)

• Security Policy/Standards – About half participants had some kind of security policy or standards addressing how to securely use open source software within the organization.
• Training – About one quarter of participants trained their developers to use some portion of their open source securely. Interestingly enough, the one quarter of our respondents that trained developers did not line up well with the one half that had security standards. Wild.

Policy (Contribution)

• Legal permission to contribute to open source – Some organizations see open source contribution as a key activity. Others did but have suffered clamp-down from their legal departments because legal fears liability. Others never liked the idea.
• Community Notification on Vulnerability – When an organization contributes to open source and later finds (or is notified of) vulnerability in its contributions, it may need a way to notify the broader community. Organizations also complained about very high latency in the community notifying them of vulnerability in their code. This proved a surprising problem in our brainstorming session. Why? Contributors complained that this was because, often, their contributions were either 1) forked or 2) baked into other products that masked use of the contributions. In either case, it wasn’t evident to those disclosing the vulnerability that the (contributing) organization was responsibly for vulnerable code.

I will absolutely not let it go without saying that though this entry contains many of my own thoughts it heavily relies on the work of many in our breakout session, well-lead by HP’s Brian Chess. Thanks all for a great discussion.
-jOHN

There has been a lot of talk about BEAST (Browser Exploit Against SSL/TLS) lately. The attack against SSL 3.0 / TLS 1.0 was recently publicized by Thai Duong and Juliano Rizzo. Do you know what the risks are, and how to protect yourself?

How does it work?

The attack has two components, and the goal is for the attacker to get your cookie so that he can hijack your session:

• Some client-side code (Java applet, Silverlight application, etc.) that is injected into a page delivered over HTTP that can make requests to a site that uses HTTPS. This may require bypassing (or finding loopholes in) same-origin security policies.
• A sniffer on the network that the victim is on, to record encrypted data generated by the client-side code. The sniffer and the client-side code need to communicate with each other for the attack to work.

The easiest way to carry out this attack is over a public Wi-Fi network; however, attackers on other types of networks including wired networks can also do this. The main requirement is that the attacker and the victim need to be on the same LAN. The attacker will need to conduct a man-in-the-middle attack to inject malicious code into a HTTP page that can make requests to the targeted HTTPS site. This attack only works if the HTTPS connection is established using SSL 3.0 or TLS 1.0 and a block cipher (e.g. 3DES or AES) in CBC mode is chosen. Unfortunately, most HTTPS sites currently support only SSL 3.0 and TLS 1.0, and prefer using block ciphers in CBC mode.

Some technical details

The main techniques used by the attack are described below. Feel free to skip this section if you don’t care about the technical details.

• Some types of client-side code (Java applets, Silverlight applications, etc.) have the ability to send partial HTTPS requests. They keep the SSL/TLS connection open, and send data as it becomes available. With SSL 3.0 / TLS 1.0, each time a new block of data is sent, a new random initialization vector is not generated. The data is simply appended to the previous stream. An attacker who sees the last ciphertext block and can control the next plaintext block can gain complete control over the next ciphertext block (this is a consequence of how CBC mode works).
• Since HTTP headers preceding cookie headers are predictable and can be obtained by an attacker who sniffs a single HTTP request from the victim, and since the attacker can control the URL of requests, he can control exactly where the cookie header’s bytes end up in relation to ciphertext block boundaries.
• If an attacker knows the previous block of ciphertext and can completely control the next block of plaintext, he can determine whether a previously seen block of ciphertext corresponded to a given block of plaintext. Let’s assume that the attacker wants to determine whether the plaintext, P_i, for a previously seen block of ciphertext, C_i was x. Now, the attacker knows C_j-1, the last block of ciphertext, and wants to set P_j, the next block of plaintext to be encrypted, to a value that helps him determine whether P_i was equal to x. If he sets P_j = C_j-1 ⊕ C_i-1P_i ⊕ x (note that C_j-1 and C_i-1 are sniffed from the network, and x is the attacker’s guess), and P_i was indeed equal to x, then C_j = E(C_j-1 ⊕ P_j) = E(C_j-1 ⊕ C_j-1 ⊕ C_i-1 ⊕ x) = E(C_i-1 ⊕ x) = E(C_i-1 ⊕ P_i) = C_i. Therefore, if the attacker’s guess is correct, then the next block of ciphertext, C_j, will equal the previously seen block of ciphertext, C_i.

Given the above details, let’s say that an attacker makes a request to /AAAAAAAAAA, and knows that it will result in a block of ciphertext containing part of the cookie header: “ookie: x” (this is realistic if a 3DES cipher suite is used). Now, the attacker can make all 256 possible guesses for x, and can determine the first byte of the cookie header. Next, the attacker can make a new HTTP request to /AAAAAAAAA (one less A, which shifts the cookie header one position to the left such that the ciphertext block is now “okie: xy”) and can guess y. The attacker can continue in this manner until he guesses all bytes of the cookie. In reality, there are a lot less than 256 possibilities for each byte of the cookie header, and so the attack requires less work. There are also several details required for the attack to work that are omitted here.

Why the problem can’t be fixed quickly

TLS 1.1, which fixes this issue, was defined in April 2006. As you may have guessed, this problem was known before April 2006. However, it was considered mostly a theoretical issue until Thai Duong and Juliano Rizzo showed how it can be used to decrypt cookies sent over HTTPS. Even though this issue has been known for a while, it is probably not going to be fixed anytime soon because most websites do not support TLS 1.1 or TLS 1.2. According to Opera, only about 0.25 percent of web servers support TLS 1.1, and only 0.02 percent of web servers support TLS 1.2. There are workarounds that some browser vendors are currently implementing and testing; however, this problem is not going to be completely fixed until most web servers start supporting TLS 1.1 or TLS 1.2.

Risks

Should you be worried? Probably not. This does not significantly increase the risk of connecting to untrusted networks. There are easier attacks that can be used to steal your cookies (or your username and password) for many websites, or even install arbitrary software on your computer if you connect to untrusted networks. Some examples are:

• Many websites do not set the ‘secure’ flag on their session cookies, which means that a tool like sslstrip can be used by an attacker on your network to get your cookie.
• Many websites provide login forms over HTTP (even though your password may actually submitted over HTTPS), and attackers on your network can modify the login pages to get your username and password.
• Many users ignore certificate warnings provided by browsers, or may not even notice that a tool like sslstrip is being used and that they are not actually accessing a site over HTTPS before entering their credentials.
• Tools such as Evilgrade can be used to install arbitrary software on your computer by leveraging software that has insecure automatic update mechanisms.

Protecting yourself

If you want to protect yourself against BEAST-like attacks, you can take several steps:

• Delete all your cookies before you connect to an untrusted network.
• Limit the amount of time you spend authenticated to HTTPS sites on untrusted networks, and remember to log out as soon as you are done.
• Until your browser vendor releases a fix, disable all cipher suites that use block ciphers in your browser (leave only cipher suites with RC4 enabled).

Note that the last workaround may cause you to be unable to access many websites. Of course, when browser vendors release security updates, install them immediately.

Cigital has always favored delivery of holistic assessments (those assessments that consider architecture/design as well as code, leveraging static & dynamic tools, and using both tool-assisted and manual techniques). During my tenure as Director of Security, we found the following distribution of findings:

• Static tool: 20%


• Dynamic tool: 5% (*1)


• Manual SCR: 15%


• Architecture Risk Analysis: 60%

Those numbers may surprise and demand some explanation. For instance, though Microsoft at one point reported 50% of their findings were “flaws”, Cigital was finding 60% (on average, with as much as 70%) of its issues as ‘flaws’. If anything, our classification of ‘flaw’ was less inclusive though, and at the time I looked into this, I concluded that our larger percentage could be due to technique, focus on architecture in methodology (LoE), or level of customer familiarity/maturity with the tech-stack they developed on.

Likewise, our process started with a “whitebox” triage then shifted to exploratory testing (aka: dynamic) rather than the other way around (which I find common elsewhere) and that order-of-operations probably moved balance from dynamic to static. So, in practice we “found” vulnerabilities statically and then supported those findings with dynamic tests to validate their exploit-ability and impact (though perhaps dissatisfying, static gets the point in our data set here).

Slicing by technique
Our experience made it pretty evident to us: skip assessment techniques and expect to miss some critical vulnerabilities.

Slicing by vulnerability type
As I’ve mentioned in previous blog entries and mailing list posts, security managers need the capability to assess and report on their ability to find particular classes of security vulnerability as well. WASC’s Web Application Security Statistics provide a starting point for both classes of attacks you might expect to find and their probability. There’s no shortage of vulnerability taxonomy work out there, so we’ll leave this topic at this superficial level.

Optimizing a Security Practice
Whether you know what percentage of your reported findings come from each assessment technique, or you’re just starting to build your assessment practice, optimization (measurement, analysis, iterative improvement) should be a goal. To do this we combine our knowledge of the techniques we’re employing and the enumeration of vulnerabilities we want to find and report.

Confined to the context of static analysis, I described the following loop in a previous blog post to do just that:

• Use a representative sample of your applications to observe a static analysis tool’s performance, not a contrived test suite.
• Consider the tool’s findings relative to pen-testing findings on the same app.
• Did new and interesting findings result?
• Did pen-testing findings found by the static tool provide adequate root-cause analysis and remediation advice to fix problems earlier?
• How long did it take to on-board an app? How will this scale to your portfolio of apps?
• How long did it take to triage the results? How will this scale?

The purpose of this trial experimentation was to provide some basic comparison between assessment techniques (though you could easily come up with a more thorough approach). Of course, you’d want to consider findings from the perspective of all techniques, not just static tools.

Internally, we had our assessment lab managers write up a qualitative comparison of assessment techniques we employ, comparing how well each is “suited” to finding vulnerabilities against common vulnerability taxonomies (*2) and came up with the following:

• ARA: 14%


• Static tool: 12%


• Manual SCR: 21%


• Manual Pen: 21%


• Dynamic Tool: 12%


• Sec Testing: 20%

At this point, the reader may be asking why my experience differs from our lab managers’. Specifically, for instance, “why are dynamic techniques over-estimated here as compared to to historical findings rates?” Two reasons (at least) come to mind:

1. The balance of techniques that constitute our assessments are “moving back” towards dynamic as software-under-test and market forces evolve


2. There’s a difference between how suited a technique is to discover something and how we choose to find it (I guarantee that in some circumstances this gap represent ‘optimization opportunities’, while in others it represents explicit technology or methodological choices).
   

The Caveat You’ve Come to Expect From Me
Whether your considering a vendor’s assessment service, an assessment standard, or your own internal assessment practice, it’s key to be able to detect biases. When others supply you assessment data you’ll likely uncover biases (though it may not be apparent to those reporting initially). Be careful, I doubt you’ll be able to compile, combine, or relate numbers you receive from varying sources for this reason.

If you’re compiling assessment data for the purpose of motivating practice improvement, you may want to either 1) extract universal similarities from the data you get and present those lessons learned to management, or 2) just list the others’ experience separately. This can provide those to whom you’re presenting a compelling view to your perspective on differing assessment approaches.

In fact, many factors will muck up assessment comparison. For instance, when people report a percentage (like I did above), did they mean finding instances, findings, by type, or something else (*3)? What was their methodology, level-of-effort, and how did they conduct assessment. For instance, risk-based assessments produce a very different distribution (with wider variation) than time-boxed techniques, which more frequently drive to an explicit (or worse, implicit) checklist.

One thing universally seems true though: the more assessment techniques a software security practice makes use of, the more types of vulnerabilities it can expect to report.

-jOHN

(*1) – Clarification: our use of dynamic tools is human-assisted (both in terms of crawl and in follow-up with manual effort or leveraging internally developed tools).

(*2) – WASC, OWASP Top 10, CWE Top 25

(*3) – My data is comprised of reported vulnerabilities. Answering questions about what risk-level I reported requires some abstraction as clients differ in their requirements there. Likewise, behavior regarding instances vs. buckets of ‘related vulns’ requires addl. explanation.

The implication of these techniques is that bolt-on security controls like WAFs can be evaded using obfuscation techniques. Similarly, simple-minded input validation can also be bypassed.

You can get a copy of the book at Amazon. I highly recommend this book for software security practioners and developers.

Remember to click on the link to tell the publisher that I want the book on my Kindle.

The implication of this book (IMO) is that simple input validation is a less compelling mitigation as the sole control for malicious input. So, WAFs and regex-based input validation frameworks are going to need to either become much smarter (unlikely) or adding output encoding is going to be required.

Dan Cornell wrote a blog post on static analysis coverage. He observed that while the static tool he used completed its scan, it didn’t find things it should. He implicitly referred to this as coverage.

I’ve often put things as follows:

Your tool can’t find what it can’t see and it can’t see what it doesn’t parse.

This got said often years back because at the time it was quite difficult to integrate tools effectively into a builds so that they would in fact ‘see’ everything and successfully parse it (*1). We referred to this concept of how well we, as operators, had integrated as providing visibility into the code. These days, market leading tools don’t have nearly the problem they used to with finding the code they need to parse… or parsing that code successfully (*2).

These days, the frameworks developers use for persistence, object remoting, application navigation, and view display challenge visibility most. Tools see the code for your controller and parse it successfully but they don’t necessarily understand what it represents. This happens in almost every application I scan, even the painfully simple ones.

We couldn’t have our Consultants going out into the world armed with tools bearing only spotty visibility into applications so I wrote a utility identifyEntryPoints to find entry points on its own. We don’t want Consultants guessing at what their static analysis tool missed or popping the hood open to try to discern what it found and not doing what they’re paid to do: assess the application, so I wrote another utility entryPointCompare to compare what the tool found vs. what I was able to find programmatically. See actual output below (I’ve replaced the static tool’s name with ‘TOOL XXX’):

Sanguine:demo jsteven$ /Users/jsteven/code/demo/working/engine/util/entrypointCompare.py
2011-02-06 09:07:08,158 INFO     bois_1.0        bin.execPythonRobot STARTING: /Users/jsteven/code/demo/working/engine/util/entrypointCompare.py
Factory 31, [TOOL XXX] 0
 TOOL XXX missed set(['Index', 'csrchat', 'accountsList', 'Transfer', 'editProfileFormBean', 'newAccountFormBean', 'fileList', 'Auth', 'newClientFormBean', 'qaFormBean', 'UserProfile', 'SetupProfileAction', 'uploadFile', 'VerifyAnswerAction', 'BeanEditProfileAction', 'existingClientFormBean', 'GetDocuments', 'ChatPoll', 'announcements', 'modifyDocument', 'BeanCreateAccountAction', 'ChangePasswordAction', 'VerifyClientAction', 'ChatSend', 'Help', 'fileUploader', 'BackOfficeAdmin', 'passwordFormBean', 'BeanCreateClientAction', 'action', 'Accounting'])

Here, I ran the test on Cigital’s own internal “Bank of Insecurities”, which is a simple Struts1 application. The utility’s output represents all the actions and forms that account for web-based input in their own respective ways. Sure looks like some pretty important stuff was missed in there huh?

The next step was to write a third program registerEntryPoints which would write rules for the missing entry/exit points as appropriate so that the beefy commercial static analysis tool could do what it does best. These utilities represent just some of the internal workings of Cigital’s ESP platform.

Yes, but my tool finds stuff all the time
Would a tool not find any findings without detecting entry points? Well, it would likely find potential vulnerability but only using more local syntactic analysis. Remember, “you can’t find what you can’t see…”

I’m sensing a theme in your posts jOHN
OK, so, that’s two blog posts (Mr. Cornell and my own) complaining about what I refer to visibility. Now what?

Spend time cataloging your chosen tool’s performance
Cigital experience indicates that as few as 10-13 reasonably chosen applications can serve as a representative sample for as many as 300 applications on the Java platform. When you pilot those 10-13 applications, conduct the following steps to avoid the visibility failures seen above in the portfolio as a whole:

1. On-board the application using whatever tool interface gives the most feedback about missing artifacts (*3)
2. Explore scan logs for identified entry points
3. Manually explore the application’s deployment descriptors and critical configuration files
4. Document controller logic as framework default or developer extended
5. Identify key entities within the DAO/persistence framework

When you’ve done that for 10-13 apps, our representative sample, stop and take stock of what you’ve collected. You’ll have created a very good list of items for which you’ll want to create custom rules. Different tools call the kind of rules you’ll create different things, but suffice it to say you’ll be writing rules to tell the tool that it’s:

• Entry: Taking input from untrusted web sources
• Entry: Taking input from untrusted partner applications
• Exit: Placing data in a untrusted view (browser, service repsonse, etc.)
• Exit: Conducting CRUD operations on entity data

The above list is by no means exhaustive. Indeed, you’ll want to look at data originating from the persistence tier and headed to the web (not listed above) but we often consider this as a second phase of customization. You’ll also want to begin considering data entry/exit from 2nd and 3rd party components within the applications being tests. Again, another good subsequent customization phase.

Gosh, this sounds expensive
Yes; ‘well. Compared to simply running a static analysis tool using its IDE-based GUI, triaging results, and calling it quits this is darned expensive. However, it dramatically raises your visibility into an application’s vulnerability (you should also expect false-positive reduction).

Compare this to the potential difference in quality and depth of a manual, tool-assisted penetration test and an automated vulnerability scan. An engineer running AppScan costs much less than an expert penetration test and organizations have come to not only expect different results, but see these as two very different services. I predict that we’ll see a similar distinction between two services (automated and more expert-driven) in the static space in a few years.

Leveraging Manual Efforts Across Assurance Activities
If your organization doesn’t conduct manual penetration testing and is unwilling to pay your engineers to properly on-board applications into its static analysis tool, then neither static nor dynamic analysis will benefit from high visibility into applications under test. The cost will look “high” and “extra” as well.

Conducting the kind of analysis described by this entry (targeting entry and exit points) can inform both static and dynamic analysis alike. Conduct this exercise for applications built with representative technology stacks. On the static front, write custom rules based on your work. On the dynamic side, furnish this work to your dynamic testers (even if they are external vendors) so that their exploration benefits from it.

(*1) – Coverity’s Prevent was always a notable exception and able to integrate with even challenging build environments without much pain.
(*2) – Limitations still exist with complex (distributed) build systems, code-generation schemes, and even the more mundane JSP compilation process.
(*3) – Ask if you need help here, last time I published tool details, vendors got cranky.

If selecting and adopting a tool is so hard, even for experts, why should I bother?

Good question. The article was not written as indictment or defense of static analysis–it was a cautionary tale whose moral: “organizations can get wrapped around the wrong axle when selecting and adopting a tool” I believe in to the utmost.

If it is worth adopting static analysis, as I believe it is, then the question becomes: How do I avoid doing it poorly? I’ll start by revisiting some suggestions made in the article’s conclusion and continue.

Seek Experience
Expert consulting can dramatically improve the speed and effectiveness of an organization’s definition of and scaling static analysis and SCR practices. But, you don’t need experts’ help to successfully pick and adopt a tool.

Leverage communities you’re already amongst to absorb their experience. Reach out to:

1. Your local OWASP chapter
2. Organizations within your vertical
3. Similarly sized/structured organizations within your geography

Within the communities, others have likely already selected and adopted a tool. Though some will not share their selection or adoption experiences openly, others will. And, I’ve found that human nature can’t resist sharing at least _some_ aspects of a good war story.

War stories hint at underlying data more valuable than the particular selected tool: the aspects of adoption that posed the most challenge or required the most effort. Having your troops pointed in the right direction when fighting breaks out will benefit you more than making the better choice between equipping troops with a carbine or a rifle.

During last year’s BSIMM conference in Annapolis, MD, I saw tremendous inter-organizational knowledge sharing on the static analysis front. Not only was I impressed, but as usual, I learned things. War stories? Those improved with volume imbibed.

Eschew Deep Scientific Comparison for Trial Experience
Unless you graduated with an advanced degree focused on static analysis, or spent the last ten years building/analyzing it, I don’t feel that a comparative study will be satisfying. Many have asked me, “With you experience–come-on–tell me… which is better, A or B?” Individuals asking me almost always hear a variant of the following candor:

Comparing the analysis engines of the two market leaders is a lot like wondering about the Audi S4 vs. the BMW M3. Both take a comparable approach to their flag-ship consumer performance car: both switch between ~3.xL six-cylinder blown engines or ~4.xL naturally-aspirated V8s. In the end, both gobble premium fuel and get to 62mph in 4.x seconds. Both are fine pieces of engineering and each has its own religious following.

Like the previous advice, knowing how to drive it and where the car’s limits are likely provides lower track times than the selection of one car over the other for the average-to-enthusiast driver. It simply doesn’t matter whether the trappings of boost pressure produced by the S4′s supercharger beat out the direct-injection, independent throttles of the M’s high-revving V8 (*1)

I’ve digressed a bit too far, haven’t I? Don’t do the same with your static tool

• Use a representative sample of your applications to observe a static analysis tool’s performance, not a contrived test suite.
• Consider the tool’s findings relative to pen-testing findings on the same app.
• Did new and interesting findings result?
• Did pen-testing findings found by the static tool provide adequate root-cause analysis and remediation advice to fix problems earlier?
• How long did it take to on-board an app? How will this scale to your portfolio of apps?
• How long did it take to triage the results? How will this scale?

As your trial encounters problems, make explicit note of them. As you move from 3-10 apps to the portfolio as a whole, can your organization withstand stresses at scale? If not, you may want to switch tools or approaches.

I’ve seen security managers I respect get incredibly far without any consulting help just by taking an iterative approach and asking questions like those listed above.

Worry about what you can control
You control your organization’s staff size, skill set, scanning policy, and infrastructure. You do not control the architecture, implementation, or bugs associated with the static analysis tool you buy and deploy.

So, as you talk to others about their experiences selecting and adopting a tool… as you conduct trials with a selected tool… and as you plan a larger roll-out at scale of your static practices, think about strengths and weaknesses in a chosen tool not in terms of how its competition might perform relatively but in terms of whether or not they compliment or expose your organization’s staff-based, skill, policy, and infrastructure weaknesses (or play to its strengths in those same areas).

And finally, don’t be afraid to ask for help.
-jOHN

(*1) – In the interest of full-disclosure, I cast my lot with the 2011 M3 over its Audi competition despite BMW’s ‘heretic’ departure from their I6 engine architecture heritage in the 3-series. I can make no such clear claim to my allegiance on the static front.

I can help James and community out by giving some of that guidance myself. I’ll try to do so in a tool-independent way. This topic deserves more than a “blog entry” format (admitting Cigital’s propensity for longer entries ) so if people want more information on the topic, they can ping me.

In essence, you want to know the following about static analysis tools:

1. Who is going to run it (how will owners shepherd it)?
2. How much effort is it going to take to run (what budget will I need to support it)?
3. When do I run it and how often (Where does it fit within my SDL?)
4. What it’s going to find (How will the results enter/impact my organization)?
5. What won’t it find (How does it play with the rest of the assurance ecosystem)?

[Market Adoption and Making Your Choice]
Gartner seems underwhelmed by Microsoft’s ability to effect the commercial static analysis market. Cigital has always believed these technologies would make great features of a compiler/IDE and so we’re also disappointed. They’ve got great technologies but don’t exploit them commercially. I still like their freely available FXCop but this opinion isn’t popular. Perhaps that’s because I see a lot of value to FXCop as a unit testing framework that enables security test and other people are comparing it to glossy commercial tools like Fortify and Ounce run by security analysts in a very hands-off fashion. Who’s considering the tool really does make a huge difference in how it will fare.

From my perspective, the most successful SA tool vendors have made answering the “who should adopt the tool” question more difficult as they’ve waffled on licensing models. By the seat? By the core? By the KLoC? I’ve gotten complaints from prospective tool buyers indicating that a leading tool vendor’s price was 10X another vendor’s. Before you ask “Which one?”: I’ve heard specific examples in opposite directions!

Certain tools will look better to different potential buyers. Experience has shown positive response to Coverity’s C/C++ results from developers. Application security groups like Fortify and Ounce’s products, and several years ago, I saw QA Managers in two very different firms go ga-ga over Klockwork’s tool. I attribute people’s impressions not to a product vendor focusing on quality or security but to how closely the tool vendor’s conception of how roles interface with their tools throughout the vulnerability management and software development life cycles matches the organization’s own structure. Up until a few years ago I don’t believe SA vendors had fully conceived the “developer”, “analyst”, “security manager” roles in their products. Ounce was the one exception: they’ve always staunchly believed themselves to be principally a Security Analyst’s tool. Some vendors attempted to unify roles into use of a single interface while others attempted to split the product up into different configurations for each. This created confusion and friction between some adopting organizations and the tool vendor’s license model.

In the end you want SA tools to affect as many developers as possible and to get low-latency scan results as early in software’s life cycle as possible. However, having successfully deployed tools a variety of ways, I’m convinced central deployment is the way to go. Application Security should run the tool. This will help manage rule/configuration update, as well as central measurement collection, risk management, and issue tracking. How do we get low-latency and early-lifecycle centrally? Treat the tool as a service. More on that later.

Notable exceptions to the central deployment model include business units possessing one or more of the following characteristics:

• Acquisitions of previously successful, culturally-independent companies
• High-quality product development teams
• Fundamentally different SDL, development platform, language, and toolkits

For in-unit deployment one can “run locally” but must “report centrally” supporting security goals involving measurement and risk management. And, let’s be clear: if a business unit has a wicked-good continuous integration/build-management group, they-by all means-should be tapped to integrate with the SA service. Cigital and its clients have had success in integrating both the front (code submission) and back-end (bug/issue tracking) with Fortify and a variety of open source build/CI/bug-tracking products. This maintains a central model, but reduces latency dramatically.

[How Much Effort]
I’m sorely disappointed in the honesty of tool vendor sales-folk regarding level of effort. One of my first analogies for SA tools is this: Did your company buy Mercury products for load and performance testing? Think of your SA tool like Mercury–you’ll need at least as much money to deploy and implement the tool organization-wide as the licenses cost you. Sorry.

Organizations with more than 2-300 developers should plan on the following:

• 3-4 man weeks to do initial tuning and pilot implementation
• 4-16 man hours / large application to integrate with the static analysis tool and assure appropriate configuration
• 4-8 man hours / 50-100KLoC (language-/complexity-dependent) to triage results on a new-application scan
• 1 person / year as tool shepherd (app integration, custom-rule creation, rule-pack maintenance and release, support)

Organizations get into trouble when their SA tool enjoys broad acceptance by the organization before they effectively manage the submission and results triage/documentation process. My data indicates code submission can burn 8-24mhrs / application if security groups don’t ‘remember’ (or automate) the submission and scan setup process. Security groups doing scans centrally and hand-writing code review results can waste another 16hrs documenting their findings / application.

…think about it: if your organization allots a week / tool-assisted code review and wastes 24 hours getting the tool’s first result set and 16hrs documenting findings… that only left 10 hrs to actually review the code! Since I believe that 4-8 hrs / 50-100KLoC is necessary to triage, you can imagine how deep I think such code reviews are At Cigital, we’ve got dramatic benefits to depth by decreasing cost of submission and reporting. This is key to scaling SA efforts. Those adopting tools will feel overwhelmed if they don’t plan to solve submission and reporting problems before wider roll-out.

[How Often]
As often as possible. If you’ve got a Continuous Integration initiative (CI), engage those folk. If you deploy your tool purely centrally, plan to scan each high-risk application at least once / release. No, to my knowledge, there isn’t a good answer to the “can these tools do incremental (partial) scans”. You should always keep old results around though, as they’ll (along with other measures) help you understand whether more or less vulnerabilities are being caught during each scan. Remember though, every time you add a rule to the scan, your vulnerability-finding bar just moved and your metrics might be thrown off.

Ounce, Fortify, and Klocwork seem pretty good about determining whether or not they’ve found a particular issue in a previous scan or not. This includes situations where code blocks move around a bit. This means that running regression scans to determine whether or not an issue has been fixed is viable. I can’t comment on Coverity’s capabilities here.

[What's it going to find]
I’m impressed that Fortify has published its vulnerability taxonomy and a bit shocked that others haven’t produced as public a resource. I’m excited too that vendors have agreed to comply with CWE labels when they report findings. I will say what I always do on this topic though: test the tool on your own code though. You simply don’t know what it will find until you throw all your idiosyncratic build, language, toolkit, and platform nonsense at the particular tool you’ve selected.

[What won't it find]
I’m continually shocked as to how much one can find by running tool Y after tool X on the same piece of code (replace X and Y as you see fit). As I’ve written time-and-time-again, the corner cases and exceptions missed by each engine are baffling. Let me put things in perspective for you: three of my clients have reported that their SA tool deployment find 17, 33, and 50% of the total number of issues found by assessment efforts respectively. If these percentages hold for your organization, inside a coding bug realm then your job is at best 50% done having run the tool and triaged its results.

Considering that Microsoft and Cigital believe that “bugs” account for only 50% of the vulnerability space, that leaves you unaware of 75% of your application’s vulnerability at the end of triage. As I always say, “use the tool to facilitate code review and increase a reviewer’s understanding of the code–not as a code reviewer that finds bugs automatically.”

Good luck out there, I’d love to hear about your particular experiences.
-jOHN

In my mind, these two separate issues exist here, intertwined.

Almost every logging framework has an in-place configuration option for increasing/decreasing log-level; most allow you to reconfigure on a running system without restart. The point here being that an application developer, with use of such a package, get to 1) keep a single code base, 2) turn a high level of logging off by default in a production environment, and 3) ratchet up log-level in circumstances requiring it (incident response or in-place debugging).

Often, people mix the above guidance with “don’t roll debug code out to production.” Code Analysis tools, for instance, give this advice. Two things principally concern me here. First, you don’t want a bunch of “main” or other “debug-only” entry points to the code. Attackers capable of compromising host systems could shoe-horn their way into an application this way. The situation becomes worse when said debug interfaces are web-based. Second, especially for thick clients or code running in less trusted environments, debug code tends to ‘express’ things about the design (or leak critical data) in an undisciplined manner. Removing such code from deployments to untrusted zones CAN be advisable, but one must balance complete removal with the need to ‘turn on’ nascent debug code in times of need as described by the previous paragraph. Only protection of the most sensitive information, and most invasive code qualifies for the extreme guidance “remove before deploying” that this paragraph describes.

The best thing one can do is give the developer guidance as to what types of test/debug code might qualify for wholesale removal. I’ll start you a list:

1. Client ‘harness’ code allowed to connect to server interfaces w/o
   authentication
2. Client code allowed to edit/manipulate local data repositories/stores
   directly
3. Command ‘harnesses’, allowed to (re)set or manipulate server state
4. Client code that dumps secret keys, tokens, or stored-but-otherwise-protected credentials
5. Backdoors
6. Code allowing for ‘insecure failure’ or back off, say to an older protocol
7. ‘Re-implementation’ outside a sandbox (native equivalents of a managed binary)

Just a small tidbit, away from the glib towards concrete guidance.

-jOHN

[tags]logging,debugging[/tags]

We’re pleased to have a guest blogger for this Justice League entry. Michael Cohen is a Senior Security Consultant at Cigital where he is responsible for leading, assessing, architecting and implementing secure software for Fortune 500 companies. Michael also works with Cigital teams on enterprise-wide security solutions intended to improve an organization’s security posture and help them meet audit and regulatory requirements. Michael just gave a talk to the Washington, DC IASA chapter that was well received, and is now the subject of this entry:

When I hear software architects talk about the architecture they’ve crafted to depict the various structures and behaviors of a system, they point out interesting techniques they’ve applied that best convey how the system should work and be put together. An architect’s enthusiasm for the little details reveals a great sense of accomplishment and creativity, but most of all good architecture conveys sorely needed information in order to help those who develop, test and maintain the system. Sharing the tips and tricks gathered from the field is what helps our community move forward to building better software. A classic example of this is the Design Patterns book written by the Gang of Four.

Not too long ago I was also sharing my ideas on architecture and security at a local IASA chapter here in Washington, DC. The group was a crowd of like-minded architects who work for large Fortune 500 companies, government agencies, and promising local startups. My topic was pragmatic secure architecture. The basic idea was to look at some real ways to incorporate security into architecture using Cigital’s risk management and threat modeling practices. You can find the slides for my presentation here.

For the uninitiated, threat modeling is a way of depicting where threats (think malicious people, attackers, and so on) can touch a software architecture and how they may be able to exploit critical assets using various attack patterns. Threat modeling is valuable for determining an architecture’s security posture. In addition, identifying risks in user requirements and business goals, and tying those risks to a threat model results in a map of how design flaws impact the system, its users and the overall business. Threat models coupled with identified high-level risks are a great way to get other stakeholders involved with security decisions. And mind you, these are stakeholders who would otherwise be unable to contribute and supply critical feedback.

The attendees at the chapter meeting were glad they attended the presentation and heard something worthwhile that they could use in their daily architectural activities. Many people brought up interesting points about how to best protect critical assets, what a real risk to a system is, and what is considered a good enough mitigation. What I found particularly interesting was how threat modeling provided a way for everyone to contribute interesting ideas about where security vulnerabilities may exist and how they could be mitigated (which, if you think about it, is the entire point of threat models).

I encourage any architect to share their ideas on building a better architecture with their peers, whether it be at a local organization or at a conference. As a senior security consultant at Cigital, I like to take shared ideas and mold them into my own unique way of thinking about the world. The best results come when I can apply new ideas to my daily activities as I help our customers assess and create more secure software.

… On an entirely separate note, McGraw still owes me money. I’m watching you, Gary.
[tags]software architecture,software quality,software security,touchpoints[/tags]

For most programs, Static-SQL is sufficient. Only a small subset of features such as user driven filtering interfaces need Dynamic-SQL’s flexibility. Unfortunately, the common database access protocols, ODBC and JDBC, are based on Dynamic-SQL and there’s no provision in the interface for static queries. It’s into this void that SQL stored procedures stepped in.

But SQL stored procedures aren’t limited to Static-SQL. It’s possible to use Dynamic-SQL through the PREPARE, EXECUTE and EXECUTE IMMEDIATE statements (these are the ANSI SQL-92 statement names, your RDBMS have different names). In fact, ODBC and JDBC are just wrappers that ship the SQL text from your program to the database server and make calls to the RDBMS through the runtime support for these Dynamic-SQL statements (actually). So, stored procedures that execute Dynamic-SQL statements are equally vulnerable to SQL injection. It doesn’t matter if it’s Java code concatenating strings and passing them to JDBC or if the strings are concatenated in PL/SQL. Both are equally vulnerable.

By now you’re probably wondering why I’m splitting this hair. I’m splitting it because I want to ensure that we protect ourselves from the programmer we’ve not yet met. It’s the programmer that goes and dutifully reads our coding guideline that says “Use SQL stored procedures to prevent SQL Injection” and now has to extend our application with the feature that introduces Dynamic-SQL. But, because we haven’t properly articulated the guidance, s/he creates an SQL injection vulnerability.

Implementing your queries using Static-SQL will go a long way to protect your application from SQL Injection. It’s also possible to simulate Static-SQL using ODBC/JDBC type interfaces with the religious use of SQL parameter markers. This leaves the use of Dynamic-SQL only for truly dynamic queries like the dynamic user-defined filter I mentioned above. There’s a pattern for securely implementing such a filter that’s floating around in my head. I need to write that down, but you’ll have to wait until next month. Sorry.

[tags]sql,defects,flaws[/tags]