Ben Tomhave wrote a decent post musing “Which came first: The Software or The Security?”.

In particular, Ben asks whether the response an organization has to its security problems should possess a time component. “Yes”, he answers his own question emphatically. I agree, and for a few reasons worth expounding on.

…the only difference between today and yesterday
If you’re effective at assessment, you’ll often leave an organization’s apps in shambles. Somewhat older data I have indicate an expectation for 10 critical vulnerabilities / 100KLoC, with 30-40 high-risk vulnerabilities following suit(*1). Organizations holding their broken app at arms length panic. “We’ve got to pull it from production!”, they exclaim. Others take a different approach, “We’ll stop everything until those vulnerabilities are fixed–no more features.”

This behavior is irrational and needs to be quelled. The only difference between “today” and “yesterday” in this situation is that the organization knows how screwed they are (*2).

Ben indicates that visible risks should be prioritized based on a risk management framework. Ostensibly, we’re very much in agreement on that one. There may be vulnerabilities that deserve such “stop the press” kind of treatment, but there is often a normalization process that must occur before that set is culled from an assessment (*3).

After that triage, the rest (of the vulnerabilities), and I’ve said this before, should enter the change-management/bug-tracking process like everything else: features, customer complaints, and so forth. Dealing with security vulnerabilities in-band both sobers security analysts’ dreams of the all-important ’sploit and raises the respectability of security requirements within the normal development process by treating them as first-class citizens of a release process.

Depending on whether an organization has rapid-turn around (as happens when development engages in weekly sprints, organized through SCRUM) or more monolithic develop cycles, these change requests can be weighed from the perspective of whether or not they should be:

1. Released out-of-band as a patch
2. Included in the next planned release, potentially:
   
   • At the expense of other functions/features
   • At increased expense,

   or:
3. Deferred until a future release

Sometimes, the ‘correct’ answer is to conduct a usability study, or revisit the SLA promised by a system/service, and get feedback as to what form the fix will take before any of the above can happen.

In summary, time continues to elapse at a constant rate. In this case visibility is increasing. Visibility should allow for a better risk management decision, not a knee-jerk reaction.

Moreover, it’s important to discern the difference between 1) the organization’s increased visibility into vulnerabilities, not 2) an increased discoverability of the vulnerability itself. Yes, if the assessor used an automated pen-testing tool to find the vuln., it is probably pretty discoverable, but the organization’s knowledge of the vuln. does not increase it’s discoverability(*4).

Onto the next element of time…

0 to 60mph in?
Ben also rightly indicates (paraphrasing) knowing there’s a gap between you and the security posture you’d like to have, you’re not going to get there instantaneously. Very true. This, in my mind, is where organizations could improve the most. Most organizations I talk to have dashboards showing quantities like

• Percentage of applications in compliance with assessment policies;
• Number of outstanding critical + high vulnerabilities;
• Time to remediate critical + high vulnerabilities;
• <Metric du jour>

I’d like to see organizations begin managing to security goals like:

…Next year, we expect no application providing access to [asset X] to be vulnerable to discoverable by an external Threat, be he/she authorized or not, accessing our systems only through the web;

…in three years, we expect no vulnerability that would provide access to [asset X] with any Internet-facing software, regardless of immediate discoverability or exploitability (by an external threat);

…in that same three-year time frame, we also expect no Internet-facing apps. to fall prey to [common web attacks, listed in the SANS Top 25(*5) (aka: CSRF et. al.)] that would result in impersonation and therefore allow for inappropriate access to [asset X] on behalf of an unwitting user.

I’m attempting to extend Ben’s call for a time-based (I’ll call it iterative) approach to responding to security gaps. The previous paragraph proxies for what a more advanced risk management framework would incorporate in the form of probability (discoverability x exploitability x [other factors]). Remember, however, that the [asset X] clauses of these statements proxy for different but important factors in your risk model: intrinsic value of the information asset.

So, whether you’re using a formal risk management framework, or doing it more informally, you can respond iteratively and get time back on your side: having a story about how you’re measurably reducing organizational risk in a meaningful fashion.

Compare this with the alternatives:

1. A straight linear approach - fixing 80% of XSS bugs within an application has what effect on the overall security posture?
2. The bug-of-the-month approach - ‘fixing’ 100% of the CSRF problems within an app, but leaving all the XSS has what effect on the overall security posture?
3. Stopping the presses - fixing 100% of the bugs, and not allowing business functionality to evolve in the balance does what to your business opportunity/revenue stream?

Finally,

Within a single vulnerability’s remediation, I’ve got trade offs
Yes, within the scope of a single killer finding, organizations have to decide how to trade off the time it takes to field a mitigation and the effectiveness of that mitigation on the security posture (queue the “dynamic patching” vs. “fix it in the code and re-deploy” flame).

This, just like our previous two topics, is a matter of risk management. I implore organizations to consider what the capabilities and motivations of their opponent are in this case. If you’re considering protecting against a web-vulnerability delivered through a URL parameter, a WAF rule will prevent a vulnerability scan from finding the ‘fixed’ problem but is unlikely to thwart a skilled penetration tester from manipulating the order of the innocuous and malicious parameters. WAF rules, then, are likely to provide a rapid response to failed scans and simpleton attackers but not concerted attacks of this sort. This same sort of analysis can be done on code-based fixes. When assessments find input validation problems, the development team will often respond by doing a simple (arbitrary) length check, or by black-listing particular SQL characters.

When I was quoted in a Darkreading article as saying costs to fix were [so low], it was these spot-fixes to which I referred. These fixes don’t (often) fully remediate that particular vulnerability (even in that narrow locale) let alone improve the application’s overall security posture. But, I’d say they’re the most common response to an assessment finding. Like McGraw says regarding the BSIMM data, “we observed monkeys eat bananas; is that good? I don’t know.”

Having considered an attacker’s capabilities and motivation, and the value of what you’re protecting… an organization might in fact choose to engage in such cat-and-mouse play as the trivial fixes above imply, engineer a real fix, employ some combination of both, or off-load risk in an entirely different way.

Just like before, however, it’s unlikely the vulnerability was created yesterday, so the software has been vulnerable for some time. When risk analysis has occurred, and the full cost of a complete solution is resolved, complete with its own residual risk calculation, it’s very likely that a practical organization will tier its response, iterating in time.

Thanks for the post, Ben.
-jOHN

(*1) - Within a Java EE app found through tool-assisted code review with no preference for having been reviewed prior or not (be it prior code review or penetration test). Critical and High in these cases varied somewhat, but always having been seen by the client as exploitable and accepted for remediation by development.

(*2) - This is somewhat glib. More accurately, the organization has increased its visibility into what vulnerabilities its software possesses. It can not say these are the only vulnerabilities the software possesses, and, using purely code review, it can not always say with 100% confidence that detected vulnerabilities are exploitable; some statically determined findings are obviously exploitable whereas others will require dynamic verification.

(*3) - This normalization is even more important if the assessment was delivered by a 3rd party tool or assessor conducting their first run/engagement with an organization. Internal assessment teams often have it easier; they should know/conform to the business unit’s risk management measurement memes.

(*4) - Unless they have a really terrible vulnerability management process or a shady security vendor ;-)

(*5) - Indeed, I have suggested use of a Top N list. Here, I’m using it to indicate a priority implicit in industry opinion. Better to augment that clause with [the most common types of attacks observed in our production environment and the #1 emerging attack our security research team advises us on].

My last post spawned some questions, which I responded to in turn. Here was my response:

[Adapters]
Adapters for assessment results can take a few forms, but let’s address three specific scenarios that fan-in to an assessment results/presentation step and a few that fan-out.

[Fan in]
Fan in typically comes from three sources: 1) static tools, 2) testing tools, and 3) manual analysis.

Adapters that deal with fan-in have three challenges to surmount:
A)Technically trans-code results from #1, #2, or #3 into a single tool’s format for roll-up, or a tool-independent alternative

B) Normalize results between #1, #2, and #3 so that ‘apples’ and ‘oranges’ get reported rather than ‘apples’ and ‘cars’

C) Code results with organization-specific “whathaveyou”

[Challenge A - Trans-code]
The good news about the tools is that they nearly all export to some XML format you can manipulate. This tackles output from #1 and #2. Organizations that adopted any tool early have struggled with keeping up with format changes but have indicated to me that they’re willing to pay this small price for the ability to ‘plug’ that many results into their developers’ bug-tracking systems.

A bigger challenge (effort and therefore cost-wise) is getting manual results (#3) into either the format driven by the tool responsible for reporting or that independent format I described. Smart consulting firms have built ‘emselves workflow to manage this. In the ’90’s we at Cigital used to pine for the report-generating automation of our then-competitor @Stake and waffle about whether our client-base was too disparate and our assessments too varied in scope to support a similar trick. I see no reason not to, within an organization, code up something that helps-technically-integrate your manual review findings with those found by tools.

[Challenge B - Normalize]
Manual reviewers report to me their discomfort with the rigidity of systems like “The Seven Kingdoms” and its competitors. Mind you, I don’t dislike these taxonomies, but people get crotchety and might complain if they’re forced to cram their findings down into a SAST or DAST report. They perceive, it would seem, the vulnerabilities they find to be like snowflakes ;-)

An independent format can be successful as well. One can roll their own (for their organization), or lean heavily on the community and go with something like Mitre’s CWE. At Cigital, we’re involved with CWE and follow-on work but have also helped people organize their own. A critical discussion of this trade off is possible, but beyond the scope of this email. That leads us into…The cost of normalizing results between #1 and #2 eclipses the technical challenges of building the adapter.

[Challenge C - 'Whathaveyou']
I would say this though: if your organization has progressed to the point where it possesses security standards (prescriptive or otherwise), they should absolutely be referenced by appropriate findings in the report. This goes for both violations of the standards and the applicability of standards that would serve to prevent or mitigate a particular risk.

References to policy/standards can be augmented with best practices, if your organization makes that distinction. I’ve seen organizations link to internal/external resources for training and/or further information as well. If you’re going to try to transition from “the bug hunt” to “building security in”, one great way is to provide developers immediately-available information as they’re making the mistake.

[Fan out]
Now, you have to ‘fan out’ into support for bug-tracking systems. Most organizations’ security groups have at least as much battle to fight here as a security consultancy, actually. Why? Because the organization hasn’t mandated a single development toolkit. The good news here is that while there may be more fan-out than there was fan-in, bug-tracking tools were built to be supplied data. Writing this portion of the adapter–a conduit between what normalized findings you’ve compiled and the offending team’s bug tracking system is fairly (technically) straightforward.

[Industry Std. 'Schema']
Sean Barnum has done a flotilla of work on this topic with Bob Martin at Mitre. Though it can be cumbersome or uninteresting to practitioners, I think the work they’re doing is important because whether its admitted or not, work on audit, testing, and verification methodologies/standards must implicitly take a stand on defining the words you listed (finding, root cause, vuln., etc.). Where such efforts take a stand, one’s organization can find alignment with their own notions quite challenging. Where it’s implicit (or worse, ambiguously and poorly defined) you get lots of wasted time as assessors argue with development over semantics, next steps, and responsibilities.

Each methodology has its own limitations in this department, resulting from its focus and perspective, IMO. If you look at OSSTM, there’s a wealth of definition around activities, which really helps those implementing it differentiate what techniques they could apply in testing their system. Their template reporting form falls short on defining constructs such as root cause and finding and ’speaks’ like an auditor’s report. This doesn’t do the depth and breadth of their assessing techniques justice which means, ultimately, adopting it will take a lot of work in the realm of that normalization task we treated earlier. NIST’s methodology formalized controls even more producing the 800-53 publication. I need to look at their recent foray into app sec and reconsider ASVS much more closely and for much longer to make judgments in this realm. Currently, I’ve only considered it in the insanely and unfairly narrow context of “a set of stuff to look for.” I’ll follow up with you on t!
his later this week or next.

[Correlating Risk Systems]
Taking your question literally: Risk systems? Most risk management companies wield Powerpoint and Excel, and as such, glue is hard to come by–let alone ‘open glue.’ I don’t have much experience with Archer, but their glue is proprietary but their suite includes the ability to weave together policy, requirements, findings, and change/bug management. It sits outside the MS Office stack, but what little experience I’ve had with it wasn’t necessarily positive ;-)

A few back, I wrote about Maturity Models vs. ASVS. On SC-L, a ‘discussion’ broke out regarding Maturity Models (MM) vs. Top N lists. Like ASVS, Top 10 lists target a different problem than MMs. In particular, the discussion focused around how one should enhance their assessment practices.

I’ve edited and reproduced my SC-L post here for those of you who don’t read the list. My objective? provide specifics on what one can do to enhance assessment practices. I provide a commentary on the on-going discussion throughout. The central theme of my approach?

security vulnerabilities don’t count unless they affect development

Now the approach:

Integrate Assessment Practices

[What?]
Wrap the assessment activities (both tool-based and manual techniques) in a process that:

• Normalizes findings under a common reporting vocabulary and demonstrates impact
• Include SAST, DAST, scanning, manual, out-sourced, & ALL findings producers in this framework
• Anchors findings in either a developmental root cause or other software artifact:
• Use Case, reqs, design, spec, etc.
• Builds adaptors so that bugs are automatically entered in tracking systems
• Adaptors should include both tool-based and manual findings
• Calculates impact with an agreed-upon mechanism that rates security risk with other factors:
• Functional release criteria
• Other non-security non-functional requirements

[Realistic?]
I believe so. Cigital’s more junior consultants work on these very tasks, and they don’t require an early-adopter to fund or agree to them. There’s plenty of tooling out there to help with the adapters and plenty of presentations/papers on risk (www.riskanalys.is), normalizing findings (cwe.mitre.org/) , and assessment methodology (www.cigital.com/papers/download/j15bsi.pdf).

[Panacea?]
No. I’ve done research and consulting in functional testing. If you think security is powerless against development, try spending a few years in a tester’s shoes! Functionality may be king for development and PMs, but I’ve found that functional testing has little to no power. While a lack of features may prevent software from going out the door, very rarely do I find that functional testing can implement an effective “go/no-go” gate from their seat in the org. That’s why testing efforts seek muscle from their friend Security (and its distant cousins under quality “Load and Performance”) to stop releases from going out the door.

There’s no reason NOT to integrate with testing efforts, reporting, and groups: we should. There’s every reason security should adhere to the same interface everyone else does with developers (let them produce code and let them consume bugs)… I think the steps I outlined under ‘what’ bring us closer. I don’t think we need to (or can expect to) flip organizational precedent and behavior on its head to make progress.

[Steering]
The above scenario doesn’t allow explicitly for two key input/outputs from the software security ecosystem:

1. Handling ultra-high-priority issues in real time
2. Adjusting and evolving to changing threat landscapes

Establish a Steering Committee

[What?]
Establish a steering committee on which a software security, dev, architecture, operations, and corporate risk sit. These folk should manage the risk-model, scoring, security standards that drive the assessment verification standard, and the definition of both short-term and longer-term mitigating strategies. I’d argue that you’d like Industry representation too. That organization could come in written form (like the Top N lists) or in the form of consulting or a panel.

When incidents or firefights come into play, absolutely allow them to be handled out of band (albeit through a documented process), but! Not until they’ve been rated with the agreed-upon model.

[Realistic?]
Yes. I have several clients that use this structure. I speak with non-clients that do the same. Data gathering for scoring and prioritization is easy if you’ve done the steps in the previous section. The operations guys help you grade the pertinence of your efforts to what they’re seeing ‘in the wild’ too.

[Panacea?]
Does a steering committee help you respond with agility to a high-priority threat in real time? Not explicitly. But, it does help if your organizational stakeholders already have a working relationship and a mutual respect. Also: I think one root cause of the underlying discomfort (or dislike) with people’s perspectives on this thread has been:

“OK… you don’t like Top N lists… So what do you do?”

In my mind… The above answers that question.

[Assessment and Tools]
Do I believe that the normalized findings will emerge only from static analysis (or any other kind of vulnerability detection tool)? Absolutely not. People who follow my writings know I expect dramatic (ally high and low) numbers to be associated with tools.

Let’s summarize my data. Organizations can expect:

1. Static analysis tools to account for 15-20% of their total findings, out of the box
2. An initial false positive rate as high as 75-99% from a static analysis tool, without tuning
3. Less than 09% code coverage (by even shallow coverage metrics) from pen-testing tools

Qualitatively, I can tell you that I expect an overwhelming majority of static analysis results produced in an organization to come from customization of their adopted product.

Simply: if you base your world view on only those things a tool (any tool) produces, you’re world view is too narrow and will prove ineffective. The same is true of those who narrow their scope to the OWASP Top-10 or the SANS Top 25.

[Top N Redux]
Some have left the impression that starting with a Top N list is of no use. Please don’t think I’m in this camp. In my last two public presentations I’ve indicated, “If you’re starting from scratch these lists (or lists intrinsically baked into a tool’s capabilities for detection) are a great place to start.” Yes, one of these presentations was entitled “Why Top N lists are bad” ;-) Also:

If you can’t afford frequent industry interaction-use Top N lists as a proxy for it. They’re valuable, but like anything, only to a point.

For me, this discussion will remain circular until we think about it in terms of measured, iterative organizational improvement. Why? Because when an organization focuses on getting beyond a “Top N” list it will just create their own organization-specific “Top N” list :-) If they’re smart though, they’ll call it a dash board and vie for a promotion ;-)

From the other side? People building Top N lists know they’re not a panacea, but also know that a lot of organizations simply can’t stomach the kind of emotional investment that BSIMM (and the ilk) come with.

This leaves me with the following:

[Conclusions]

1. Top N lists are neither necessary nor sufficient for organization success
2. Top N lists are necessary but not sufficient for industry success
3. Maturity models are neither necessary nor sufficient for organizational success
4. Maturity models are necessary but not sufficient for industry success
5. Always avail yourself of what the industry produces;
6. Never confine yourself to a single industry artifact dogmatically;
7. Whatever you consume from industry, improve it by making it your own; and
8. Where-ever you are in your journey, continue to improve iteratively.

[Related Perennial Rabbit Holes] (bonus)

• Bugs vs. Flaws: John Steven’06 - www.mail-archive.com/sc-l@securecoding.org/msg00888.html
• Security Vs. Quality: Cowan ‘02 - www.securityfocus.com/archive/98/304766

The Geekonomics blog is often good. A new post indicates Apple’s veneer of more secure than Microsoft is cracking.

It was only a matter of time. I wanted to clarify that though you see a lot of security consultants carrying Macs, in Cigital’s case, it’s not an endorsement. Again, in the interest of disclosure: though I own and operate many platforms I operate more OS X at home and office than the others.

I attribute Mac adoption amongst security folk to two reasons: the platform combines a Unix-like environment with the ability to interact through email and MS Office and that the machines, for the most part, are a snappy bit of hardware, cobbled together into a “shiny object” (desirable) form. This second aspect goes a long way to explain the recent jump in ownership in the security community: “fan-boys.”

Any claims that it’s because “they’re more secure” should be considered with a fair amount of skepticism.

Yes, historically, the platform has suffered less pain of viruses and malware. Yes, certain aspects of their OS/platform and design did make improvements over XP.

The truth is this:

Apple doesn’t “Build Security In” very well at all. You don’t have to be an insider to understand why. Redmond is exporting security blogs, books, and value like never before. You don’t see a lot of Apple security people in the community though. You don’t see good solid standards-based support for authentication or web-services that would help you interact securely with your enterprise (Apple hides behind their ‘vision’ on this one). You don’t see a lot of support for Objective-C in the static analysis tool realm.

I believe that though Apple paid great lip service to security as a differentiator initially, (they even talked about phones like the iPhone becoming the basis of identity moving forward) but yet they abandoned it when they realized the cost of a real enterprise-level program. They also abandoned what was probably their best protection: the PPC processor. Their security proposition, IMO, is based on obscurity.

I’ve said before, “Woe to Apple when market share (and thus Economics) garners the attention necessary to motivate attackers to focus on the platform—attackers may find their task easier than with Vista.”

Ben Worthen broke the BSIMM story on wsj.com as was posted earlier.

I was shocked when someone said, “Oh and ASVS is also available, great” on an OWASP list. Super, I thought, but I don’t understand the connection. When I looked at the WSJ site, I noticed Jim Manico (of OWASP, Aspect, and ASVS fame) wrote, “But for those of your programming web applications, consider looking at http://www.owasp.org/index.php/ASVS - it is focused specifically on web application security evaluation.” I know Jim, he’s a good guy, and my curiosity about where the link between maturity models and verification standards was sated, but I thought I’d spend some time here quickly disambiguating them:

Verification standards, like ASVS, enumerate techniques with which an application’s correct use of security controls can be verified. It also posits (what it calls) verification requirements that serve as specifically enumerated tests of particular security controls (such as input validation). Such a standard is best operationally deployed by an organization’s application security group. In some cases, the audit group may own verification and reporting.

Auditors especially love NIST’s 800-53 standard, which recommends security controls for Federal Information Systems and Organizations. This document has frustrated me historically because of its focus on OS configuration, topology, and patch level. Largely, it ignores the application layer of an organizational stack. As such, organizations applying it have found little impact in preventing the web’s most common application security vulnerabilities. I reckon one aim of ASVS was closing this particular gap.

Regardless of what you think of ASVS or NIST 800-53, one applies the bulk of their guidance to an application and systems.

A maturity model such as BSIMM measures what an organization is doing to secure their software. Organizations will be graded on whether they’ve defined and adopted assessment (verification) techniques, but also as to the state of other organizational constructs such as a training program, post-deployment incident response capabilities, and specific application security management. A maturity model helps organizations place their personnel, use of tools, and practices against industry best practices in a broader context (a software security framework). Verification of applications is only a small piece of that governance activity. As such, BSIMM data is produced, consumed, and managed at the CISO/CIO level, rather than within the application security group (as ASVS).

Organizations will need an ability to consistently and comprehensively verify the security of their applications but this is only a piece of what those same organizations will also need to do, in a more broad context, to make sure they reduce, other-wise manage, or transfer software-induced business risk.

To this extent, comparison of BSIMM to ASVS is even a poor fit than comparing the Rational Unified Process (RUP) to Capability Maturity Model (CMM). -shiver-

James McGovern recently wrote a post on Gartner’s static analysis (SA) report. Among other things, he lamented the lack of actionable guidance within the report. A lack of implementation guidance doesn’t shock me from Gartner, I can’t say I expect that from them.

I can help James and community out by giving some of that guidance myself. I’ll try to do so in a tool-independent way. This topic deserves more than a “blog entry” format (admitting Cigital’s propensity for longer entries ;-)) so if people want more information on the topic, they can ping me.

In essence, you want to know the following about static analysis tools:

1. Who is going to run it (how will owners shepherd it)?
2. How much effort is it going to take to run (what budget will I need to support it)?
3. When do I run it and how often (Where does it fit within my SDL?)
4. What it’s going to find (How will the results enter/impact my organization)?
5. What won’t it find (How does it play with the rest of the assurance ecosystem)?

[Market Adoption and Making Your Choice]
Gartner seems underwhelmed by Microsoft’s ability to effect the commercial static analysis market. Cigital has always believed these technologies would make great features of a compiler/IDE and so we’re also disappointed. They’ve got great technologies but don’t exploit them commercially. I still like their freely available FXCop but this opinion isn’t popular. Perhaps that’s because I see a lot of value to FXCop as a unit testing framework that enables security test and other people are comparing it to glossy commercial tools like Fortify and Ounce run by security analysts in a very hands-off fashion. Who’s considering the tool really does make a huge difference in how it will fare.

From my perspective, the most successful SA tool vendors have made answering the “who should adopt the tool” question more difficult as they’ve waffled on licensing models. By the seat? By the core? By the KLoC? I’ve gotten complaints from prospective tool buyers indicating that a leading tool vendor’s price was 10X another vendor’s. Before you ask “Which one?”: I’ve heard specific examples in opposite directions!

Certain tools will look better to different potential buyers. Experience has shown positive response to Coverity’s C/C++ results from developers. Application security groups like Fortify and Ounce’s products, and several years ago, I saw QA Managers in two very different firms go ga-ga over Klockwork’s tool. I attribute people’s impressions not to a product vendor focusing on quality or security but to how closely the tool vendor’s conception of how roles interface with their tools throughout the vulnerability management and software development life cycles matches the organization’s own structure. Up until a few years ago I don’t believe SA vendors had fully conceived the “developer”, “analyst”, “security manager” roles in their products. Ounce was the one exception: they’ve always staunchly believed themselves to be principally a Security Analyst’s tool. Some vendors attempted to unify roles into use of a single interface while others attempted to split the product up into different configurations for each. This created confusion and friction between some adopting organizations and the tool vendor’s license model.

In the end you want SA tools to affect as many developers as possible and to get low-latency scan results as early in software’s life cycle as possible. However, having successfully deployed tools a variety of ways, I’m convinced central deployment is the way to go. Application Security should run the tool. This will help manage rule/configuration update, as well as central measurement collection, risk management, and issue tracking. How do we get low-latency and early-lifecycle centrally? Treat the tool as a service. More on that later.

Notable exceptions to the central deployment model include business units possessing one or more of the following characteristics:

• Acquisitions of previously successful, culturally-independent companies
• High-quality product development teams
• Fundamentally different SDL, development platform, language, and toolkits

For in-unit deployment one can “run locally” but must “report centrally” supporting security goals involving measurement and risk management. And, let’s be clear: if a business unit has a wicked-good continuous integration/build-management group, they-by all means-should be tapped to integrate with the SA service. Cigital and its clients have had success in integrating both the front (code submission) and back-end (bug/issue tracking) with Fortify and a variety of open source build/CI/bug-tracking products. This maintains a central model, but reduces latency dramatically.

[How Much Effort]
I’m sorely disappointed in the honesty of tool vendor sales-folk regarding level of effort. One of my first analogies for SA tools is this: Did your company buy Mercury products for load and performance testing? Think of your SA tool like Mercury–you’ll need at least as much money to deploy and implement the tool organization-wide as the licenses cost you. Sorry.

Organizations with more than 2-300 developers should plan on the following:

• 3-4 man weeks to do initial tuning and pilot implementation
• 4-16 man hours / large application to integrate with the static analysis tool and assure appropriate configuration
• 4-8 man hours / 50-100KLoC (language-/complexity-dependent) to triage results on a new-application scan
• 1 person / year as tool shepherd (app integration, custom-rule creation, rule-pack maintenance and release, support)

Organizations get into trouble when their SA tool enjoys broad acceptance by the organization before they effectively manage the submission and results triage/documentation process. My data indicates code submission can burn 8-24mhrs / application if security groups don’t ‘remember’ (or automate) the submission and scan setup process. Security groups doing scans centrally and hand-writing code review results can waste another 16hrs documenting their findings / application.

…think about it: if your organization allots a week / tool-assisted code review and wastes 24 hours getting the tool’s first result set and 16hrs documenting findings… that only left 10 hrs to actually review the code! Since I believe that 4-8 hrs / 50-100KLoC is necessary to triage, you can imagine how deep I think such code reviews are :-P At Cigital, we’ve got dramatic benefits to depth by decreasing cost of submission and reporting. This is key to scaling SA efforts. Those adopting tools will feel overwhelmed if they don’t plan to solve submission and reporting problems before wider roll-out.

[How Often]
As often as possible. If you’ve got a Continuous Integration initiative (CI), engage those folk. If you deploy your tool purely centrally, plan to scan each high-risk application at least once / release. No, to my knowledge, there isn’t a good answer to the “can these tools do incremental (partial) scans”. You should always keep old results around though, as they’ll (along with other measures) help you understand whether more or less vulnerabilities are being caught during each scan. Remember though, every time you add a rule to the scan, your vulnerability-finding bar just moved and your metrics might be thrown off.

Ounce, Fortify, and Klocwork seem pretty good about determining whether or not they’ve found a particular issue in a previous scan or not. This includes situations where code blocks move around a bit. This means that running regression scans to determine whether or not an issue has been fixed is viable. I can’t comment on Coverity’s capabilities here.

[What's it going to find]
I’m impressed that Fortify has published its vulnerability taxonomy and a bit shocked that others haven’t produced as public a resource. I’m excited too that vendors have agreed to comply with CWE labels when they report findings. I will say what I always do on this topic though: test the tool on your own code though. You simply don’t know what it will find until you throw all your idiosyncratic build, language, toolkit, and platform nonsense at the particular tool you’ve selected.

[What won't it find]
I’m continually shocked as to how much one can find by running tool Y after tool X on the same piece of code (replace X and Y as you see fit). As I’ve written time-and-time-again, the corner cases and exceptions missed by each engine are baffling. Let me put things in perspective for you: three of my clients have reported that their SA tool deployment find 17, 33, and 50% of the total number of issues found by assessment efforts respectively. If these percentages hold for your organization, inside a coding bug realm then your job is at best 50% done having run the tool and triaged its results.

Considering that Microsoft and Cigital believe that “bugs” account for only 50% of the vulnerability space, that leaves you unaware of 75% of your application’s vulnerability at the end of triage. As I always say, “use the tool to facilitate code review and increase a reviewer’s understanding of the code–not as a code reviewer that finds bugs automatically.”

Good luck out there, I’d love to hear about your particular experiences.
-jOHN

Myself and others have been getting Webinar invites from IBM’s developerworks regarding Rational’s AppScan Developer Edition.

This is of course part of the re-launch of the re-tooled AppScan product. It now includes a set of analysis types, some static and some dynamic. They’ve got fancy new names for subsets of each, some hair splitting to my ear and I’ve been reading/writing on this topic for 10 years.

The market for static analysis is plumping nicely year over year. Static analysis (SA) vendors, such as Fortify, are one-or-two revisions into producing dynamic analysis tools and suites that leverage hybrid approaches. With the big dynamic analysis tools controlled by IBM and HP a certain amount of this cross-over (and much more I predict) was inevitable.

It was also inevitable that that dynamic shops would take shots at the SA space. The dynamic guys got shot up pretty bad as their tools–sold as application security’s first ’silver bullet’–begun to meet with resistance on how much manual effort was still required and on how many false positives and/or missed vulnerabilities remained. Static analysis swept in promising a better solution. “Look at the source code”, they smiled, “and you can be more complete and accurate than if you just poke the application from the outside.” Well, now it’s SA’s turn in the tank and the dynamic shops are enjoying the reversal.

But, in a hybrid world–where the major tool vendors offer both static and dynamic tooling this is technically absurd. AppScan’s explanation on “String Analysis” is particularly absurd. They’ve chosen to attack the effectiveness of static analysis with their ’solution’, a static analysis technique.

It’s true that both Fortify and Ounce principally find SQL-injections through data flow techniques that propagate ‘taint’ and rely on tagging source, sink, and cleansing logic at the resolution of function calls. But I can say with authority that both tools model the potential values of strings as they propagate their data flow analysis as well. AppScan’s innovation isn’t so innovative.

Yes both SA tools, in my opinion, leave a bit to be desired in the core products’ support for what aspects of string propagation and manipulation they can follow. I won’t get into detail, but I advise security managers that it’s worth it to understand where the tool will fail you in this regard.

Coverity and the now defunct CodeAssure did an exceptionally good job modeling string values throughout code’s ’speculative execution’, as part of their static analysis. When I tested these tools, they surprised me in both what they were able to find and what other tools’ false positives they left out of reports. Alas, these two don’t help today’s security manager much if they’re focusing on Java EE or .NET web applications. CodeAssure is gone and Coverity’s product has (in my opinion) limited language support outside C/C++ (though, again, I can not stress enough how positive my experience with it in C/C++ has been).

The dynamic shops had to level the playing field. But, as near as I can tell, the current situation is this:

• The major vendors believe there’s benefit to static and dynamic analysis
• There’s a lot of room for technical improvement in the market leaders’ SA products, with respect to modeling
• The future’s tool will leverage static and dynamic techniques, because each is suited for find a particular class of problems well.

You, as a security manager, will still need to sift through the marketectures and promises and figure out which tool works best on the kind of code your organization builds/buys. A major component of this will be how customizable the tool is.

Over-reliance on ANY automated tools (static or dynamic) leaves you with un-found vulnerabilities and a false sense of security. Cigital’s assessment services rely on these tools for speed and scale, and so we’ve taken great pains to understand where their modeling bends and where it breaks. In our own practice, we augment static techniques with dynamic tests and have even begun writing some next-generation static techniques to counter these limitations. We’ve spent hundreds of hours helping many many clients wring more out of their preferred suite of tools with the same understanding: opting instead for less invasive customization and tuning.

The bottom line is (and I’ve been saying this since at least ‘03 now), there are strengths to each tool and each type of analysis. NONE will get you to the assessment finish line alone. Anyone who tells you otherwise is sellin’ you a load.

Let the posturing begin.

In the last entry, I responded to dashboard initiatives and organizations’ attempt to measure. Examples focused on vulnerability-detection because a lot of software security expenditure has as well.

Some practitioners, as well as software security managers, have realized that a focusing only on uncovering the problem won’t get it fixed. They’ve begun to document secure sample code, construct secure-development frameworks, and even harden existing open-source frameworks in order to make secure application development easier. This trend cuts across those IT shops supporting non-software businesses (such as finance, insurance, gaming, and the like) as well as software vendors. Let’s look at the 2nd surprising finding from SAMM data gathering, as published in Gary’s InformIT article:

8. Secure-by-default frameworks can be very helpful, especially if they are presented as middleware classes (but watch out for an over focus on security “stuff”).

Secure-by-default frameworks for aiding in development are absolutely my favorite part of software security. Not only do they make sense as part of any program, they represent an essential corner-turn from a hopeless attempt to keep developers abreast of what they’re doing wrong to the downward push towards the finish line by helping developers release better software. In 2001, plainly sick of documenting and demonstrating the same vulnerabilities over-and-over again in meetings and reports, I begun to talk about, seek money for, and outline what such a framework might look like for use of Servlets in the Java EE platform. Frankly, sometimes it’s just plain easier to describe how to do something correctly than it is to conceive of pen-tests or static analysis rules that will find all instances of vulnerability, some omission, or an incorrect implementation.

But, I’m not the only one that’s been climbing this hill: A broad range of approaches to constructing such frameworks are visible within my own client base.

Some organizations have taken their cue from what the network and host security guys have done in their own spaces. These software security groups see software security as a mapping between vulnerability and the need for control. This drives two sets of behavior:

With regards to the software they build, these organizations tend to strive towards having some central engineering team (or a particular development group) provide a ’security control’ that they can plug in for each vulnerability. This has had two effects in my experience:

1. The ’security framework’ ends up as an amalgam of functions designed in very point-solution fashion. For example OWASP ESAPI’s addCSRFToken() method. (see code)
2. ‘design review’ becomes a checklist activity in which security folk (playing the role of auditor) walk through a list of features they expected to include, such as those API calls described in #1 or at a higher level the inclusion of a particular security package: IE: “You’ve used ClearTrust for authentication–right?”

With regards to the software, infrastructure, and COTS these organizations purchase, they tend to apply a modified checklist (such as #2) during acquisition, and then–in the case in which they’re acquiring a security component itself–add it to the checklist for future reviews.

There are benefits to this level of framework creation/use (namely, it raises awareness of the problem, standardizes solutions making compliance easier to gauge, and does in fact place some useful functionality in the hands of developers they might have otherwise omitted or bumbled themselves).

However, I think this approach misses the mark for several important reasons.

Others build their security frameworks from scratch. Convinced that their organization is different enough to warrant their own unique approach they set to construction. Some organizations cull framework material from ‘centers of excellence’ within their organization: development teams that have already done a great job at something like demanding authentication and invalidating sessions. Others architect the security framework centrally, construct it, and then deploy it mandating development teams integrate with it as they continue otherwise normal maintenance of their apps.

There’s intrinsic benefit and difficulty to a purely bottom-up or top-down approach. If an organization goes purely bottom-up, their difficulty lies in coordinating and architecturally cohering different teams’ widgets into a common framework. The strength they can rely on is buy-in: developers are hard-pressed to complain about what they and their peers have written. The top-down approach lacks buy-in and (at times) practicality, but often enjoys a coherent unified design.

Early in ‘07, I wrote on a model for mixing the two approaches successfully.

Whether executed top-down, bottom-up, or through some compromise, I like this approach better than the one I described above. My biggest issues with it are:

1. It can become a political monster, especially if top-down
2. Being overly unique (ala: “not-invented here” or “I’m a snowflake” pathologies) can be damning

The more top-down an approach is, the more likely developers are to balk at mandates for the framework’s use. “Pfft, but that won’t work the way we built our app.”, they’ll lament. If you hear that, prepare for a deluge of exception forms. ;-)

So, how are people overcoming limitations of an ineffective feature-centric audit-model and the pains suffered from inventing their own framework from scratch? By, “meeting the developer on ‘their side of the river’”, as it was once said to me.

When a developer sits down to write or maintain a web application, they’re doing so within a very rich platform already. The .NET platform has well-integrated set of tools from the webserver up to application-support functionality provided by a single vendor: Microsoft. The Java EE platform focuses on open standards, and as such has a bevy of open-source packages that support the lower-levels of the stack that Sun provides. Simply put: a developer writing a Java EE web app is already conforming to the Servlet framework, if not the Struts, JSF, or some other MVC package. They know the platform’s packages, they’re already using them, and there are tons of great books, tutorials, and examples out there.
This is their side of the river. Demanding they build on top of those packages, but integrate with another package–for the sake of security alone–is folly.

A better answer to creating one’s own security framework–and I’ve seen this in play in more than one organization–is to secure a framework the developers already know. This:

• Maintains current platform design idiom - If Spring is ‘the right design paradigm’–great, you’re using with without having to have invented it;
• Allows consideration of the framework’s threat model - It’s easier to secure a system you know the design/deployment model of, than it is to write ‘the perfect’ security feature in a vacuum;
• Maintains transparency, as best as possible - Those securing the framework can place implementation ‘under the hood’ allowing developers to call functions as normal;
• Moves security ‘lower in the stack’
• Reduces integration points for developers

To me, these advantages are overwhelming. Showing the benefits of faithfully maintaining the underlying framework’s design could be the subject of its own entry entirely and–I feel–places the nail in the coffin of those arguing that security can be represented simply as features. I’ll defer that for now however.

Fixing problems instead of identifying them? Security guys getting traction by helping developers write code and get it out the door faster? Meeting them within the realm of the frameworks they’re already using: hardening their configuration, implementation, and deployment? This works?

Not surprising ;-)

Like Ken and Brian I’m pleased that results from this study are getting out there. I recently participated in the OWASP EU Summit, where Pravir and I conducted a session on the maturity model. The session had valuable industry participation.

One of the things I’ve harped on for years now is that experts need to pull their heads up from whatever they’ve gotten so good at doing in the weeds and avail themselves of some of the lessons learned by organizations that have been working on software security themselves. In 2006, I presented on “building one’s own software security group” and one of my first five slides uncovered a dirty little secret–true now as it was then: Fortune 500 security groups often possess more man-power and budget than security security consultancies. I see this maturity model study and the response/involvement of the people I’ve mentioned here as a giant step in the public recognition of the value security groups within large commercial entities have been providing.

To follow up on Gary’s InformIT article, I’ll be writing a series of posts on why these “Top 10 surprises” don’t come as a shock to us at Cigital and what it might mean for organizations. I’ll go in reverse order:

9. Not only are there are no magic software security metrics, bad metrics actually hurt.

First, a clarification. Gary’s article indicates that:

but even the most advanced programs don’t use any sort of balanced scorecard approach.

I certainly talk to organizations that rely on a scorecard, especially at the executive level, to report on software security activities. What’s interesting to me is the horror stories organizations tell about chasing the wrong metric. First, they lament how tracking “the wrong thing” motivated unexpected (and undesirable) behavior changes in the staff they were trying to effect. I’ve personally seen things like developers manipulating their code to break static analysis tools, signing a training sign-in sheet then leaving for the day, and vulnerability fixes that cause terrible residual risk (But pass a regression test in the form of a penetration testing exercise). Developers ‘hacking’ the system of control? We shouldn’t be surprised.

In my opinion, organizations charged the scorecard hill too early. They were frustrated with how “Fear-driven” security was but simply didn’t know what underlying measures would raise their visibility into key areas. Just as bad, they didn’t know how to take measurements precisely and inexpensively. I want to explicitly except organizations like GE from this analysis, BTW, as their rich history of scorecard management deserves its own treatment. For information on this, see GE’s presentation at Metricon 2.0.

Though there is often some pretty easy low-hanging fruit, leading organizations have begun to realize that deploying sensors to answer questions about software security spending is trickier than expected. Success involves looking at things on different levels than their budgetary line-items have defined. For instance, people are paying for penetration testing and they may have allotted money for a static analysis tool or code review services (or both). However, to determine and increase the efficiency of their assessment practices, organizations are finding more success with normalizing vulnerability findings reports, tying them back to what detected them, and generating “cost per findings” from there, rather than micro-optimizing each assessment technique itself. In my experience, this causes organizations to shift focus between assessment activities in ways that surprise them, but provide a profoundly cheaper overall assessment cost. As a result, organizations are surprised that a slight resurgence in penetration testing allows them to uncover authentication problems in their web applications more cheaply than the recently more favored static analysis tools are allowing them to. Likewise, simple customization of static analysis tools has allowed them to delete whole swaths of test cases that distracted or bloated their penetration testing efforts. More on this phenomenon can be found in my recent article.

I believe a scorecard will emerge. I believe metrics like “cost per finding” will begin to make sense and provide valuable decision support when organizations further mature the consistent application of their software security practices and realign the things they’re measuring. But, right now, great software security managers are conducting more “under the hood” diagnostics than “roll-up” reporting to increase effectiveness of their groups, reduce their cost, and weather this economic storm. No surprise there ;-)

Developers often ask security folk, “Hey, how do I protect credentials in config/property files?” or “Do I need to encrypt my production binaries?” I admire their asking security for help, but often times 1) they’ve not asked the question well enough to get a good answer and 2) security folk have a hard time getting to the root of the problem to provide decent guidance.

After teaching a threat modeling course recently I thought I’d demonstrate how considering an application’s architecture, who might attack it, and what impacts a business might face as a result of attacks interrupting or subverting the system’s ability to accomplish its business objective can be very helpful in framing context-free questions like I’ve posed above. I’m of course, talking about why threat modeling, even informally, is useful.

Let’s consider the above questions in a few contexts. I’ll ‘invent’ contexts for the purpose of this conversation, but you can likely see overlap in what you do with one or two, and incredible differences in the others.

Some circumstances will warrant protecting against threat actors making changes to configuration or the code loaded into the production environment in which extensibility of that environment’s code base and flexibility of its configuration are a reality. Examples of such environments might be:

• A mobile device or one distributed to consumers (such as phone, embedded device (TiVO, AppleTV, etc.) or similar)
• A outsourced hosting environment (Savvis, Google Cloud, etc.)

In both cases, you want to push updates or new applications/configuration in the production environment. Really, any organization that separates development from operations/deployment will have to ponder trust of its operators. Outsourcing is not a necessary condition.

“Insider” and “Administrator” threats apply in the second case, whereas a “Malicious User/operator” applies in the former. In all these circumstances, we need to rely on the actor to operate and in some cases maintain the system, but we don’t trust the threat actor with code and certain sensitive configuration elements. That’s a pickle.

In terms of the threats’ capabilities, we expect an Administrator to have intimate knowledge of (and ostensibly administrative control over) the host on which our application is deployed. They do not, probably, have reverse-engineering capabilities, or deft programming skill in non-script languages except in rare cases. Insiders may simply have access to the host (physically or through remote login), may possess the same role as the running application, but probably do not possess root privileges on our underlying host without successfully exploiting it. Their skill level will probably be less than that of an Administrator in most cases.

A malicious operator will run the complete gamut of skills as a matter of fact: there are plenty of deep technologists and tinkerers you want to sell TiVOs and cell phones to. Depending on the size of the market, the data or functionality to be protected, and the possibilities through which a “hack??? could be replicated and executed by non-skilled users (my mom will not not log into her TiVO but will definitely send her cell phone away to be “unlocked???), protective schemes may or may not be of much use.

It is because these threats’ vary widely in their skill set and their understanding of the construction of the application that I’ve combined consumer devices with hosting—seemingly unrelated architectures. Opaqueness and user/operator knowledge are slippery slopes. Some people hack their TiVO and some admins haven’t the foggiest of how to tickle an app

We’re drawn to potential impacts to make sense of the capabilities we discussed above and what they might be used for. I’ll treat each system architecture in term, giving a single attack scenario for each threat.

Hosting

The administrator lifts plain-text credentials out of a config file (u: oracle, pw: oracle) and conducts his/her own SQL-injection on the database, pulling tables and tables of user records, sells ‘em to organized crime: motivation, payoff, and impact. Let’s consider a code-replacement problem too, this time from the perspective of insider. An insider watches bug reports for a particular dev. team and after seeing a particularly insidious one, rolls back the version of production software to a vulnerable release, and follows the ‘script’ laid down by the bug report. He makes decent use of some coupon-codes, gets about 3-62??? flat-panels shipped to his house for $3, and runs.

Consumer Devices

Here, let’s consider two scenarios as well. First, the malicious operator—a consumer—twiddles with his/her DirectTV until I figure out how it stores/sends its username/password up-stream. Discerning a weak password scheme (computable from a username) and having heard a neighbor brag about his “NFL Sunday Ticket??? purchase, he/she downgrades service to the cheapest, then updates his/her DirectTV to send the neighbor’s username/password instead of their own. Beaucoup sports at petite pricing. In another scenario, imagine the malicious operator adding their own naughty code to the device to collect streams of content and drop ‘em to disk DRM-free.

One could imagine other goals, attack paths, and impacts easily. I’ve chosen these off the top of my head because they’ll demonstrate dramatically different protective schemes. I don’t know that they’re the most interest, common, or valuable scenarios to consider. I know-in some cases-that the scenarios I’ve listed are contrived.

Now, onto attacks and protective schemes:

Hosting: Code Update

Preventing insider (or even a malicious system administrator) from loading code can be accomplished by combining an organizational tweak with a platform feature (At least where Java and .NET are concerned). By demanding code and configuration (even code security policy) be signed before promotion into a production environment, and configuring the application container to check for these signatures, one can control what code executes in a particular container. Separating the “Application Deployer??? role (the one who signs and delivers code) from “Application Developer??? and “System Administrator??? prevents either from placing a rogue binary in place of the expected (signed) one.

Outstanding issues with this scheme include how to validate code to be signed. Did the developer inject back-door functionality that remains undetected? And, of course, how do you prevent an administrator from loading malware outside the application container (either side-by-side on the host or as a proxy between the application and a connecting system)?

Consumer Device: Code Update

Code signing has worked for consumer devices as well. Other encrypted binary format schemes have been employed to not only disallow unauthorized parties from loading code, but also for protecting the IP contained within code updates while in transport over Teh Intarweb.

An aside: I’ve seen a code signing scheme “to prevent malware from being placed on victims’ devices.??? However, they granted code signing certificates to anyone willing to register (with their address and some other information). Because they’d give signing ability to (basically) anyone who asked, their scheme did NOT prevent malware from getting onto the device. Instead, it only allowed for tracking of who was responsible for signing that malware ;-) Likewise, a system whose dynamic update function was protected by “proprietary encryption???, thus disallowing anyone from seeing patches or deploying malware in their stead. Because this “proprietary scheme??? was basically just LZW, it was trivially broken.

Consumer Device: Storing Credentials

Preventing a malicious operator from observing and gaming credentials will take an entirely different tack. Think about how smart cards or GSM cell phones work. In these cases (both potentially operated by unsavory folk), a card contains both logic and a secret, known to a central authority. When the central authority (a web site, something the smart card is docking with, or inserted in to) needs to authenticate the user device, it issues a challenge to the card. Logic on the card then computes a response to this challenge based on the secret it holds and issues a response. Even if the device’s user is able to intercept both challenge and response, it should remain opaque (and therefore not suffer replay).

Advanced techniques, such as Differential Power Analysis, may be able to extract key information from such schemes even when implemented correctly. A more interesting limitation of this scheme is plausibility of its deployment. Fobs are expensive to deploy and may not even fit within the architecture being analyzed.

Hosting: Storing Credentials

‘The trickiest of problems of the bunch to solve. I’ve seen a lot of different schemes discussed, some commercial, some home-rolled. One can thwart an administrator not skilled in reverse engineering and slow down a well-trained one by encrypting configuration, true. However, schemes that involve placing encryption secrets in the code that protect those config files can ultimately be reversed by a skilled administrator. Situations in which said secret was stored elsewhere and passed to the application to be used during start-up raise complexity a fair amount and can also be reversed if the administrator can debug or attach to the application’s process. And, to finish the summary, the credentials must ultimately be used somewhere. Often, it’s easier for administrators to MitM the application and that somewhere rather than do all this reversing we all find fun.

Once some kind of reasonable encryption scheme is employed (to prevent trivial attack), protective scheming should focus on detection of access to down-stream resources using lifted credentials, rather than making the problem THAT much more difficult to reverse. Think about it this way, the way I wrote up this threat, its attack scenario, and impact, it seems to me these efforts would be sufficient to exonerate the company from negligence in the case of actual attack. This may in fact be their goal, rather than preventing attack, as always depending on the impact of successful attack.

In turn, you can see that the way each problem was presented, its potential solutions vetted, and next steps were discussed depended on the threat and the architecture under consideration. I hope these few examples help frame and answer (or ask) your next “how do I secure” question.