Cigital

Follow-up: Integrating Assessment Tools

Posted On March 31, 2009 by jOHN

My last post spawned some questions, which I responded to in turn. Here was my response:

[Adapters]
Adapters for assessment results can take a few forms, but let’s address three specific scenarios that fan-in to an assessment results/presentation step and a few that fan-out.

[Fan in]
Fan in typically comes from three sources: 1) static tools, 2) testing tools, and 3) manual analysis.

Adapters that deal with fan-in have three challenges to surmount:
A)Technically trans-code results from #1, #2, or #3 into a single tool’s format for roll-up, or a tool-independent alternative

B) Normalize results between #1, #2, and #3 so that ‘apples’ and ‘oranges’ get reported rather than ‘apples’ and ‘cars’

C) Code results with organization-specific “whathaveyou”

[Challenge A - Trans-code]
The good news about the tools is that they nearly all export to some XML format you can manipulate. This tackles output from #1 and #2. Organizations that adopted any tool early have struggled with keeping up with format changes but have indicated to me that they’re willing to pay this small price for the ability to ‘plug’ that many results into their developers’ bug-tracking systems.

A bigger challenge (effort and therefore cost-wise) is getting manual results (#3) into either the format driven by the tool responsible for reporting or that independent format I described. Smart consulting firms have built ‘emselves workflow to manage this. In the ’90′s we at Cigital used to pine for the report-generating automation of our then-competitor @Stake and waffle about whether our client-base was too disparate and our assessments too varied in scope to support a similar trick. I see no reason not to, within an organization, code up something that helps-technically-integrate your manual review findings with those found by tools.

[Challenge B - Normalize]
Manual reviewers report to me their discomfort with the rigidity of systems like “The Seven Kingdoms” and its competitors. Mind you, I don’t dislike these taxonomies, but people get crotchety and might complain if they’re forced to cram their findings down into a SAST or DAST report. They perceive, it would seem, the vulnerabilities they find to be like snowflakes

An independent format can be successful as well. One can roll their own (for their organization), or lean heavily on the community and go with something like Mitre’s CWE. At Cigital, we’re involved with CWE and follow-on work but have also helped people organize their own. A critical discussion of this trade off is possible, but beyond the scope of this email. That leads us into…The cost of normalizing results between #1 and #2 eclipses the technical challenges of building the adapter.

[Challenge C - 'Whathaveyou']
I would say this though: if your organization has progressed to the point where it possesses security standards (prescriptive or otherwise), they should absolutely be referenced by appropriate findings in the report. This goes for both violations of the standards and the applicability of standards that would serve to prevent or mitigate a particular risk.

References to policy/standards can be augmented with best practices, if your organization makes that distinction. I’ve seen organizations link to internal/external resources for training and/or further information as well. If you’re going to try to transition from “the bug hunt” to “building security in”, one great way is to provide developers immediately-available information as they’re making the mistake.

[Fan out]
Now, you have to ‘fan out’ into support for bug-tracking systems. Most organizations’ security groups have at least as much battle to fight here as a security consultancy, actually. Why? Because the organization hasn’t mandated a single development toolkit. The good news here is that while there may be more fan-out than there was fan-in, bug-tracking tools were built to be supplied data. Writing this portion of the adapter–a conduit between what normalized findings you’ve compiled and the offending team’s bug tracking system is fairly (technically) straightforward.

[Industry Std. 'Schema']
Sean Barnum has done a flotilla of work on this topic with Bob Martin at Mitre. Though it can be cumbersome or uninteresting to practitioners, I think the work they’re doing is important because whether its admitted or not, work on audit, testing, and verification methodologies/standards must implicitly take a stand on defining the words you listed (finding, root cause, vuln., etc.). Where such efforts take a stand, one’s organization can find alignment with their own notions quite challenging. Where it’s implicit (or worse, ambiguously and poorly defined) you get lots of wasted time as assessors argue with development over semantics, next steps, and responsibilities.

Each methodology has its own limitations in this department, resulting from its focus and perspective, IMO. If you look at OSSTM, there’s a wealth of definition around activities, which really helps those implementing it differentiate what techniques they could apply in testing their system. Their template reporting form falls short on defining constructs such as root cause and finding and ‘speaks’ like an auditor’s report. This doesn’t do the depth and breadth of their assessing techniques justice which means, ultimately, adopting it will take a lot of work in the realm of that normalization task we treated earlier. NIST’s methodology formalized controls even more producing the 800-53 publication. I need to look at their recent foray into app sec and reconsider ASVS much more closely and for much longer to make judgments in this realm. Currently, I’ve only considered it in the insanely and unfairly narrow context of “a set of stuff to look for.” I’ll follow up with you on t!
his later this week or next.

[Correlating Risk Systems]
Taking your question literally: Risk systems? Most risk management companies wield Powerpoint and Excel, and as such, glue is hard to come by–let alone ‘open glue.’ I don’t have much experience with Archer, but their glue is proprietary but their suite includes the ability to weave together policy, requirements, findings, and change/bug management. It sits outside the MS Office stack, but what little experience I’ve had with it wasn’t necessarily positive