Improving Software Security (Maturity Models and Their Ilk?)
Ben Worthen broke the BSIMM story on wsj.com as was posted earlier.
I was shocked when someone said, “Oh and ASVS is also available, great” on an OWASP list. Super, I thought, but I don’t understand the connection. When I looked at the WSJ site, I noticed Jim Manico (of OWASP, Aspect, and ASVS fame) wrote, “But for those of your programming web applications, consider looking at http://www.owasp.org/index.php/ASVS – it is focused specifically on web application security evaluation.” I know Jim, he’s a good guy, and my curiosity about where the link between maturity models and verification standards was sated, but I thought I’d spend some time here quickly disambiguating them:
Verification standards, like ASVS, enumerate techniques with which an application’s correct use of security controls can be verified. It also posits (what it calls) verification requirements that serve as specifically enumerated tests of particular security controls (such as input validation). Such a standard is best operationally deployed by an organization’s application security group. In some cases, the audit group may own verification and reporting.
Auditors especially love NIST’s 800-53 standard, which recommends security controls for Federal Information Systems and Organizations. This document has frustrated me historically because of its focus on OS configuration, topology, and patch level. Largely, it ignores the application layer of an organizational stack. As such, organizations applying it have found little impact in preventing the web’s most common application security vulnerabilities. I reckon one aim of ASVS was closing this particular gap.
Regardless of what you think of ASVS or NIST 800-53, one applies the bulk of their guidance to an application and systems.
A maturity model such as BSIMM measures what an organization is doing to secure their software. Organizations will be graded on whether they’ve defined and adopted assessment (verification) techniques, but also as to the state of other organizational constructs such as a training program, post-deployment incident response capabilities, and specific application security management. A maturity model helps organizations place their personnel, use of tools, and practices against industry best practices in a broader context (a software security framework). Verification of applications is only a small piece of that governance activity. As such, BSIMM data is produced, consumed, and managed at the CISO/CIO level, rather than within the application security group (as ASVS).
Organizations will need an ability to consistently and comprehensively verify the security of their applications but this is only a piece of what those same organizations will also need to do, in a more broad context, to make sure they reduce, other-wise manage, or transfer software-induced business risk.
To this extent, comparison of BSIMM to ASVS is even a poor fit than comparing the Rational Unified Process (RUP) to Capability Maturity Model (CMM). -shiver-
March 9th, 2009 at 3:04 pm
Thing one: “application security group” = SSG
Thing two: one fly in the BSIMM ointment is the question of whether the 110 activities we observed actually result in more secure software. good question.
science dictates that we correlate software security observables (such as bugs spotted in the field) with activities.
gem
March 9th, 2009 at 5:21 pm
I see a connection between ASVS and BSIMM, but they are not directly related of course.
BSI-MM is a maturity model, but nobody really cares about maturity models — although it’s a ripe time for marketing spin and analyst headaches. Personally, I’d rather be selling charting software and Advil, but YMMV.
What we do care about is how to “Get Things Done”, and while it is nice to know where we are going, it’s really just a bunch of guesswork right now. I think Ken van Wyck said it’s too early to be looking at maturity models right now.
What we need today are plans using open standards (for future reference, see ISO/IEC 27034). A security plan typically consists of strategic, operational, and tactical measures. CertifiedSecureWeb (based on the OSSTMM 3.0, including STAR and RAV calculations) is the best operational model we currently have for application security today, although I’m also a big fan of the Microsoft SDL-IT and HP’s STLC (aka Quality Model). Not to knock on Cigital TouchPoints or OWASP CLASP, or even my own version called the CPSL – but most of these are lacking good metrics that can be brought down to the tactical level or up to the strategic level. CSW/OSSTMM 3.0 bring in those metrics, including ISECOM SCARE metrics and many other good ones.
In Cigital terms, the operational process would be the focus of an SSG on an application assessment factory. Your tactical processes are fairly well known and defined in the literature (your books + Paco/Walther’s book), and papers on your website and the BSI website (Eric Dalci and others). It doesn’t take an analyst to figure out that Cigital uses a mix of Fortify and Ounce for web application and managed code artifacts, while using Coverity and Klocwork for C/C++ artifacts. It’s relatively assumed that any built-application that has an HTTP layer interface is probably hit by IBM AppScan, as well. Finally, I’m sure there are some finishing touches such as design flaw findings, as found in Chapter 9 of the Web Security Testing Cookbook that use a mix of artifact and HTTP layer review.
OWASP ASVS is a more formal model for that tactical process. It doesn’t make specific recommendations to a certain tool or even really class of tool. This is actually much more important right now than a maturity model or even operational process. In fact, the only thing that’s more important than ASVS to our industry at this moment would be an entire security program or plan i.e. ISO 27034.
March 9th, 2009 at 10:18 pm
Andre,
It’s clear from your comment you’ve read earlier blog entries. Thank you for following the blog. Yes, maturity models and verification standards are related. I expect an organization’s maturity to increase as it increases the consistency of validating its applications’ security. Writing down (or adopting) a standard will undoubtedly help this pursuit.
I’d like to tease apart some of the rest of your response:
[SDL, STLC?]
You bet. I like ‘em too. I especially like them as models that other product companies should strive to borrow from. In past writing I’ve given companies like Microsoft a nod for publishing some of their trials, tribulations (remember DREAD?), and successes. I’ve consistently argued that organizations producing software products crack a different nut than those IT shops supporting their own business only. Notably, I’ve argued that the thousands of unimaginable deployment configurations a product company has to support produces dramatically different security stresses to an IT shop producing a single deployment for a relatively fixed user population (There’s a gagillion different deployment scenarios for Oracle’s DB, but only one NYSE). So, I do not see SDL or STLC as our panacea. It’s not a blue-print for ‘clean pipes’ through which no ‘dirty water’ may flow.
[Open Standards?]
Eh. Organizations benefit from open standards; sure. Don’t confuse formality with openness though. I’m aware of OSSTMM. In particular, I like that it takes a solid whack at defining terms that organizations struggle to place in their own domains (what is black box testing and how does my organization use it relative to white box techniques?) I’ve seen organizations grind to a halt wound around axles such as this before. Honestly, I didn’t mention it because though I’ve had several people ask me about implementing 800-53, I’ve never seen a US-based firm go after OSSTMM adherence.
I can’t get wound up about whether an organization uses an open standard to make its assessment more consistent and thorough or whether they use proprietary means. It’s like the open source vs. COTS debate. Which results in more security? I’ve not seen evidence that either can claim causality.
Instead, I urge organizations to focus on modeling risk to their portfolio of apps and then (at a deeper level) threats in/across each of those environments. I’d rather an organization be able to consistently identify signatures indicating risk to their applications… risk that they know has affected them adversely in their own production environments… than to be compliant to either an open source verification standard or a whiz-bang proprietary assessment scheme. Data over religion.
[Cigital's Verification Std.]
What is Cigital’s methodology? Well, I’ve built and iterated a lot of methodology for Cigital and its clients, leveraging a wide variety of tools and techniques. Sure, I focus on SA and pen-testing tools in the blog. Why? Because the Architectural Risk Analysis manual I wrote Cigital is 200+ pages long…. and makes for an outrageous blog entry. But seriously: because these tools have penetrated the market reasonably well and afford scale at a small cost (compared to expertise). Yes, “a fool with a tool is still a fool.” I try to mention as many as possible not because Cigital favors them in specific ways but rather to show that there isn’t a single “best” one out there. Frankly, I think success can be obtained with the correct application of efforts to many of the commercial tools I’ve mentioned and (perhaps more readily to some) with open source equivalents! Those who’ve talked with me know how painfully aware of tools’ limitations I am.
I’d argue we spend a lot of time listening to our clients and drive verification standards to meet the needs of their risk aversion and the imposition of their chosen architectures. As for application assessment-I’m glad you brought that previous entry up: I -do- believe that verification/assessment standards can take one of two tacks (or both):
1) Define what results (rest results) one expects from an assessment
2) Define what behaviors (tests) should be conducted
My previous entry’s point was this: if you have a list of things that can go wrong you’ve got a list of expected test results. By -all- means, focus on implementing the cheapest and most consistent sensor to detect the presence (or absence) of those results. Don’t get hung up on a particular tool/technique as being the silver-bullet scenario for all detection. Why did I say this? Not because we’re geeked for static analysis tools… quite the contrary: because we’re pissed that people are spending thousands on those tools when hundreds on dynamic analysis tool would find some vulns. better. Heck, perhaps monitoring in production would acceptably mitigation impacts for pennies on the dollar in comparison to both.
Finally, I think you mistake me… unless verifications standards unduly mislead priorities, or glaringly miss the point, I see them as part of an essential maturation of the industry. Does that mean that they represent a maturity model? Nope. As I said in my post, those with great assessment practices will find they yet have other things to tackle (like training, audit, incident response, and so forth).
As for marketing and headaches, and our general readiness for maturity models?
“Cheers”, keep your advil ’cause my medicine’s’a'beer.
-jOHN
March 10th, 2009 at 2:45 am
Well then I consider that you owe me a beer now, after making those comments — and to your biggest blog fan, maybe it should be a night of drinks?
DREAD is dead (and useless; and thank god). OSSTMM is also dead, but who said it was useless? Not true. I also added CSW on top of the other [should be?] obvious benefits to OSSTMM 3.0.
Certainly, I agree on so many points you made. You and I seem to really understand the direction that this industry is moving. I, too, am in shock by the sheer amount of open-source tools available.
Yet, you also seem to agree with me that ASVS is more important right now than BSI-MM. ASVS seems to be an open, modern, app-focused version of some sections of DIACAP (which in turn is based on DITSCAP, which in turn was based on NSA IEM IPP, which in turn was based on the original OSSTMM).
ASVS is more timely “right now”. As in, “it applies to a lot of organizational needs in the app/software sec space”. So it’s not shadowed by BSI-MM (at least not for some time). I think that BSI-MM can deserve more than just a few improvements and near-certain iterations.
Somehow I’m starting to question the value of commercial SAST/DAST tools, which seems to be a trend. Why is the process stuff so hard and varied? I mean I know why, but how do we best deal with it? There are too many unknowns. Again, a hard reason to start penning down a maturity-model.
If you built the document (and Pravir as well) focused more on a pre-cursor to the models with an emphasis on definitions or request-for-comments, perhaps it would look a bit different?
Thanks for the long reply, Cheers again
March 10th, 2009 at 7:08 am
Andre,
A few things quickly before I make with the industry of wars I’m fighting today…
[SAST/MM Malaise]
…I’m seeing a LOT of push-back against SAST/DAST tools right now. Your sentiments resonate with me. I think there is a lot of blowback from hype in the market. The Gartner SAST report seems to have provided seed-crystal for negative sentiment for SAST (especially Fortify) that I’m hearing. (Perhaps the WSJ article generated some of the same sentiment in you re: BSIMM?)
I do believe that things tend to adhere to Gartner’s hype curve though, and urge people not to forsake SAST (and to revisit DAST anew) anyways. Throughout my tenure in Cigital research and then in its consulting ranks, I’ve built and customized my own SAST/DAST tooling several times. As much as I crank, I think the current tool set is very useful. ‘FAR from perfect, but useful.
[Maturity]
I think one of the things that attracted Cigital to the BSIMM effort when it was presented to us was the opportunity for data. I spent time with Pravir, folk implementing their own software security programs, and other bright-minded folk at OWASP Portugal and we made important changes to SAMM but ultimately McGraw, Chess, and Sammy craved a model built off what actual organizations were doing. They used what they had as straw-man enough to garner their nine conversations.
[Weighing one's options]
As for the timeliness or relative importance of a verification standard over a maturity model… you seem to have much stronger feelings about this than I do. There are definitely organizations I work with that need to lock down how they raise visibility into their applications’/systems’ vulnerabilities. They’ve each got a meaningful security framework in place that doesn’t warrant much change if any. While others, listing from here to there, would enjoy far more benefit from building themselves an organizational security framework that directs their efforts more broadly.
…now that I think of it, perhaps I was just an ultra-early adopter. I begun speaking about my “Enterprise Software Security Framework” (ESSF) — a precursor to SAMM or BSIMM in ‘06. So, I guess I’m (or was) more into security frameworks and one’s adherence to them than the next guy ;-)
March 10th, 2009 at 1:19 pm
Regarding the Ken van Wyk statement back there in the thread. I was with Ken in Belgium when we released the BSIMM model. I’m pretty sure he would disagree with you. We spent considerable time on the plane looking over the data. In my view, Ken is ecstatic about turning software security alchemy into emiricism.
Hey ken, what *do* you think?
gem
March 10th, 2009 at 2:28 pm
Thought I’d chime in here, as my name has been used in various ways here…
I do and still believe we’re worlds away from a set of standards like other engineering disciplines have — say, bridge designers. That said, I think the BSIMM is still a hugely valuable piece of work for all of us. It captures, really for the first time, what the best practitioners in the field are actually doing. It’s not just us pontificators blathering on what we think they _should_ be doing.
That said++, I think that comparisons with other maturity models are inevitable, if only because of the name. SEI’s work in its capability maturity model has grown over many years and is in fact used to measure and certify organizations. If we don’t want people to interpret the BSIMM in the same light, we should have not used the “MM”.
Even still, I’m really happy that the BSIMM is out. Even the top 10 surprises list is useful. Knowing that all the “big guys” are doing fuzzing is enormously useful, for example. It represents the state of the practical today, and it’s a fabulous starting point to continue to push things further along.
Cheers,
Ken
March 30th, 2009 at 9:12 am
[...] few back, I wrote about Maturity Models vs. ASVS. On SC-L, a ‘discussion’ broke out regarding Maturity Models (MM) vs. Top N lists. Like [...]
April 3rd, 2009 at 10:34 pm
Maturity Models and verification standards are like apples and oranges, as you clearly state above. My comment on the WSJ blog was not appropriate. Sorry gents.
April 6th, 2009 at 10:52 am
Jim,
Inappropriate? I’m calling to mind about 400 things Peter Griffin said in Family Guy… but not your WSJ post ;-)It’s big (humble) of you to say so, but don’t worry about it. I for my part, regret being as salty as I was in my WSJ-counter-comment. For that, accept my personal apologies.
I’m actually incredibly pleased the thread generated clarifying discussion. Hopefully it’s useful to those trying to bolster their assessment practices and define/improve a security program.
-jOHN