James McGovern recently wrote a post on Gartner’s static analysis (SA) report. Among other things, he lamented the lack of actionable guidance within the report. A lack of implementation guidance doesn’t shock me from Gartner, I can’t say I expect that from them.

I can help James and community out by giving some of that guidance myself. I’ll try to do so in a tool-independent way. This topic deserves more than a “blog entry” format (admitting Cigital’s propensity for longer entries ;-)) so if people want more information on the topic, they can ping me.

In essence, you want to know the following about static analysis tools:

1. Who is going to run it (how will owners shepherd it)?
2. How much effort is it going to take to run (what budget will I need to support it)?
3. When do I run it and how often (Where does it fit within my SDL?)
4. What it’s going to find (How will the results enter/impact my organization)?
5. What won’t it find (How does it play with the rest of the assurance ecosystem)?

[Market Adoption and Making Your Choice]
Gartner seems underwhelmed by Microsoft’s ability to effect the commercial static analysis market. Cigital has always believed these technologies would make great features of a compiler/IDE and so we’re also disappointed. They’ve got great technologies but don’t exploit them commercially. I still like their freely available FXCop but this opinion isn’t popular. Perhaps that’s because I see a lot of value to FXCop as a unit testing framework that enables security test and other people are comparing it to glossy commercial tools like Fortify and Ounce run by security analysts in a very hands-off fashion. Who’s considering the tool really does make a huge difference in how it will fare.

From my perspective, the most successful SA tool vendors have made answering the “who should adopt the tool” question more difficult as they’ve waffled on licensing models. By the seat? By the core? By the KLoC? I’ve gotten complaints from prospective tool buyers indicating that a leading tool vendor’s price was 10X another vendor’s. Before you ask “Which one?”: I’ve heard specific examples in opposite directions!

Certain tools will look better to different potential buyers. Experience has shown positive response to Coverity’s C/C++ results from developers. Application security groups like Fortify and Ounce’s products, and several years ago, I saw QA Managers in two very different firms go ga-ga over Klockwork’s tool. I attribute people’s impressions not to a product vendor focusing on quality or security but to how closely the tool vendor’s conception of how roles interface with their tools throughout the vulnerability management and software development life cycles matches the organization’s own structure. Up until a few years ago I don’t believe SA vendors had fully conceived the “developer”, “analyst”, “security manager” roles in their products. Ounce was the one exception: they’ve always staunchly believed themselves to be principally a Security Analyst’s tool. Some vendors attempted to unify roles into use of a single interface while others attempted to split the product up into different configurations for each. This created confusion and friction between some adopting organizations and the tool vendor’s license model.

In the end you want SA tools to affect as many developers as possible and to get low-latency scan results as early in software’s life cycle as possible. However, having successfully deployed tools a variety of ways, I’m convinced central deployment is the way to go. Application Security should run the tool. This will help manage rule/configuration update, as well as central measurement collection, risk management, and issue tracking. How do we get low-latency and early-lifecycle centrally? Treat the tool as a service. More on that later.

Notable exceptions to the central deployment model include business units possessing one or more of the following characteristics:

• Acquisitions of previously successful, culturally-independent companies
• High-quality product development teams
• Fundamentally different SDL, development platform, language, and toolkits

For in-unit deployment one can “run locally” but must “report centrally” supporting security goals involving measurement and risk management. And, let’s be clear: if a business unit has a wicked-good continuous integration/build-management group, they-by all means-should be tapped to integrate with the SA service. Cigital and its clients have had success in integrating both the front (code submission) and back-end (bug/issue tracking) with Fortify and a variety of open source build/CI/bug-tracking products. This maintains a central model, but reduces latency dramatically.

[How Much Effort]
I’m sorely disappointed in the honesty of tool vendor sales-folk regarding level of effort. One of my first analogies for SA tools is this: Did your company buy Mercury products for load and performance testing? Think of your SA tool like Mercury–you’ll need at least as much money to deploy and implement the tool organization-wide as the licenses cost you. Sorry.

Organizations with more than 2-300 developers should plan on the following:

• 3-4 man weeks to do initial tuning and pilot implementation
• 4-16 man hours / large application to integrate with the static analysis tool and assure appropriate configuration
• 4-8 man hours / 50-100KLoC (language-/complexity-dependent) to triage results on a new-application scan
• 1 person / year as tool shepherd (app integration, custom-rule creation, rule-pack maintenance and release, support)

Organizations get into trouble when their SA tool enjoys broad acceptance by the organization before they effectively manage the submission and results triage/documentation process. My data indicates code submission can burn 8-24mhrs / application if security groups don’t ‘remember’ (or automate) the submission and scan setup process. Security groups doing scans centrally and hand-writing code review results can waste another 16hrs documenting their findings / application.

…think about it: if your organization allots a week / tool-assisted code review and wastes 24 hours getting the tool’s first result set and 16hrs documenting findings… that only left 10 hrs to actually review the code! Since I believe that 4-8 hrs / 50-100KLoC is necessary to triage, you can imagine how deep I think such code reviews are :-P At Cigital, we’ve got dramatic benefits to depth by decreasing cost of submission and reporting. This is key to scaling SA efforts. Those adopting tools will feel overwhelmed if they don’t plan to solve submission and reporting problems before wider roll-out.

[How Often]
As often as possible. If you’ve got a Continuous Integration initiative (CI), engage those folk. If you deploy your tool purely centrally, plan to scan each high-risk application at least once / release. No, to my knowledge, there isn’t a good answer to the “can these tools do incremental (partial) scans”. You should always keep old results around though, as they’ll (along with other measures) help you understand whether more or less vulnerabilities are being caught during each scan. Remember though, every time you add a rule to the scan, your vulnerability-finding bar just moved and your metrics might be thrown off.

Ounce, Fortify, and Klocwork seem pretty good about determining whether or not they’ve found a particular issue in a previous scan or not. This includes situations where code blocks move around a bit. This means that running regression scans to determine whether or not an issue has been fixed is viable. I can’t comment on Coverity’s capabilities here.

[What's it going to find]
I’m impressed that Fortify has published its vulnerability taxonomy and a bit shocked that others haven’t produced as public a resource. I’m excited too that vendors have agreed to comply with CWE labels when they report findings. I will say what I always do on this topic though: test the tool on your own code though. You simply don’t know what it will find until you throw all your idiosyncratic build, language, toolkit, and platform nonsense at the particular tool you’ve selected.

[What won't it find]
I’m continually shocked as to how much one can find by running tool Y after tool X on the same piece of code (replace X and Y as you see fit). As I’ve written time-and-time-again, the corner cases and exceptions missed by each engine are baffling. Let me put things in perspective for you: three of my clients have reported that their SA tool deployment find 17, 33, and 50% of the total number of issues found by assessment efforts respectively. If these percentages hold for your organization, inside a coding bug realm then your job is at best 50% done having run the tool and triaged its results.

Considering that Microsoft and Cigital believe that “bugs” account for only 50% of the vulnerability space, that leaves you unaware of 75% of your application’s vulnerability at the end of triage. As I always say, “use the tool to facilitate code review and increase a reviewer’s understanding of the code–not as a code reviewer that finds bugs automatically.”

Good luck out there, I’d love to hear about your particular experiences.
-jOHN

Myself and others have been getting Webinar invites from IBM’s developerworks regarding Rational’s AppScan Developer Edition.

This is of course part of the re-launch of the re-tooled AppScan product. It now includes a set of analysis types, some static and some dynamic. They’ve got fancy new names for subsets of each, some hair splitting to my ear and I’ve been reading/writing on this topic for 10 years.

The market for static analysis is plumping nicely year over year. Static analysis (SA) vendors, such as Fortify, are one-or-two revisions into producing dynamic analysis tools and suites that leverage hybrid approaches. With the big dynamic analysis tools controlled by IBM and HP a certain amount of this cross-over (and much more I predict) was inevitable.

It was also inevitable that that dynamic shops would take shots at the SA space. The dynamic guys got shot up pretty bad as their tools–sold as application security’s first ’silver bullet’–begun to meet with resistance on how much manual effort was still required and on how many false positives and/or missed vulnerabilities remained. Static analysis swept in promising a better solution. “Look at the source code”, they smiled, “and you can be more complete and accurate than if you just poke the application from the outside.” Well, now it’s SA’s turn in the tank and the dynamic shops are enjoying the reversal.

But, in a hybrid world–where the major tool vendors offer both static and dynamic tooling this is technically absurd. AppScan’s explanation on “String Analysis” is particularly absurd. They’ve chosen to attack the effectiveness of static analysis with their ’solution’, a static analysis technique.

It’s true that both Fortify and Ounce principally find SQL-injections through data flow techniques that propagate ‘taint’ and rely on tagging source, sink, and cleansing logic at the resolution of function calls. But I can say with authority that both tools model the potential values of strings as they propagate their data flow analysis as well. AppScan’s innovation isn’t so innovative.

Yes both SA tools, in my opinion, leave a bit to be desired in the core products’ support for what aspects of string propagation and manipulation they can follow. I won’t get into detail, but I advise security managers that it’s worth it to understand where the tool will fail you in this regard.

Coverity and the now defunct CodeAssure did an exceptionally good job modeling string values throughout code’s ’speculative execution’, as part of their static analysis. When I tested these tools, they surprised me in both what they were able to find and what other tools’ false positives they left out of reports. Alas, these two don’t help today’s security manager much if they’re focusing on Java EE or .NET web applications. CodeAssure is gone and Coverity’s product has (in my opinion) limited language support outside C/C++ (though, again, I can not stress enough how positive my experience with it in C/C++ has been).

The dynamic shops had to level the playing field. But, as near as I can tell, the current situation is this:

• The major vendors believe there’s benefit to static and dynamic analysis
• There’s a lot of room for technical improvement in the market leaders’ SA products, with respect to modeling
• The future’s tool will leverage static and dynamic techniques, because each is suited for find a particular class of problems well.

You, as a security manager, will still need to sift through the marketectures and promises and figure out which tool works best on the kind of code your organization builds/buys. A major component of this will be how customizable the tool is.

Over-reliance on ANY automated tools (static or dynamic) leaves you with un-found vulnerabilities and a false sense of security. Cigital’s assessment services rely on these tools for speed and scale, and so we’ve taken great pains to understand where their modeling bends and where it breaks. In our own practice, we augment static techniques with dynamic tests and have even begun writing some next-generation static techniques to counter these limitations. We’ve spent hundreds of hours helping many many clients wring more out of their preferred suite of tools with the same understanding: opting instead for less invasive customization and tuning.

The bottom line is (and I’ve been saying this since at least ‘03 now), there are strengths to each tool and each type of analysis. NONE will get you to the assessment finish line alone. Anyone who tells you otherwise is sellin’ you a load.

Let the posturing begin.

My last project for my former employer (Software AG) was a study of what software vendors do to achieve software assurance. The goal of the study was to see whether we (Software AG) were at, above, or below the norm, and to adjust investments in assurance accordingly. All but one of the vendors who participated are household names - these weren’t mom & pop shops, but major multi-national ISVs, most of them with sales of a billion dollars a year or more.

I presented a brief summary of the study results at the recent “Making the Business Case for Software Assurance” workshop hosted by Carnegie Mellon’s Software Engineering Institute, and sponsored by the US Department of Homeland Security. I’ll also be presenting an even briefer summary of the results at the 24th Annual Computer Security Applications Conference in December.

In my new role at Cigital, I’m hoping to be able to expand the survey beyond software vendors into e-commerce vendors, embedded software suppliers, financial institutions, etc., as well as to systematize the survey so it can be done by filling out a web form instead of as an interview. I welcome your suggestions as to how to make this project more relevant to vendors and software purchasers - and also welcome your participation in the survey, as well as suggestions on how to fund the ongoing work!

And finally, thanks to the (anonymous) vendors who participated in the first phase of the project. While I can’t thank them by name, I very much appreciate their input.

There has been a recent flurry of activity regarding security assurance on a hush-hush open source mailing list I lurk on. The debate recently has to do with formal methods versus code scanning… apples and oranges in my view. However, there’s a new flurry of press over Coverity’s use of their tool to analyze well-known globs of open source. (One poster suggested that passing a scan like this with flying colors means security has been attained… argh!)

Some pointers:

From Slashdot
Posted by: kdawson, on 2008-01-09 01:20:00

Stony Stevenson alerts us to a US Department of Homeland Security program in which subcontractors have been examining FOSS source code for security vulnerabilities. InformationWeek.com takes a glass-half-empty approach to reporting the story, saying that for FOSS code [1]on average 1 line in 1000 contains a security bug. From the article: ‘A total of 7,826 open source project defects have been fixed through the Homeland Security review, or one every two hours since it was launched in 2006…’ ZDNet Australia prefers to emphasize those FOSS projects that [2]fixed every reported bug, thus achieving a clean bill of health according to DHS. These include PHP, Perl, Python, Postfix, and Samba.

Firstly, I am a big fan of code scanning and believe that use of static analysis tools should always be one of the basic security steps integrated into every SDLC. However, there are huge problems with declaring security after passing a code scan with an arbitrary tool and a random set of rules. The most obvious issue is that security defects come in two flavors—bugs and flaws—each accounting for roughly 50% of defects in practice. Code scanning tools can only find bugs. Here are two stupid examples for effect: can a code scanning tool determine that no user authentication was performed? How about whether or not a playback attack will work?

The second most obvious problem is that the list of rules enforced by a static analysis engine can never be complete. Discussion about this is left as an exercise for the reader.

Architectural risk analysis (crazily called “threat modeling” by Microsofties) is, like code scanning, an essential software security best practice. Formal methods are one way to go about attacking the flaw problem. In the US we rely on flakier heuristic-based approaches such as the one we use at Cigital. But no matter the approach, we can’t ignore the architecture.

References

1. http://www.informationweek.com/story/showArticle.jhtml?articleID=205600229&cid=RSSfeed_IWK_All
2. http://www.zdnet.com.au/news/security/soa/11-open-source-projects-pass-security-health-check/0,130061744,339284949,00.htm

We’re pleased to have a guest blogger for this Justice League entry. Michael Cohen is a Senior Security Consultant at Cigital where he is responsible for leading, assessing, architecting and implementing secure software for Fortune 500 companies. Michael also works with Cigital teams on enterprise-wide security solutions intended to improve an organization’s security posture and help them meet audit and regulatory requirements. Michael just gave a talk to the Washington, DC IASA chapter that was well received, and is now the subject of this entry:

When I hear software architects talk about the architecture they’ve crafted to depict the various structures and behaviors of a system, they point out interesting techniques they’ve applied that best convey how the system should work and be put together. An architect’s enthusiasm for the little details reveals a great sense of accomplishment and creativity, but most of all good architecture conveys sorely needed information in order to help those who develop, test and maintain the system. Sharing the tips and tricks gathered from the field is what helps our community move forward to building better software. A classic example of this is the Design Patterns book written by the Gang of Four.

Not too long ago I was also sharing my ideas on architecture and security at a local IASA chapter here in Washington, DC. The group was a crowd of like-minded architects who work for large Fortune 500 companies, government agencies, and promising local startups. My topic was pragmatic secure architecture. The basic idea was to look at some real ways to incorporate security into architecture using Cigital’s risk management and threat modeling practices. You can find the slides for my presentation here.

For the uninitiated, threat modeling is a way of depicting where threats (think malicious people, attackers, and so on) can touch a software architecture and how they may be able to exploit critical assets using various attack patterns. Threat modeling is valuable for determining an architecture’s security posture. In addition, identifying risks in user requirements and business goals, and tying those risks to a threat model results in a map of how design flaws impact the system, its users and the overall business. Threat models coupled with identified high-level risks are a great way to get other stakeholders involved with security decisions. And mind you, these are stakeholders who would otherwise be unable to contribute and supply critical feedback.

The attendees at the chapter meeting were glad they attended the presentation and heard something worthwhile that they could use in their daily architectural activities. Many people brought up interesting points about how to best protect critical assets, what a real risk to a system is, and what is considered a good enough mitigation. What I found particularly interesting was how threat modeling provided a way for everyone to contribute interesting ideas about where security vulnerabilities may exist and how they could be mitigated (which, if you think about it, is the entire point of threat models).

I encourage any architect to share their ideas on building a better architecture with their peers, whether it be at a local organization or at a conference. As a senior security consultant at Cigital, I like to take shared ideas and mold them into my own unique way of thinking about the world. The best results come when I can apply new ideas to my daily activities as I help our customers assess and create more secure software.

… On an entirely separate note, McGraw still owes me money. I’m watching you, Gary.

Technorati Tags: software architecture, software quality, software security, touchpoints

The process of writing security guidance is just as important to the quality of the resulting standards as is the target: technology-specific, code-centric constructive statements. How do you succeed? By using the same muscles you exercise when you conduct secure design.

When I write Security guidance, such as the technology-specific standards I blogged about last week, I start with a threat model for the system (or design paradigm) I’m trying to protect. This could be as specific as a single system or as generic as all my client’s customer-facing 3-tier J EE Apps. I aim to remove attack vectors with sets of standards. I don’t write standards that don’t directly prevent a particular attack I’ve already documented in the same way Agile Developers ignore abstraction not mandated by the current user stories. Eventually though, removing a subsequent attack vector will require a smidge of refactoring or (more likely) augmentation.

Thinking about each attack vector individually gives necessary focus to the process and will help get past hapless statements of context-free specificity. Years ago I saw, “All data must be protected with 128-bit encryption???, within a customer’s standards. Admirable though the specificity was, the limitations of this as a standard become immediately obvious. Would 128-bit encryption suffice for archival-grade needs—such as log files? What does the plain-text or cipher-text data get used for? If we’re looking at attacks on session management within our imaginary web-app, simply encrypting a session token doesn’t effectively protect against theft and replay does it? Remember last week’s entry. There, in my example, I sought to protect against the circumstance in which (due to omission or later maintenance) a Developer forgot to explicitly demand authentication prior to accessing a piece of functionality.

Think of security standards as requirements driving a design that resists the attacks you outlined in your model. This focuses authoring and often alleviates the writer’s block a blank page can impose. Improving and completing guidance becomes tractable. In my crypto example above, the client was attempting to secure exchange of data between two systems (thick Java clients) over the Internet in a circumstance in which they could not rely on SSL being present. Data from each transition persisted beyond a single session, but had a short span of value—say a day, or week (depending on the volatility present in pricing). You’re probably already conjuring ideas about what you’d expect to see in a secure implementation. With as little context as this, my explanation of the design begins to illuminate attack vectors. From these, you can begin to answer the questions I posed above about ciphers, structure of data, and from there secure design becomes an exercise in trading off difficulties of key maintenance, performance/overhead of the encryption, and others with resisting the attacks you’d enumerate.

Not shockingly, HOW you build this guidance is as important as ending up in courier font. And, (again) it shouldn’t surprise you that I’ve chosen to get end up with code (courier) through design (via threat modeling). This is how software is built—and security should be no different. This does mean you’ll need security architects (who can code—not the corporate librarian types) to write good, detailed security standards though.

Technorati Tags: software security

I was a young Air Force lieutenant when Ken Thompson released his 1984 piece, Reflections on Trusting Trust. Assigned to a data center in the Pentagon, I was working on the B2 evaluation of Honeywell Multics with the fine folks at the National Computer Security Center and contributing some words to the growing Rainbow Series books. We were in ongoing debates over the meaning of phrases such as “top-level specification” and “on behalf of” in the Orange Book (a mail thread that went on for several years) and trying to perpetuate the wave of talking about “trusted computers” instead of “secure computers.”

We talked a lot about “how much trust do I get for how much analysis and testing” and “how much trust do I need for certain scenarios, like allowing a computer to automatically downgrade data from Top Secret to Secret.” These kinds of concepts ended up in capability maturity models, testing models, and risk models.

I took Ken’s words to heart and quickly made up my mind that we could really only move the trust bar up slightly (like from 10% to 20%), even with enormous expense. It was immediately obvious that getting up to, say, 75% would require so much calendar time that no one would wait for the result (and I think later experience with Orange Book evaluations, Common Criteria evaluations, and related programs have borne this out). Between hardware, software, firmware—and the completely unpredictable human factors—we really had no idea how even the most reviewed code would operate in relatively controlled environments (e.g., in a government facility where everyone was cleared), much less how it would operate in a hostile environment (hostile mobile code was not really a problem yet) where people might actively be up to nefarious deeds.

Why should you care about my reminiscing? Well, because I think a flavor of trust is still a major problem today and it’s costing everyone a bunch of money that could be put into real long-term solutions.

As I talk with my operations friends these days, I’m seeing a subtle shift in their thinking. They’re thinking more and more about appliances (web application firewalls, IDS and stuff like that), but not for direct security value. They seem to be thinking that since the software they install, whether purchased or built internally, will certainly have security problems, they have to install more bells and whistles so that operations can protect itself. This is beyond healthy paranoia; it’s an unhealthy distrust of people who should be active partners, even if it’s been earned by years of spectacular failures.

Then I started wondering why so many development organizations throw out requirements documents from product managers and just start over. Is it only because the requirements are so bad, or is it also because the developers just don’t trust the managers to know what they’re talking about?

And why do so many managers try to tell development organization how to create applications, instead of simply what the creation must accomplish? Do they simply not trust the developers to be aware of business objectives, or do they just not understand the creative processes involved?

And so on.

What an enormous amount of wasted cycles that could be used to greater organizational good.

We may never get to the point where we can implicitly trust software. But, can’t we at get to the point where we can trust each other?

Technorati Tags: software security, software quality

If I were to say penetration testing, what would you expect? After conversations with lots of colleagues and friends it’s become fairly clear to me that the term is massively overloaded. It actually got me thinking about the evolution of pen-testing over the last several years. And I’m not going to preach from the you-need-design-and-code-review-too pulpit. I’m just gonna talk about the innards of the pen-test itself (breaking a running system).

The old-school notion of a pen-test was one where you’d hire a bunch of l33+ h4×0r5 and turn them loose on a system to see what they are able to bust. The general idea here was to simulate a real-world attack scenario (black-box pen-test). Given adequate time, Cheetos^® and Red Bull^® you’ll probably be left with some findings that are interesting. But take a closer look at the breakdown of how those testers spent their time. Those guys likely spent a ton of their time trying to figure out how the app works, trolling for non-obvious interfaces and partially reverse-engineering the logic of the app. When you’re paying by the hour, this is bad news since they didn’t spend as much time as they could have in actually trying to break the app (they spent a bunch in just getting their arms around it.)

Evolving from these obvious limitations in the completely black-box approach the notion of the “pen-test with code” was born. This was a similar scene, but instead of poking at the running app as much, the testers would go and read code. While the testers are still at arm’s length from the dev team, it’s still better since access to the code gives a truer picture of what’s actually happening under the hood without time spent on reverse engineering. Sometimes, there would even be design docs available which really helped move things along. In the end, this is close to the modern-day version of what many expect when they buy a “pen-test.”

The major weakness of this style of pen-test is that there’s often no business context to ground the direction of the testing (on which aspects of the system you concentrate) or to baseline the value of the findings (to the company using the app, what do the discovered exploits really mean?) Some pen-testers are starting to bridge the technical-to-business gap, but they’re definitely in the minority. This step in the advancement of the pen-test is definitely a step in the right direction since it focuses the testers on generating and prioritizing findings that show demonstrable impact to your core business (as opposed to sheets of meaningless XSS vulns that are all prioritized “high” since XSS is bad.) The pen-tester gets to concentrate on breaking business logic in your application and scheming about combinations of technical vulns that lead to an interesting business problem. You’ll still get all the technical vulns, they’ll just be low priority if they can’t directly contribute to a bigger problem. Since this all plays into making risk management decisions, I call this style of white-box pen-testing the “risk-based pen-test.”

If you’re using pen-test today, you really want as close to a risk-based pen-test as you can get. Although I have heard a few reasons why you might not:

• “I don’t have access to code or design docs.” I hear this reason all the time as justification for why a black-box approach is used. In the end, even if you don’t have code, someone has gotta know something about how the application actually works. Even if it’s only from a user or sysadmin perspective, it goes miles to setting the testers off in the right direction. Find those people and link them up with the pen-testers! And business risk mapping is still possible even without code (but you’ll probably need at least ad hoc design info).
• “I use black-box pen-test nowadays as a way to simulate real-world attacks.” This notion is just off-base due, perhaps, to the lack of understanding about real-world attacks. Yes, the people “in the wild” that might attack you will have a similar skill-set to your pen-testers. The kicker is that the time available to an outside attacker is virtually infinite compared to that of a for-hire pen-tester. When it comes down to exploiting software often it’s just a matter of how much time you spend bashing your head against the problem. Outsiders can spend 2 hours a day for 6 months working on getting a single exploit working. Your pen-testers have 8 hours a day for a week to find lots of impressive results. Thus, cheat the problem by giving the pen-testers all the info you have.

In any case, let’s get back to talking about that notion of the risk-based pen-test. When I think about the requisite skills to complete that job I see many similarities to what QA folks do on a routine basis. Foremost, the QA testers will already know the application components and UIs. They should know the business value of the app and in many cases they’ll also know how the business logic of the app is supposed to work. And also, they are routinely asked to assess the specific business impact of problems in the application (they do it every time they open a bug and assign a priority.) That’s a huge advantage over someone from the “outside” coming into a pen-test engagement. Now what about differences? The biggest red-flag is that the QA people aren’t trained to attack applications to find security flaws. That’s a big disadvantage since you need those skills in order to be even remotely effective at a pen-test. But do keep in mind, QA folks are very much trained to break applications in general (it is, in fact, their job.)

So where are we going with this? Simple: let’s give the pen-testing responsibility to the QA team. They’ve already got a leg-up since they know the application. To counter the point about them not being trained in security attacks employ two simple techniques: tools and training. Some of the automated pen-testing tools out there are really great now. They’re basically extremely potent packages of electronic subject-matter expertise. Now, I would never advocate just buying a tool and hitting the “go” button and calling the job done (you don’t get very good results at all this way.) Enter the training. The automated pen-testing tools, in many ways, are very similar to other types of tools in a QA tester’s belt. They run a series of attack test-cases and report vulnerabilities when a test fails (this is a massive oversimplification of what they’re doing under the hood, but in terms of usability, the analogy holds.) So, train the QA testers on effectively using those tools. Teach them about how to feed application-specific details to the tools to make sure your coverage is high and ensure the results are more accurate.

Further, teach them about the classic notion of security testing (starting with requirements and deriving test-cases to ensure that functional requirements are implemented securely) and show them how to automate it with the tooling. In fact, that’s where this is all ending up: a risk-based security test with a wide blast-radius for getting the bulk of the benefit of a pen-test. It’s cheaper in the long-run (front-loaded cost for buying tools and building the internal skill vs. repeated, indefinite fixed-price cost.) In terms of effectiveness vs. an external pen-test, it’s definitely on the positive side of the 80%-20% rule (and all the things you might miss in that 20% will likely be caught by code review or architecture analysis.) It’s even “faster” in the sense that an organization could move through each assessment more quickly, thus enabling more assessments within a given timeframe.

In summary, I see the future of pen-testing as a push toward QA environments. Since the notion of a code review and architecture review are becoming more mainstream, this makes sense. Why continue to spend a lot on a service that ultimately should be used as a sanity-check of the running system?

Red Bull^® and Cheetos^® are trademarks of their respective companies.

Technorati Tags: penetration testing, software security

Frequently, those directing software security initiatives ask what others in their space are doing. I believed this was a perfectly reasonable question and answered, dutifully protecting each side’s confidentiality as best as humanly possible. Indeed, this kind of perspective represents one key value Cigital provides to our clients.

Over time my relationship with clients deepened, as did their maturity in software security. Their questions also deepened, getting more specific: “How far down the static analysis tool adoption path are my competitors?” I can’t see any way of answering questions this specific without giving away others’ competitive advantage, potentially exposing them to risk, or violating their trust (not to mention NDAs). Stuck wondering if I would be unable to provide further perspective, I began to question this perspective’s real value:

“Is the Jones family really the goal?” I asked myself. Actually, I’m pretty sure it isn’t. Each organization’s security efforts should grow very differently from one and other. They’ll start at different places, sure. Not only that, but even if you tackle the same problem as your competitor chooses to tackle, the ‘optimal’ approach for each organization differs. Why? Because each IT shop grew up to support their business differently. Metaphorically both you and the Joneses have children—but both sets of children have very different special needs.

Certainly, hallmarks of a good program exist. At this point, I consider organizations behind unless they:

• Conduct source code and architectural reviews of software produce and acquire;
• Leverage a static analysis tool suite in reviews;
• Train developers using hands-on instructor-lead courses;
• Augment training with continuing education;
• Distribute technology-specific security guidance, showing developers proper use of open source toolkits, frameworks, and COTS components;
• Subject applications to penetration testing in production;
• Measure both the vulnerability of software as well as the effectiveness of means by which software is secured.

While not exhaustive, the above list indicates a firm commitment to whitebox analysis. Cigital helps companies deploy the above means, in lock-step with each other, in order to get maximum benefit. Without help though, which measure you focus on next depends your organization. But additionally, how you make each effective will also depend on specifics of your organization. I’ve listed several factors that play into software security initiative decision making:

• Do you build, acquire, or buy most of your deployed software?
• Is your company an IT shop, supporting a business, or a product company?
• How do you characterize the bulk of your developers: Off-shore, out-sourced, contracted, seasoned, new to the field?
• What background do your Security resources possess: Network security, Development, Architecture, Management?
• What languages do you use, primarily, for software development?
• How varied is your development platform and build environment?
• Where do security resources report in the organization?
• Is the organizational culture centralized, one of strong governance? Or, does the organization rely on engineers to direct its efforts?
• What architectural paradigms does your application suite conform to? Are they primarily n-tier, monolithic, client/server, or federated? What messaging, transaction, structural, and behavioral patterns do they implement?
• What assets do the applications protect: functionality, data?

Obviously, when we conduct an organizational assessment we augment the above questions with a wealth of others.

Having explored data the above prompts gather, I suggest organizations build on their strengths rather than attacking weakness head-on. For instance, rather than foisting security analysts on development to do architectural analysis, I suggest teams subscribing to an agile methodology begin by augmenting use cases with misuse/abuse cases. Later, those teams can build towards threat modeling and begin to add elements of architectural analysis into their process of re-factoring. Augmenting a familiar activity always proceeds more smoothly than adding another (perhaps foreign) step. This advice stands in complete opposition to what I commonly give to IT shops with strong central governance: begin conducting architectural risk analysis immediately.

Over time, your security efforts will achieve coverage over prescribed best practices. Playing off your organization’s strengths can keep development more comfortable as it maneuvers to deal with security, often a new objective. If your security group succeeds in rolling out security practices synergistically development will view them as resources, not annoying cops. Ideally development begins to pull security into their efforts—requesting their time proactively because they see the value.

Remember, there’s no one-size-fits-all solution, especially in a discipline as young as software security. What will work best for you depends on the specifics of your organization. So rather than worrying about the Joneses look to understand your own situation better and respond to it in turn.

-jOHN

Technorati Tags: software security