Like Ken and Brian I’m pleased that results from this study are getting out there. I recently participated in the OWASP EU Summit, where Pravir and I conducted a session on the maturity model. The session had valuable industry participation.

One of the things I’ve harped on for years now is that experts need to pull their heads up from whatever they’ve gotten so good at doing in the weeds and avail themselves of some of the lessons learned by organizations that have been working on software security themselves. In 2006, I presented on “building one’s own software security group” and one of my first five slides uncovered a dirty little secret–true now as it was then: Fortune 500 security groups often possess more man-power and budget than security security consultancies. I see this maturity model study and the response/involvement of the people I’ve mentioned here as a giant step in the public recognition of the value security groups within large commercial entities have been providing.

To follow up on Gary’s InformIT article, I’ll be writing a series of posts on why these “Top 10 surprises” don’t come as a shock to us at Cigital and what it might mean for organizations. I’ll go in reverse order:

9. Not only are there are no magic software security metrics, bad metrics actually hurt.

First, a clarification. Gary’s article indicates that:

but even the most advanced programs don’t use any sort of balanced scorecard approach.

I certainly talk to organizations that rely on a scorecard, especially at the executive level, to report on software security activities. What’s interesting to me is the horror stories organizations tell about chasing the wrong metric. First, they lament how tracking “the wrong thing” motivated unexpected (and undesirable) behavior changes in the staff they were trying to effect. I’ve personally seen things like developers manipulating their code to break static analysis tools, signing a training sign-in sheet then leaving for the day, and vulnerability fixes that cause terrible residual risk (But pass a regression test in the form of a penetration testing exercise). Developers ‘hacking’ the system of control? We shouldn’t be surprised.

In my opinion, organizations charged the scorecard hill too early. They were frustrated with how “Fear-driven” security was but simply didn’t know what underlying measures would raise their visibility into key areas. Just as bad, they didn’t know how to take measurements precisely and inexpensively. I want to explicitly except organizations like GE from this analysis, BTW, as their rich history of scorecard management deserves its own treatment. For information on this, see GE’s presentation at Metricon 2.0.

Though there is often some pretty easy low-hanging fruit, leading organizations have begun to realize that deploying sensors to answer questions about software security spending is trickier than expected. Success involves looking at things on different levels than their budgetary line-items have defined. For instance, people are paying for penetration testing and they may have allotted money for a static analysis tool or code review services (or both). However, to determine and increase the efficiency of their assessment practices, organizations are finding more success with normalizing vulnerability findings reports, tying them back to what detected them, and generating “cost per findings” from there, rather than micro-optimizing each assessment technique itself. In my experience, this causes organizations to shift focus between assessment activities in ways that surprise them, but provide a profoundly cheaper overall assessment cost. As a result, organizations are surprised that a slight resurgence in penetration testing allows them to uncover authentication problems in their web applications more cheaply than the recently more favored static analysis tools are allowing them to. Likewise, simple customization of static analysis tools has allowed them to delete whole swaths of test cases that distracted or bloated their penetration testing efforts. More on this phenomenon can be found in my recent article.

I believe a scorecard will emerge. I believe metrics like “cost per finding” will begin to make sense and provide valuable decision support when organizations further mature the consistent application of their software security practices and realign the things they’re measuring. But, right now, great software security managers are conducting more “under the hood” diagnostics than “roll-up” reporting to increase effectiveness of their groups, reduce their cost, and weather this economic storm. No surprise there ;-)

I’ve been an organizer of ACSAC in one capacity or another for close to 20 years now, and I’ve managed to attend most years. The conference always meets in early December in a southern US city (2008 in Anaheim, 2009 in Honolulu). This year’s keynote speakers were Sami Saydjari (formerly of NSA and DARPA, and now president of his own consulting company) and Whit Diffie (Distinguished Fellow at Sun, and famous inventor of public key cryptography).

I found Sami’s talk (”Structuring for Strategic Cyber Defense: A Cyber Manhattan Project Blueprint”) both energizing and frightening. I’ve known Sami for many years, and worked with him in several capacities, and have seen his concerns for the safety of the US develop over the years. Sami views cyberspace not just as an extension of “traditional” warfare (which he dubbed “kinetic” some years ago), but as a space in its own right, with its own assets - instead of land and people controlled (or damaged/destroyed/killed), it’s information that’s at risk. He pointed out that our enemies - whom he doesn’t hesitate to name, most notably the Chinese government - are investing in all-out attacks against the US, focusing on destroying our infrastructure including power and banking. (If you think the meltdown on Wall Street of the past few months has been bad, think of how much worse it could have been if coupled with a cyberattack to wipe out banking records!) While Sami’s focus is on the US, he points out that the threat is not only to the US, but also to any other technology-dependent country, which includes most of the developed world. He believes the threat is the same order as the nuclear threat during the cold war, which was a threat to national sovereignty.

Sami noted “It’s time to stop speculating about the threat, and time to do a risk assessment”, and then described the “Dark Angel” tabletop exercises that showed how an adversary with $500M (a tiny fraction of what we’re spending on bailing out the auto industry!) and three years could take out 70% of the power infrastructure of the US for an extended period of time - think about the lawlessness in the wake of Hurricane Katrina, and extend that across the entire country for a period of months or longer.

Sami noted that “the debate between talking heads has to stop” and called for an expert congressional commission similar to the Rogers commission that studied the Space Shuttle Challenger disaster, in which Richard Feynman played a key role.

On the positive side, the incoming Obama administration seems more aware of the issues of cybersecurity than any prior administration.

Sami is calling for a Cyber Manhattan Project with the vision (to overcome national strategic vulnerabilities ASAP) and urgency (because there’s plenty of means, motive, and opportunity for an attacker) to develop strategic capabilities, including continuity of critical information infrastructure operations (perhaps including “dark power” similar to “dark fiber” available in case of emergency), addressing the systems that are designed to “fail safe fail by having catastrophic failures”, figuring out how to “reboot the power grid” in case of a failure (the power grid requires power to get started), asymmetric threats (a $500M attack could cause $1T damage), situation awareness, and metrics-based quantifiable security.

Finally, he called for the Right organizing model (along the lines of DARPA for research and NASA for operations rather than a bureaucracy like DHS), at the Right Place (in the White House, to be above the fray), with the Right Clout (a presidential statement of support), and the Right Authority (bureaucracies cannot have power over the money).

Sami certainly isn’t the only one commenting on the urgent need to address cybersecurity threats. If you haven’t read the report of the “Commission on Cybersecurity for the 44th Presidency” it’s well worth it, although it’s not as demanding as I’d like to see. Also, see a recent Wall Street Journal article (titled “Internet Attacks Are a Real and Growing Problem: A new report says cyberwar isn’t science fiction”) .

After his talk, several of us asked Sami what we could do as individuals to further the Cyber Manhattan Project. Following is his answer, reproduced with his permission.

1. Educate yourself. Learn about the nature and gravity of the national strategic cyber risk and bold and effective moves that could mitigate those risks. Learn about the policy realm and what the issues are so that youcan speak intelligently about good and bad cyber defense policy. Take a policy expert out to lunch. Learn about related history such as the early formation of NASA and of the (U.S. and U.K) Air Force and how these big changes were possible against the barriers that the establishment always puts up to such big changes.
2. Educate your peers. Once you educate yourself, help educate your professional peers through one-on-one debates and discussions. Give professional talks to peer group at conferences and to graduate level students about to emerge into the professional world.
3. Educate the public. Securing the national cyber vulnerability will require significant investment of resources and significant educational, legal, and cultural attitude changes. This requires that the public be educated on this issue and that a national public discussion take place. Facilitate this discussion by educating the public through lectures and talking to the press in terms that they can understand. I find that analogies like the Manhattan project and the space race are helpful to frame the debate. Wrote Op Ed pieces.
4. Educate leadership. Advocate for sound public policy in strategic cyber defense. Talk to executive branch leadership at whatever level you have access to help them understand the types of actions our country needs to address these risks and encourage them to take an active role. Meet with members of the legislative branch and their staff and advocate for legal and policy changes and appropriate investments to mitigate national risk. Government has a responsibility to “provide for the common defense” as laid out in the constitution. Hold the government accountable to step up to that duty to the emerging cyber space territory in the information age.
5. Serve. This is a hard one. There are at least two important ways to serve, but both will involve your most precious asset: your time. Consider investing 10% of your career in public service. This could include joining non-profit advocacy groups that help move for change and improvement in some aspect of national cyber defense. The second major category is to serve inside the government to change things from within. This can be extremely hard because it means disrupting your normal career path and perhaps your family. It is also hard when the degree of change is great and the system resists that change with all its might. Yet, this can be one of the most effective ways to cause significant change. Try hard to answer the call, particularly in later parts of your career when you have the experience and network to be most effective.

Brian Chess, Sammy Migues and I have been building a maturity model for software security. We decided to base our model on data gathered by interviewing 9 top software security programs. We developed a framework to guide a series of interviews for data acquisition.

Though we have not completed the maturity model (analysis continues apace), a number of surprises bubbled up from the data soup. We wrote them up in an article that we thought you might find interesting.

Though our approach is only “science-y” and may well be closer to anthropology than computer science, we do think it’s important to focus on what works in the real world. If you have any questions or comments about our study, we welcome contact.

Two of Cigital’s thought leaders, Paco Hope and Ben Walther, just published a new book from O’Reilly called the Web Security Testing Cookbook. I wrote the foreword for the book which is reprinted below. More information about the book can also be found on Facebook.

Web applications suffer more than their share of security attacks. Here’s why. Websites and the applications that exist on them are in some sense the virtual front door of all corporations and organizations. Growth of the Web since 1993 has been astounding, outpacing even the adoption of the television and electricity in terms of speed of wide spread adoption.

Web applications are playing a growing and increasingly prominent role in software development. In fact, pundits currently have us entering the era of Web 3.0. The problem is that security has frankly not kept pace. At the moment we have enough problems securing Web 1.0 apps that we haven’t even started on Web 2.0, not to mention Web 3.0.

Before I go on, there’s something I need to get off my chest. Web applications are an important and growing kind of software, but they’re not the only kind of software! In fact, considering the number of legacy applications, embedded devices, and other code in the world, my bet is that web applications make up only a small percentage of all things software. So when all of the software security attention of the world is focused solely on web applications, I get worried. There are plenty of other kinds of critical applications out there that don’t live on the Web. That’s why I think of myself as a software security person and not a Web application security person.

In any case, Web application security and software security do share many common problems and pitfalls (not surprising since one is a subset of the other). One common problem is treating security as a feature, or as “stuff.” Security is not “stuff.” Security is a property of a system. That means that no amount of authentication technology, magic crypto fairy dust, or service-oriented architecture (SOA) ws-* security API will automagically solve the security problem. In fact, security has more to do with testing and assurance than anything else.

Enter this book. Boy, do we need a good measure of web application security testing! You see, many “tests” devised by security experts for web app testing are not carried out with any testing rigor. It turns out that testing is its own discipline, with an entire literature behind it. What Paco and Ben bring to the table is deep knowledge of testing clue. That’s a rare combination.

One critical factor about tests that all testers worth their salt understand is that results must be actionable. A bad test result reports something vague like “You have an XSS problem in the bigjavaglob.java file.” How is a developer supposed to fix that? What’s missing is a reasonable explanation of what XSS is (cross-site scripting, of course), where in the bazillion-line file the problem may occur, and what to do to fix it. This book has enough technical information in it for decent testers to report actionable results to actual living developers.

Hopefully the lessons in this book will be adopted not only by security types but also by testing people working on web applications. In fact, Quality Assurance (QA) people will enjoy the fact that this book is aimed squarely at testers, with the notions of regression testing, coverage, and unit testing built right in. In my experience, testing people are much better at testing than security people are. Used properly, this book can transform security people into better testers, and testers into better security people. Another critical feature of this book is its clear focus on tools and automation. Modern testers use tools, as do modern security people. This book is full of real examples based on real tools, many of which you can download for free on the Net. In fact, this book serves as a guide to proper tool use since many of the open source tools described don’t come with built-in tutorials or how-to guides. I am a fan of hands-on material, and this book is about as hands-on as you can get.

An overly optimistic approach to software development has certainly led to the creation of some mind-boggling stuff, but it has likewise allowed us to paint ourselves into the corner from a security perspective. Simply put, we neglected to think about what would happen to our software if it were intentionally and maliciously attacked. The attackers are at the gates, probing our web applications every day.

Software security is the practice of building software to be secure and function properly under malicious attack. This book is about one of software security’s most important practices—security testing.

—Gary McGraw, July 2008

This is a guest post from Meera Subbarao, a senior consultant at Cigital.

Continuous Integration commonly known as CI is a process that consists of continuously compiling, testing, inspecting, and deploying source code. In any typical CI environment, this means running a new build every time code changes within a version control repository. Martin Fowler describes CI as:

A software development practice where members of a team integrate their work frequently, usually each person integrates at least daily - leading to multiple integrations per day. Each integration is verified by an automated build to detect integration errors as quickly as possible.

Many teams find that this approach leads to significantly reduced integration problems and allows a team to develop cohesive software more rapidly.

While CI is actually a process, the term Continuous Integration often is associated with three important tools in particular. As shown in the screen above the three pillars of CI are:

• A version control repository like Subversion, or CVS.
• A CI Server such as Hudson, or Cruise Control
• An automated build process like Ant or Nant

So, let’s look at each of these in detail:

Version Control Repository:

Version control repositories also known as SCM (source code management) play a crucial role in any software development environment. They also play a very important role for a successful CI process. The SCM is a central place for the team to store every needed artifact for the project. It is mandatory for the teams to put everything needed for a successful build into this repository. This includes the build scripts, property files, database scripts, all the libraries required to build the software and so on.

The CI Server:

For CI to function properly, we also need to have an automated process that monitors a version control repository and runs a build when any changes are detected. There are several CI servers available, both open source and commercial. Most of them are similar in their basic configuration and monitor a particular version control repository and run builds when any changes are detected.

Some of the most commonly used open source CI servers are; Cruise Control, Continuum, and Hudson. Hudson is particularly interesting because of its ease of configuration and compelling plug-ins, which makes integration with test and static analysis tools much easier.

Automated Build:

The process of CI is about building software often, which is accomplished through the use of a build. A sturdy build strategy is by far the most important aspect of a successful CI process. In the absence of a solid build that does more than compile your code, CI withers. With automated builds, teams can reliably perform (in an automated fashion) otherwise manual tasks like compilation, testing, and even more interesting things like software inspection and deployment.

Now that we have seen the important tools in our CI process, let’s see how a typical CI scenario looks like for a developer:

• CI server is configured to poll the version control repository continuously for changes.
• Developer commits code to the repository.
• CI server detects this change, and retrieves the latest code from the repository.
• This causes the CI server to invoke the build script with the given targets and options.
• If configured, CI Server will send out an e-mail to the specified recipients when a certain important event occurs.
• The CI server continues to poll for changes.

Why is CI Important?

This is one of the most frequently asked questions, and here are a few points to note about this powerful technique:

• Building software often greatly increases the likelihood that you will spot defects early, when they still are relatively manageable.
• Extends defect visibility.
• CI ensures that you have production ready software at every change.
• CI also ensures that you have reduced the risk of integration issues by building software at every change.
• CI server can also be configured to run continuous inspection which can assist the development team in finding potential bugs, bad programming practice, automatically check coding standards, and also provide valuable feedback on the quality of code being written

Now that we have a basic understanding of CI, why CI is important for us, and the tools required to run a successful CI process, in a series of upcoming articles we will see how to setup CI, how to maintain code quality, continuously run code inspections, generate documentations, provide feedbacks to the teams and so on. Stay tuned….