you will find our observations regarding static analysis shared by others. It’s encouraging to note that Flash Sheridan observes many of the same difficulties and more formally treats them in his ISSRE ’10 publication. It’s worth a read.

A few commentators shared some decent points I wanted to address in a comment. I’m cross-posting my commentary here:

Comments treating the application of static tools as a distinct (and unfavorable) alternative to “fixing your SDLC” disappoint me, especially from the OWASP community (of which I am a contributing paid member). In fact, the community’s OpenSAMM project actively prescribes use of static analysis in not one but four (4) unique activities. So, both BSIMM and SAMM see static analysis as a part of fixing one’s SDL. **Note that in both models, all activities are not compulsory.

“Fix your code” is another red-herring alternative to static analysis. I believe static analysis inseparable from effective usage of a secure programming toolkit (such as OWASP’s ESAPI or an organization’s own toolkit) because static analysis tools can meet the need for organizations to 1) detect consistent and thorough use of secure APIs (and non-use of dangerous ones) as well as 2) detect incorrect usage of such APIs (whether through broken call-order, un-paired functions, or other bugs). We shouldn’t forget, as Mr. Manico indicates, 3) running static tools on these security toolkits themselves finds problems that careful review may not otherwise have found.

So, secure SDL models indicate usage and we–as a community–see use in these tools in helping remind developers how to correctly build secure code. I think we have to agree static analysis tools find exploitable bugs (even though sometimes in an overly expensive or time-consuming way). So, I think we’re stuck with them as a ‘best practice’ in the short term, however little some of us may like them. To dynamic vendors out there, I’d even go so far as saying, “Early adopters have begun to over-rely on static means to find problems more effectively/efficiently found with dynamic tools. Don’t worry, the ones I’m working with will begin to tilt the balance back using assessment data”.

I would like to put a face-and-name on the key notion that almost all commentators have thus-far touched: the static analysis tool that gets used by a majority of application developers out there, in the future, will likely not look like the ones we have today. And that’s fine too.

I read a blog entry about “re-aligning training expectations for QA.” It has some useful points that both developers and so-called “security people” need to hear. I disagree with some implicit biases, however, and I think we need to get past some stereotypes that sneak out in the article.

Bias #1, obviously, is the focus on the web. Despite its omnipresence, there is more non-web software than web software in the world, and non-web software does more important stuff than all the web software combined. The role of security in software testing is vital, and the presence or absence of web technologies does not change that. Despite writing a book on Web Security Testing, I know my place in the universe. Quality assurance and software testing are disciplines far older than the web, and their mission is so much bigger than finding vulnerabilities.

Bias #2 is vulnerabilities über alles. By talking about weaving vulnerabilities into security test plans, we’ve overlooked the first place where security goes into the QA process: test strategy. Look at any of the prominent folks in QA (James and Jon Bach, Michael Bolton, Rex Black, Cem Kaner), the people I’m privileged to share podiums with at QA conferences like STAR West, STAR East, and Better Software, and you’ll see that security is part of the overall risk-based testing strategy. Risk-based testing has been around for a really long time. Longer than the web.

Before anyone talks about vulnerabilities to test for, we have to figure out what the business cares about and why. What could go wrong? Who cares? What would the impact be? Answers to those questions drive our testing strategy, and ultimately our test plans and test cases.

Bias #3 is that idea that a bunch of web vulnerabilities are equivalent in impact to the business. That is, you just toss as many as you can into your test plan and test for as much as you can. This isn’t how testing is prioritized.

You don’t organize testing based on which top X vulnerabilities are likely to affect your organization (as the blog suggests). Likelihood is one part of the puzzle. Business impact is the part that is missing. You prioritize security tests by risk severity—that marriage of likelihood and impact to the business. If I have a whole pile of very likely attacks that are all low or negligible impact, and I have a few moderately likely attacks that have high impact, I should prioritize my testing effort around the ones with greater impact to my business.

Bias #4 is the treatment of testers like second class citizens. In the blog article, developers are “detail oriented” have a “deep understanding of flows.” Contrast this with QA who merely understand “what is provided to them.” They sound impotent, as if all they can do is what they’re told. Software testing, despite whatever firsthand experience the author may have, is a mature discipline. It is older and more formalized than “security” as a discipline. Software testing is older than the Internet or the web. If software testing as a discipline has adopted security too slowly, given security’s rise to the forefront in the marketplace, that might be a legitimate criticism. But I don’t approve of the slandering QA by implying that they just take what’s given them and execute it. QA is hard and there are some really bright minds working in that field.

As someone who has been training in risk-based security testing for several years now, I totally agree with some points, but very much disagree with others. I agree that the “bug parade” (as we call it) of top X vulnerabilities to find is the wrong way to teach security testing. Risk management, though, has been a fundamental part of mainstream QA for a very long time. Likewise, risk management is the same technique that good “security people” use to prioritize their results. Risk management is certainly how the business is going to make decisions about which issues to remediate and when. Risk management is what ties this all together.

If there’s something that QA needs to learn that they’re not already learning, it’s the weaving of “security” into the risk management techniques they already know how to do. If testers fall short in their ability to apply risk management techniques, then they are falling short against the QA yardstick, there’s nothing particularly security-related in this observation. If they are applying mature QA practices with modern risk management, but are not adequately addressing the software-induced business risks facing their stakeholders, then some security training might be in order. That security training should be built on the foundation of modern QA practice, including risk-based testing.

So, in some ways we agree: speak the lingo of QA. But in other ways we disagree. I think the original article fails to give credit to the decades of substantial research and practice in QA. In other words, it’s a lot more than speaking the language. It is standing on the shoulders of giants, not their toes.

We’re pleased to have a guest blogger for this Justice League entry. Michael Cohen is a Senior Security Consultant at Cigital where he is responsible for leading, assessing, architecting and implementing secure software for Fortune 500 companies. Michael also works with Cigital teams on enterprise-wide security solutions intended to improve an organization’s security posture and help them meet audit and regulatory requirements. Michael just gave a talk to the Washington, DC IASA chapter that was well received, and is now the subject of this entry:

When I hear software architects talk about the architecture they’ve crafted to depict the various structures and behaviors of a system, they point out interesting techniques they’ve applied that best convey how the system should work and be put together. An architect’s enthusiasm for the little details reveals a great sense of accomplishment and creativity, but most of all good architecture conveys sorely needed information in order to help those who develop, test and maintain the system. Sharing the tips and tricks gathered from the field is what helps our community move forward to building better software. A classic example of this is the Design Patterns book written by the Gang of Four.

Not too long ago I was also sharing my ideas on architecture and security at a local IASA chapter here in Washington, DC. The group was a crowd of like-minded architects who work for large Fortune 500 companies, government agencies, and promising local startups. My topic was pragmatic secure architecture. The basic idea was to look at some real ways to incorporate security into architecture using Cigital’s risk management and threat modeling practices. You can find the slides for my presentation here.

For the uninitiated, threat modeling is a way of depicting where threats (think malicious people, attackers, and so on) can touch a software architecture and how they may be able to exploit critical assets using various attack patterns. Threat modeling is valuable for determining an architecture’s security posture. In addition, identifying risks in user requirements and business goals, and tying those risks to a threat model results in a map of how design flaws impact the system, its users and the overall business. Threat models coupled with identified high-level risks are a great way to get other stakeholders involved with security decisions. And mind you, these are stakeholders who would otherwise be unable to contribute and supply critical feedback.

The attendees at the chapter meeting were glad they attended the presentation and heard something worthwhile that they could use in their daily architectural activities. Many people brought up interesting points about how to best protect critical assets, what a real risk to a system is, and what is considered a good enough mitigation. What I found particularly interesting was how threat modeling provided a way for everyone to contribute interesting ideas about where security vulnerabilities may exist and how they could be mitigated (which, if you think about it, is the entire point of threat models).

I encourage any architect to share their ideas on building a better architecture with their peers, whether it be at a local organization or at a conference. As a senior security consultant at Cigital, I like to take shared ideas and mold them into my own unique way of thinking about the world. The best results come when I can apply new ideas to my daily activities as I help our customers assess and create more secure software.

… On an entirely separate note, McGraw still owes me money. I’m watching you, Gary.
[tags]software architecture,software quality,software security,touchpoints[/tags]

The next generation can live off the argument that they can certainly do better than the fogies doing the work now. The challenge for them, as it was for us we made the same claim, is living up to the bold promises. I have been through five distinct new approaches to building software, each designed to deliver better products and better delivery management. Each has moved the ball forward a bit, but we still have a LONG way to go.

Just how bad are we? There are many studies of failure in IT, but the results are all over the place. The most optimistic estimate I can find says that about 30% of projects fail. When I mention that number, a fair number of people tell me it is too low, but let’s grant ourselves the benefit of the doubt. We fail 30% of the time.

How does that compare to other industries? Before we take on the masters of quality lets start with an easier opponent – how about used car salesmen? Every year in United States, about 45 million used cars are sold. The Feds estimate that the odometers of about 450,000 of these have been tampered with, and that in another 500,000 cases the seller fails to disclose a serious known defect. That works out to a failure rate around 2%.

We in the IT business are 15 times worse than used-car salesmen. Yet, the demand for our products and services is tremendous. IT related professions dominate the top-20 positions on the US Department of Labor’s list of high-growth jobs. Why is this so, if our products are so bad?

The answer is that software, bugs and all, delivers real value. We would love our programs to work without error, to never crash, to never need patching, but even with these sorts of problems, software does amazing things. Once in awhile we all encounter a program that makes us go wow. Remember the first time you used online mapping? That was a wow, but we quickly get bored. The amazing becomes commonplace — part of the wallpaper. That’s sad. I wish it was possible to flip a switch, and show ourselves and the world juts how far we’ve come. It would be a cruder, poorer, and much more boring world without us.

That said, software is still crap – amazing crap so customers will continue buying, but crap nonetheless and we owe them better. The challenge of making software better is a difficult one. All of the Justice League blog entries are, at root, about this one subject. Sometimes we write about broad issues other times about the very specific, but our objective is always to make software better. Better functionality, more reliable, and more secure.

This blog, rather than being about solutions, is about the intriguing, challenging, and sometimes frustrating dichotomy of software, it is both amazing, with a capacity to dazzle and inspire while it is also mundane – just a bit of bug ridden code. The Roman God Janus has two faces – one facing forward, one backward.

Building software is much the same. We constantly envision the future and it drives us to imagine and to realize that imagination we build ever more complicated systems. There is the rub. The other face of Janus looks back to the past. In IT that is the hard job of not making the software, but making it good. Janus was the god of doorways, a wonderful image for IT. The doorway to the future in IT rests on equal measure on vision and quality, looking forward to what can be done and backwards at what we have. We must aspire to be Janus.

A final observation that will set the stage for a future blog entry. Our troubles lie in the imbalance between the forward looking face and the backward. There is so much potential in software that we are constantly reaching farther; farther in some cases than we can deliver. It is the reaching that gets us unto trouble, but it also the reaching that keeps us energized and fresh, and once in while makes our customers say “Wow.”

We talked a lot about “how much trust do I get for how much analysis and testing” and “how much trust do I need for certain scenarios, like allowing a computer to automatically downgrade data from Top Secret to Secret.” These kinds of concepts ended up in capability maturity models, testing models, and risk models.

I took Ken’s words to heart and quickly made up my mind that we could really only move the trust bar up slightly (like from 10% to 20%), even with enormous expense. It was immediately obvious that getting up to, say, 75% would require so much calendar time that no one would wait for the result (and I think later experience with Orange Book evaluations, Common Criteria evaluations, and related programs have borne this out). Between hardware, software, firmware—and the completely unpredictable human factors—we really had no idea how even the most reviewed code would operate in relatively controlled environments (e.g., in a government facility where everyone was cleared), much less how it would operate in a hostile environment (hostile mobile code was not really a problem yet) where people might actively be up to nefarious deeds.

Why should you care about my reminiscing? Well, because I think a flavor of trust is still a major problem today and it’s costing everyone a bunch of money that could be put into real long-term solutions.

As I talk with my operations friends these days, I’m seeing a subtle shift in their thinking. They’re thinking more and more about appliances (web application firewalls, IDS and stuff like that), but not for direct security value. They seem to be thinking that since the software they install, whether purchased or built internally, will certainly have security problems, they have to install more bells and whistles so that operations can protect itself. This is beyond healthy paranoia; it’s an unhealthy distrust of people who should be active partners, even if it’s been earned by years of spectacular failures.

Then I started wondering why so many development organizations throw out requirements documents from product managers and just start over. Is it only because the requirements are so bad, or is it also because the developers just don’t trust the managers to know what they’re talking about?

And why do so many managers try to tell development organization how to create applications, instead of simply what the creation must accomplish? Do they simply not trust the developers to be aware of business objectives, or do they just not understand the creative processes involved?

And so on.

What an enormous amount of wasted cycles that could be used to greater organizational good.

We may never get to the point where we can implicitly trust software. But, can’t we at get to the point where we can trust each other?

[tags]software security, software quality[/tags]

Sure, most of what you see when you walk into a big IT shop is still monolithic applications, but those are echoes of IT history—dead men walking. SOA won because it is a natural evolution and logical extension of the way we have been building systems since Grace Hopper showed us the light. Our first programs looked monolithic, but only on the surface. We learned that good practice requires us to decompose our programs into procedures, subroutines, (or whatnot depending on the language) that had limited functionality with well-defined interfaces—nothing too large to fit in the space between our ears. Code reuse as the Holy Grail.

SOA is just the extension of good programming practice except that high bandwidth, reliable, standards-based networks and emergent standards for things like message transport, service registration and service invocation mean that our systems need not operate in a single memory space. The acronym SOA may not survive. Software sales dudes always come up with new names to sell a polished up version of the same-old, same-old, but service-oriented architecture and, more importantly, service-oriented architecting is here to stay; its centrality in future systems is as inevitable as the sun rising in the east.

With that rant out of the way, what I want to say here is that there are unique security issues related to security. I plan to discuss these in forthcoming blog entries, but I have a meeting today with the Ettienne Reinecke, CTO of Dimension Data, one of the best networking companies in the business. I have been thinking hard about how the guys like him play in SOA beyond giving us reliable pipes. Let me share my thoughts.

Networks work really well—software not so well. Why? I suggest that the success of the networks lies in three factors—componentization, standardization and that management. Networks are not monolithic. They are composed of lots of individual pieces, each with a well-defined function (sound like SOA?). The performance and interfaces of each component type are very well-defined by standards. This means that different components can be integrated with moderate ease and some reasonable expectation that they will work together. It also means that the people who develop the components can continue to refine their designs—to polish them until they are sparkle, making each generation, faster, better and cheaper—as long as they keep the interface constant.

Finally, the network guys assume that the network isn’t going to work. They assume that components will break, so they instrument the network to monitor its performance and they watch it carefully. Good network engineers also design for graceful degradation—the ability to lose performance in increments rather than catastrophically (think escalators vs. elevators). We on the software side can learn from this. The presumption of fallibility and the capacity for graceful degradation are alien concepts in the software world. They shouldn’t be.

In moving to SOA, we are componentizing. In adopting web-service standards we are standardizing, though our standards are much less mature than those for networks. Good for us—we have to two of the lessons down—componentization and standardization.

Let’s take the final step, and build our SOAs with the assumption that they will fail (a safe assumption) and engineer-in monitoring and the capacity for graceful degradation from the beginning. We can “ping” network components to see if they are alive, query their state, test them and change their state during operation. We should be doing this for SOA services. Ultimately, system operators should be able to look at panels like those now used for network operations, showing in color codes the status of each service. They should be able to detect problems and fix them on the fly.

Essentially, as we move towards SOA we are shifting from thinking of our systems as discrete entities to thinking of a computing ecosystem that runs 24 hours a day, 365.2564 days per year (more or less). This model is completely unnecessary for a simple program that does one thing, and only runs once in a while, but these are becoming an endangered species.

The trend in IT has always been towards greater integration. The data from one program feeds into the next, and the next…and the next. We can think of a data store as the DMZ between two applications or we can think of it as a cache in a larger system. The boundaries between systems are often arbitrary and are often simple reflections of org-chart boundaries rather than computational or business logic. At the level of the enterprise, we are evolving towards a more comprehensive view, beginning with simple shared services like database and directory, but moving, inexorably and inevitably towards shared functionality.

The network engineers have already made this transition. We should lean from them and relentlessly componentize, standardize and manage.

[tags]software security[/tags]