Here are those numbers again in the context of a few things we’ve learned:

Cigital has always included instructor-led training (ILT) as part of its knowledge transfer to clients. From our founding in 1992 through 2006, we trained an estimated 5000 students on various aspects of software quality and software security. This was done in only “a few hundred” sessions. In addition, from the launch of our formal training offerings in January 2007 through September 2011, we delivered approximately 525 ILT days to over 7700 students. Throw in “about 50” conference tutorial sessions and other non-client-specific training sessions (but not normal conference talks or similar things) and the student number grows to about 9000, for a total of about 14,000.

There has been some shift in demand over that time. For the first 10 years or so, everything was custom. We typically spent weeks and even months building training that was very specific to platforms, frameworks, coding standards, policies, and even specific problems-of-the-day. This training was usually for relatively small numbers of people all working on something very similar. For the firm, that becomes a very expensive proposition when you get to hundreds or even thousands of developers working in multiple technologies, stacks, languages, tools, and related items. There simply isn’t enough time or dollars to make custom training for everyone.

Starting in 2006, we saw a real market demand for more standardized software security training (as differentiated from the plethora of network security, tool-specific, and generic “security” training in the marketplace, or the deep-dive, single-topic courses for things like reversing malware or DLL hooking). This demand was and continues to be much more centered on foundational training for all SDLC stakeholders (business analysts, architects, developers, quality testers, pen testers, audit, risk/compliance, and so on) and advanced training for small groups (e.g., lead architects and developers).

From early 2007 through October 2011, Cigital also deployed eLearning to firms that represent over 100,000 students who are developers, architects, testers, managers, business analysts, security operations folks, and others. The majority of clients are using our eLearning in their internal learning management systems for access by employees as well as contractors integrated into the client’s ecosystem. For external contractors without access to internal client systems, clients are using our training portal.

There has been shift in the eLearning landscape as well.

• We see almost all large firms having their own learning management system and wanting to take our material in-house. Meanwhile, smaller firms are looking to out-source everything and simply purchase access to our LMS for a given number of seats.
• There is a growing demand for tightly-focused topical modules that can be consumed in an hour or less.
• There was an initial demand for custom eLearning and then off-the-shelf became all the rage as the economy changed.
• There’s a trend to moving training closer to the activity. For example, inserting some defensive programming training directly into the developer’s IDE. We’ve actually developed plug-in technology for this one.
• As everyone sees the possibilities represented by more advanced instructional design, there is an increasing demand for what can only be described as virtual reality and flying monkeys with every image and word indexed and a holographic interface that instantly takes the student to the exact second in the module that answers with cut-and-paste content whatever question the student is pondering. Oh, and it needs to run on any device from laptops to smart phones to microwaves and in-dash satellite radios. Of course, we’re all over this, too.

As an off-shoot of our continuing BSIMM activities, Gary and I also recently wrote an article on software security training. Here are some additional thoughts.

Since the first BSIMM interview in October 2008, we’ve progressed from nine to 30 to 42 firms (and more, at this point). We’ve also measured 11 firms twice—about 19 months between measurements on average—and that has provided the BSIMM community with some unique insight on how software security initiatives change over time. Assessing 42 individual firms and performing 11 re-assessments required 81 sets of interviews in just a shade less than three years.

For my money, that’s not bad for a backyard project.

Of the 42 firms in the data pool, 27 have graciously allowed us to name them as BSIMM participants. They are: Adobe, Aon, Bank of America, Capital One, The Depository Trust & Clearing Corporation (DTCC), EMC, Fannie Mae, Google, Intel, Intuit, McKesson, Microsoft, Nokia, QUALCOMM, Sallie Mae, SAP, Scripps Networks Interactive, Sony Ericsson, Standard Life, SWIFT, Symantec, Telecom Italia, Thomson Reuters, Visa, VMware, Wells Fargo, and Zynga. To these and the other 15 firms, thank you very much for participating. You are directly responsible for advancing the cause of software security.

The BSIMM3 document is freely available under a Creative Commons license. You can get it from http://bsimm.com. Go ahead; it’s a good read. Even if you’re down the road with your software security initiative, you can get a glimpse into the actual software security activities conducted by your peers and competitors. If you’ve yet to get started, BSIMM will give you some great ideas.

As always, we are looking for more people who are interested in participating in the BSIMM study. We’d love to hear from you.

–Sammy.

There were (at least) two basic approaches to gathering the software security activity data we were looking for. One was to ask what everyone was doing and then deal with the sorting and scoring issues presented by an avalanche of ad hoc data. Alternatively, we could ask about a few of the activities we had already seen in our controlled BSIMM surveys (http://bsimm2.com). We did the latter. On one hand, had we asked general questions about if and how software security was getting done, we might have amassed more data (assuming there were specific software security activities taking place). On the other hand, I believe we would have acquired a lot of data that had little to do with software security and much more to do with security features in software and with IT security.

Because we wanted some insight into a substantial number of data points, we asked a substantial number of questions. Because we didn’t want to ask overly-simple (and transparent) yes/no question about each data point and we wanted to have some confidence (intellectual, not statistical) in the results, we asked two or three questions about each data point. In the end, we asked a bunch of True/False questions, but did not include a Not Applicable option. That was a methodological mistake on our part and several of the respondents let us know that. Also, because we didn’t want the “right” answer to be “True” in every case, we “negated” some of the questions with a strategically placed “not” that really confused some respondents.

Overall, this resulted in a daunting survey that, anecdotally, took a full hour to complete, assuming the person taking the survey actually knew all the answers for his or her organization. If the person didn’t know all the answers, they had to stop and go discover some. Unfortunately, SurveyMonkey doesn’t really lend itself to stopping and starting surveys, so some people simply gave up. A couple of respondents let us know this somewhat colorfully.

We asked 110 questions to probe different aspects of 40 software security activities from the BSIMM—all of the Level 1 activities across 12 BSIMM practices and the few most common activities from Level 2. We received 93 responses, 48 of which were complete enough to allow useful analysis. As an aside, three of the 48 (and 22 of all) respondents declined to provide their email address, preventing us from sending their survey results.

Here are some general statistics on the data pool from the 48 respondents:

These are some fairly large numbers. Even though we went well out of our way to get smaller firms to respond to the survey, we had very little success in that area. One possible knee-jerk reaction is that there simply isn’t any organizational software security initiative in most of the smaller development organizations, but I don’t believe there’s enough data to make this a defensible conclusion.

The following chart shows the average software security activity level determined via the 48 responses. Within this small data set Penetration Testing and Software Environment saw the most activity, closely followed by Code Review, Standards and Requirements, Training, and Compliance and Policy. Security Testing and Configuration Management and Vulnerability Management were noticeably less populated.

In defense of this survey, I’ll note that a handful of participating full BSIMM firms graciously took the time to complete the questions as a scientific control. There was indeed some correlation between their self-reported data and the data we had meticulously gathered through in-person interviews. Informally, however, the results did seem to indicate that self-reported data result in consistently higher scores than results gathered in-person. That is, and as I’m sure is applicable in most walks of life, our own view of whether we are getting something done won’t always match that of an unbiased outsider.

The upshot of all this is that we will step up our efforts to do more in-person interviews in a full BSIMM capacity. The consistency in the data gathering and scoring will make the results more useful to everyone as a benchmark and a yardstick. If you’d like an understanding of where your software security initiative stacks up compared to others by participating the the BSIMM, we’d like to hear from you.

As a last point, we noted through personal knowledge or bounce messages that approximately 20% (10 or so) of the 48 respondents in the usable data set no longer worked at the same place they had just a few months earlier.

As always, feedback and suggestions are welcome.

I did have a case of the “ummmmmm”s while speaking.

Here is video of my opening segment and the first question of the Q&A. To see the full session, you can view it from the NRECA conference Keynote Speakers page.

]]> http://www.cigital.com/justice-league-blog/2010/03/24/at-the-nreca-conference/feed/ 2 http://www.cigital.com/justice-league-blog/2009/11/11/bsimm-europe/ http://www.cigital.com/justice-league-blog/2009/11/11/bsimm-europe/#comments Wed, 11 Nov 2009 15:02:32 +0000 sammy http://www.cigital.com/justiceleague/?p=261 Today we officially launch BSIMM Europe, a study of 9 EU firms’ software security initiatives. We continue to focus our inital data gathering on large-scale software security initiatives at major software firms. Firms in the study include: Nokia, Standard Life, SWIFT, Telecom Italia, and Thomson Reuters.

An informIT article can be found here.

The article describes our findings regarding European software security by contrast with the original BSIMM. Overall, we have tripled the size of the BSIMM study to 27 firms with several more under way. We hope to reach 30 firms by year end.

We released BSIMM v1.5 as part of the BSIMM Europe push. The document (released under the Creative Commons) is available for download and now includes an appendix about BSIMM Europe. The original document (v1.0) has been translated into Italian (by Minded Security) and German (by Virtual Forge).

We are very excited about BSIMM progress and look forward to sharing more real data with the community. No more faith based software security!

Since the original surveys, we’ve continued to gather data in formal interviews. And, of course, more data is always better.

But, we’d really like lots more data. In that light, I’d like to announce the BSIMM Begin survey sponsored by Cigital. BSIMM Begin is a questionnaire designed to probe a firm’s progress relative to the level one BSIMM activities. It is also an experiment in self-reporting. While we exercise great care when performing in-person formal interviews, we realize that approach doesn’t scale into the hundreds in any reasonable time frame. We’re hoping that self-reported data allows for the level of analysis that will provide meaningful results to everyone in the community and, perhaps more importantly, to those participating in the survey.

If you would like to participate on behalf of your firm, please go to http://bsi-mm.com/begin/.

Thank you very much.

Please take a look at BSIMM. If you run or are active in a software security group, look at it like a yardstick. Consider the activities listed versus what your organization is doing. Look especially closely at the things all or most organizations do (e.g., http://www.informit.com/articles/article.aspx?p=1326511). If your SSG is not doing those things, you might want to consider them. If your organization needs an SSG, you should be able to use the same activities to get one started.

I want to emphasize that we could not have done this without active participation by the nine firms we interviewed. The data in BSIMM is their data. Data from the interviews we conducted were used to build the model from scratch. The examples included with the activities are real examples. After building BSIMM, we scored each organization using it. The individual scorecards, although unreleasable, are fascinating. They provide a unique glimpse into how local culture, perhaps as much or more than business imperatives, drive the approach to software security. Suffice it to say, for now, that the carrot is once again shown to be mightier than the stick.

I’ll talk more about the BSIMM and individual topics over the coming weeks.

As a final note, BSIMM is a data-driven model. The model will improve when more real-world data is added. If you’d like to discuss how to get your organization’s data into the model–and the comparison of you against others back out–please let me know at smigues -at- cigital.com.

[tags]BSIMM[/tags]

With respect to security professionals, unfortunately, TJX now appears to be “the one that got away.” Let me explain, with tongue planted firmly in cheek.

If you’re in the business of providing discount items and the economy is a little weak, then business will be good. Apparently, it won’t matter that you’ve had a mishap or two with the data entrusted to you. It has always been my contention that only a small minority of people really understand the impact of stolen personal data and, even amongst most the people who do “get it,” the theft of nearly 100 million records of personal data is almost meaningless in its enormity. The victims are worried only about one record — theirs. Either it got stolen or it didn’t. And, if it did, it’ll get abused or it won’t. And, if it does, there is an ingrained belief the credit card company will “take care of it.”

Besides, there’s a huge sale and my niece needs galoshes — the green ones that look like frogs.

Yes, I believe that, as a rule, people care about the persona of the organizations they patronize. On the other hand, I believe we are at a point where the idea that “Stuff happens” because “Hackers did it” has seeped into the public consciousness much more deeply than “It’s the big, bad conglomerate’s fault.”

Virtually everyone can feel outrage at accounting scandals and multi-million dollar salaries and insider trading and so on, especially when you’re clipping coupons to make ends meet. On the other hand, virtually everyone had been impacted by “hackers.” Phishing, spyware, malware, malicious web sites, Internet scams, spam, MySpace worms, fake wireless hotspots, lions, tigers, bears — everyone has felt the sting of “the bad guy.” I claim that for Mr. and Mrs. Average American, it’s almost natural to feel a little sorry for TJX, to feel a camaraderie even, like “Hey, welcome to the club.”

Sure, we all believe they should have done more. On the other hand, I shouldn’t have clicked on that email attachment last year. I’m quite happy with my physical attributes, but it really sounded like a good deal. And that Nigerian gentleman sounded so sincere. And who doesn’t want a few new MySpace friends. We all make mistakes, right?

But what about those of us in the consulting business who have to convince organizations to put their trust in us. They have to believe us when we say that not spending on {data|information|application|software|IT} security can have intolerable business consequences. TJX would’ve made such a great example of just what intolerable really means — a veritable poster child for “See! I wasn’t kidding!” And now it’s ruined.

Okay, I don’t really want their stock to tank and have thousands more people affected. On the other hand, the least TJX could’ve done was fire a wing of executives so they could serve the consulting industry as an example in endless PowerPoint presentations for decades to come. (I still occasionally see technology demos that claim things such as, “And this could’ve even stopped the Morris worm.”) Now, we’ll have to go out of our way to omit TJX. We can’t bring up an example where nothing that bad really happened — for the organization, not the people affected by the thieves, of course. Now, instead of answering the question “How do we prevent this happening to us?”, consultants will have to answer the question “How do they get out of it so easily?”

Yeah, I’m sure it was rough in their executive suites for a while, but I’ll bet everyone is breathing easier now, walking around saying, “Woo-Hoo, we didn’t pull a CardSystems!” TJX has a very respectable 5-year stock climb, including three splits, and the current trade price is solid. Would they prefer to have invested a little more in IT security? Of course. Would they prefer not have unplanned millions in expenses? Of course. Do huge, unplanned expenses happen to large companies with some regularity? Of course. This one is special to us because we have some insight into how relatively easily it could have been avoided or detected, but for many, it really is business as usual.

I really meant it when I said, perhaps unclearly, that I was talking about the meaning of information security “risk management” as the activity occurs in most organizations today, not any textbook definition of risk management and certainly not any definition that uses words like “exact” or “right.”

Is that better definition and more precise activity a good North Star? Absolutely. Do you tell an organization that is just learning and applying a subjective skill that if they don’t do it perfectly, they’re doing it wrong? Do you tell an organization that is doing something that works for them that they’re doing it wrong because it doesn’t match the textbook? Not if you actually want to be helpful.

The crux of the following screed is that purist “risk management” and the information security “risk management” that is (not could or should be, but is) practiced in the day-to-day business world are currently distant cousins that may meet in the future. I also feel this is the perfect time to trot out the old Yogiism that, “In theory, theory and practice are the same. In practice, they’re not.”

Many of the fundamental information sources that organizations are turning to today give no prescriptive guidance at all on risk management. The information security drivers don’t help much either. By way of example, the PCI audit procedures recommend that reviewers “Verify that the information security policy includes an annual risk assessment process that identifies threats, vulnerabilities, and results in a formal risk assessment.” FFIEC guidance for financial institutions desires a “multidisciplinary and knowledge-based approach” that identifies “all reasonably foreseeable threats.” This gives organizations no guidance at all and, after they have caused problems, all threats are reasonably foreseeable.

In my opinion, the common definition of information security risk management in the world today is the set of activities that is preventing me from failing compliance and examiner audits, that is streamlining my security operations (for better or worse; I don’t know because I don’t know how to measure it), and that appears to be reason I am not suffering even more security breaches. Again, there are always enlightened organizations that are doing more and better, but they are the minority. There is no common yardstick, no well of enlightenment. If FFIEC, Basel II, SOX, PCI, and assorted other examiners and auditors see a recognizable security program that is producing good results, then by definition whatever these organizations are doing is good “risk management.” Likewise, if I’m not suffering breaches, then I have good “risk management” in the eyes of those whose opinion I really care about (e.g., people who buy my stuff, can fine me, or can put me in jail). Call that “information security” if you want (and so it is), but saying it’s not risk management is just po-tay-to, po-tah-to.

I have my understanding of the textbook and “perfect world” definitions of risk management. I usually cast risk management as the process that starts with risk assessment. This, of course, is a catch-all term for analyses of business goals, threats, assets, event frequencies, downsides, and so on. We also need information on vulnerability, risk appetite, internal and external mandates, and so on. Some good risk models applicable to the target environment help, too. Risk management as a process then works through decision-making and continues with activities directed at risk avoidance, risk reduction (whether through reduction of loss frequency or loss severity), risk transfer, and explicit risk retention. Lather, rinse, and repeat in an ongoing “it’s a journey, not a destination” sort of way.

“Proper risk analysis,” as Alex put it, “can’t mean unnecessary controls.” I interpret this the same as I would interpret that “proper” medical care can’t produce incorrect diagnoses and “proper” maintenance can’t ever damage a vehicle. In other words, there’s no such thing in any practical sense. Smart people trying to do the right thing with the best of intentions and the best of education mess these things up all the time. Even at Harvard, someone graduates last in each and every class.

“Proper” rarely happens in the field. Proper risk management costs too much and requires to many propeller-heads to make it go. Anyone doing “proper” (full, complete, accurate, perfect) risk management would know enough not to spend the money it takes to do “proper” risk management. While accurate conclusions are crucial, very few, if any, information security decisions require that much effort or that much precision. With all due respect, I’ll wait while that sinks in.

Yes, I suppose I am dealing with some aspect of “issue management,” to throw yet another term at those activities organizations engage in to ensure their networks and applications are appropriately secure. The “issue” is “I don’t want to overspend, but I also don’t want to be the last one to do some security thing because I don’t want to explain to a jury why I’m the only organization not doing something that might have prevented a breach.” Further, the “risk tolerance of the data owner” is often of little consequence since data aggregation that increases sensitivity, public perception, and a variety of other business problems often take “risk” decisions out of the data owner’s hands.

With respect to “If your definition of risk is correct, and if your analysis is good, then all that is left is for the decision maker to figure out how willing they are to lose money,” this doesn’t happen. Even with terabytes of actuarial data, insurance companies can’t make perfect longevity predictions and odds-makers can’t reliably predict the Super Bowl winner in October. And these are really smart people, with really cool technology, with millions of dollars in backing, and the opportunity to make billions off a good decision. In other words, they’re *really* motivated. Does anyone really believe the average CIO and CISO stand a chance of having a “correct” risk definition and “good” analysis?

As to the idea that you can’t overspend because “the data owner expends only the amount of resources they are willing to allocate to reduce possibilities to their acceptable level,” well, again, I’m stymied. There’s no response for such Pollyanna thinking. It’s like a having a doctor from a world-class hospital telling a M*A*S*H surgeon to just “do it clean, quiet environment and it’ll be okay.” For example, what if the data owner postulated above is willing to spend $1B on a $1M problem? Exactly.

Activities correlating to “done correctly” and “can’t overspend” and “perfect data” and so on only happen in textbooks and fabricated business school case studies.

Information security risk management in the field is an ugly business. It’s not a matter of “when done properly” — it isn’t. It uses gut instinct for fuel and eats academics for lunch. Sometimes the process is too ethereal and we get bad decisions and too few controls. Sometimes the process is too grounded in activity and we get bad decisions and too many controls instead of more thoughtful analysis. Sometimes, bad things like public data breaches never happen even to the goofiest organizations and, sometimes, bad things seem to happen over and over even to organizations with thoughtful, resourced, and active risk management processes. It happens.

To address Hoff’s contention, I’m not “making excuses” for it; I’m documenting it. Clearly information security and risk management are not the same thing. And, to a geologist, sculptor, or someone refinishing a kitchen, marble and granite are also very different things. To the average person needing to prop up a wobbly business process, however, they’re both just rocks.

“Risk aware” organizations do spend better, but they still end up buying the same basic information security technologies as everyone else because that’s all there is. Where I see them spending more pragmatically (the perfect word here) is in training and in configuration and use of technology, which is truly a good thing. These are everyday blocking and tackling activities at most organizations. Where risk awareness truly benefits an organization is in creative areas such as software development. Thoughtfully applied security in the SDLC of an application-driven organization (think ISV or large brokerage) will result in much more overall net goodness than another internal firewall or blocking ZIP files at the Internet boundary.

“True risk management” is indeed a wonderful tool and I will always push organizations in that direction. The progression from raw art to even a glimmer of science is still a long road for information security risk management.

As to a “proper” definition of “risk management,” I spent years bemoaning the change in the definition of “hacker.” It was once a really useful word that described a true artist with code and equipment. Now, it describes any many manner of computer-based criminal activity. I wish you better luck preserving a classic definition of “risk management.”

[tags]risk management[/tags]

Let’s put a stake in the ground on what risk management means. I’m not referring to how it’s defined so much as what it actually means to business. Risk management means there is a thought process that includes ensuring the right people with adequate skills are given useful information and actually decide whether to do this or that to more effectively achieve security goals. Something like, “The available data indicate that path A at price B mitigates problems C, D, and E, but causes problem F, while path Z at price Y, mitigates problems C, E, and X, but causes problem W. What’s your decision?” Or, as happens much, much too often: “Hackers, phishers, and insiders…oh my, what are we going to do?”

Everyone makes dozens, hundreds, or perhaps thousands of risk management decisions every day. These are parents, doctors, lawyers, truck drivers, and everyone else, including CEOs, CIOs, IT security personnel, software architects, developers, and so on. Some people have good gut instincts, shoot from the hip, and end up with decisions that only occasionally burst into flames. For my risk appetite, that’s too little risk management. Others wait for every possible scrap of data, agonize over the possibilities, and end up with decisions that only occasionally aren’t completely overcome by events. That’s too much risk management.

The impact of too little risk management is usually too few security controls and, therefore, too much unpredicted expense in a variety of areas: incident response, litigation, and recovery, to name a few. These are often the result of public things that can have lasting effects on brand. This is easy to understand.

The impact of too much risk management is usually too many security controls and, therefore, too much predicted expense in a variety of areas: hardware, software, tools, people, processes, and so on. These are all internal things that can setiously impair agility, efficiency, and overhead, and this is usually much harder to understand.

Which is worse?

Let me clarify that I’m being a little fast and loose with “too much risk management” above. In my experience, the problem is almost never too much “risk management,” it’s almost always too much security fabric resulting from a fixation on or over-thinking of each and every security issue, whether applicable or not, combined with a natural tendency to equate activity with progress. As a consultant, I’ve heard some form of the following dialog hundreds of times: “What are we doing about the security problem I’ve heard about?” followed by a confident “We have people choosing A as we speak.” More security controls, especially generic plug-n-play things, does not automatically mean less risk, but it sure is highly demonstrable activity (to managers, to auditors, to examiners).

I think we’ve seen the impact of too few controls all too vividly recently. Just type “data breaches” into your favorite search engine and you’ll be inundated with examples. And that’s just one example of one kind of bad outcome associated with too few security controls.

But, too many controls can also have a direct impact on the very core of an organization. To illustrate, let’s turn to the great triad of operational excellence, product leadership, and customer intimacy.

A company whose success is based on operational excellence requires quality goods at reasonable prices. This requires very efficient operations, excellent supply chain management, and practically flawless execution. Superfluous security controls, whether in people, process, technology or embedded in software, may not prevent flawless execution, but it certainly seems like a good way to slow down a just-in-time supply chain (e.g., we quarantine all email attachments for 12 hours) and disrupt operations (e.g., by implementing ultra-fine-grained entitlements where it just isn’t necessary).

When your success is based on product leadership, you are probably innovating constantly and spending like mad to build brand. Staying ahead of everyone else requires great design and development, a fantastic knowledge of your customers, and quick time to market. Security controls that have no material relationship to any practical attack can easily derail any product process that relies on agility (e.g., SDLC security tools and processes that check for things that have no matching threat slowing down product cycles).

If customer intimacy is your key to success, then you are likely to be extraordinarily focused on customer service, with lots of choices that can be tailored to customer desires. Here you will have to excel in customer management, have product and service delivery that exceeds expectations, and engender client trust. Too few security controls (“Hey, I’m this person you’ve never met, please reset my password.” “Sure thing!”) can certainly end a client relationship, but so can unnecessary security controls (e.g., three personal questions, identify an image, and last four of your SSN just to activate the Forgot My Password page) as it also clearly indicates that you just don’t know what you’re doing. Where it’s obvious that you don’t understand the appropriate intersection between your risk management, your customer’s risk appetite, and the actual risk, a trusted partnership is pretty much out of the question.

All in all, too few security controls is probably the greater of the two evils. On the other hand, it’s probably the easiest to remedy. Even if you do no risk management at all, if you have the money to purchase and correctly install most of the major security technologies out there, the shotgun approach will in fact reduce security risk. You’ll never know if you’ve done enough or if you’ve overspent, but you’ll have a story to tell to the masses. On the other hand, a thoughtful security approach based on sound risk management will give you a story to tell to savvy customers and increasingly well-educated auditors and examiners.
[tags]risk management[/tags]