Ask any security guy/gal about how to best mitigate cross-site scripting (XSS) and what is the answer? It’s some variation on validating input. Look at my own writings about this topic and what will you find? Variations on the input validation theme. Input validation is a great solution for new applications, but it’s a horrible choice for existing applications.

Why this change of heart? Well, this is something that’s been coming for a quite a while. I’ve become more and more disillusioned with input validation. Let’s start with some basics.

The first few reasons are well written about. Black-lists, whether syntactic or semantic, suffer the problems of black-lists: they can only look for known bad data. Not only that, but they often prevent good data from being input.

White-lists are then given as the answer. But, I’ve been had a nagging suspicion that syntactic white-lists are useless except for a small number of highly structured types. What’s worried me is that when writing several language compilers I’ve always had cases where I had to write syntax rules that were more lenient than the language allowed and then sort out the problem in semantic analysis. This implementation restriction was due to the limitations of my parsers and my parsers were a lot more sophisticated then the reg-exp parsers in many validation frameworks. So, syntactic white-lists have to let “bad data” through; not a fool proof solution.

Semantic white-lists? Well, these are great for enumerations, but is all input enumerations?

Don’t even get me started about GET versus POST.

Where does this leave us? Well, it leaves us with writing guidance that looks like a patch-work quilt. You use semantic white-lists here, but not there. You use syntactic white-lists for some kinds of data, but they will let bad data through.

So, here’s the nail in the output validation coffin. Let’s say you had a sizeable application with lots of fields of different types. What are you going to do? Take a swing that you can figure how the right type of input validation? What if you get it wrong? Well, if you get input validation wrong you just broke some percentage of your existing installed base. How large a percentage - you don’t know.

Why not just go with an output encoding solution? Yes, it seems like you’re snipping off the leaves of the tree. Yes, you need to build some way to tag output so that your test team can test that you’ve snipped off all of the leaves.

When I think about this problem as managing risk, Am I worried about the N% of the XSS bugs I’ve not fixed or the N% of the input fields I’ve now broken? I dunno about you, but I’d rather not break something that worked and have to “roll back” that fix - thus [re-]opening up an XSS vulnerability.

Technorati Tags: security, usability

So it seems like almost everywhere you turn for advice about securing programs or resolving known security problems leads you to a ‘security guy’ telling you something along the lines of ‘well, you have to validate your inputs to prevent these kinds of issues’.

Perhaps I’ve heard it too many times or perhaps I’m just jaded, but I’m throwing the BS card. Of course, I’d never leave it at just that… I think I’ve got a pretty good case for why it’s BS.

Consider my favorite red-headed stepchild, cross-site scripting (XSS). The mechanics of this problem are simple: an application accepts some input data and then offers that data in the form of output back to a user without checking the content of the data along the way (this is the case for both reflected or stored XSS problems, fundamentally).

Now, consider a small flashback I’m about to have to a Computer Networking & Communications from my undergrad days. I know serial communications are sooo last week, but anyone remember putting together simple protocols to transfer data over a line? In a simple message-based protocol, you’d pick a few byte-values to represent a few control commands like ‘end of message’ or ‘close this channel down’. This seemed like a great plan until you tested it out and noticed that some messages were getting truncated in weird ways and occasionally the whole channel went down. If you didn’t just chalk it up to bit-gnomes and listened to the professor, what you learned was that since you intermixed the CONTROL channel with the DATA channel, your data was inadvertently being interpreted as control commands when the appropriate byte-values were present in the data being transferred. Hopefully, you then learned that to make the protocol reliable, you needed to have a mechanism to escape data that contained values that would be interpreted as control codes. How’d you implement the fix? Well, you certainly didn’t try to trace the origin and content of every byte that might enter a message. What you did was augment the send_message() function with logic to zip through the pending message and escape anything that was a control code and then you’d do the normal stuff of writing it to the wire.

Let’s pop back to reality and our XSS problem. The problem is that we’re mixing user DATA with our application’s CONTROL and allowing the user data to be interpreted as control commands (in this case, HTML and/or javascript elements that run in the victim’s browser). I can understand how input validation might help for common, pathological cases of XSS vulns, but in no way is it a complete or adequate solution because what’s really important (and the correct chokepoint to fix the issue at) is the point of output. Yeah, we need to ensure that every time we put content into the user-bound HTML stream we first encode it to be safe in that control channel/language.

This is not limited to XSS. SQL injection (or really any injection attack) is about taking input data and passing it with unchecked contents to a DB command (or any API with a control language of its own, e.g. LDAP queries). Again, input validation can help in simple cases, but you’ve gotta know a priori all the ways in which data might be used by an app and choose some kind of mutually safe set of characters to let through. Either that or encode the potentially unsafe characters so they don’t cause trouble somewhere down the line. Not very extensible, usable, or maintainable in many circumstances since it’s overly restricting and fragile to change.

Take for instance, a ‘Comments’ field in a web app. Many real-world applications really do need to allow users to use characters like ‘$’, ‘%’, ‘<’, or ‘>’ to represent things like money, percentages, and value comparisons, so I simply reject the idea of banning those characters because it’s unnecessary and indicative of misunderstanding of the real problem. Some folks (which shall remain nameless) have said, ‘well, if you need those characters, just HTML/URL encode them as part of the input validation process and you’re all set’. Well, now you’ve added another problem where you’ve gotta go decode and re-encode appropriate to any other output vectors. To supporters of this strategy I ask, how many loading dock foremen and warehouse employees do you know that would correctly interpret a printout containing ‘Boxes &amp; packing must be &lt;50lbs’? How about ‘Boxes%20%26%20packing%20must%20be%20%3C50lbs’?

Now, I’m not saying you should not do input validation. It adds great usability features and might limit impact of other programming mistakes. What I’m saying is that input validation alone isn’t enough. You’ve gotta have output encoding to truly solve it right. We need to ensure that architects & developers have a deeper understanding of what the problem really is in order for them to naturally build systems to these types of attacks.

(This entry is from the preface to the new book Exploiting Online Games that I co-authored with Greg Hoglund. The book is available in stores on Friday.)

Online games, including World of Warcraft, EverQuest, Second Life, and online poker, have taken the computer world by storm. Gaming has always been (and remains) among the prime drivers of PC technology, with deep penetration into the consumer market. In the last ten years, computer games have grown just as quickly as the Internet and can now be found in tens of millions of homes.

The Internet is experiencing plenty of adolescent growing pains along with its phenomenal growth. These pains are experienced mostly in terms of problematic and pervasive computer security issues. Online games, especially massively multiplayer online role-playing games (or MMORPGs for short), suffer from these security problems directly.

MMORPGs are made of very sophisticated software built around a massively distributed client-server architecture. Because these games push the limits of software technology, especially when it comes to state and time (not to mention the real-time interaction of hundreds of thousands of users), they are particularly interesting as a case study in software security. In fact, MMORPGs are a harbinger of technical software security issues to come.

Modern software of all kinds (not just game software) is evolving to be massively distributed, with servers interacting with and providing services for thousands of users at once. The move to Web Services and Service Oriented Architectures built using technologies like AJAX and Ruby follows hard on the heels of online games. What we learn here today is bound to be widely applicable tomorrow in every kind of software.

Adding to the urgency of the security problem is the fact that online games are big business. The most popular MMORPG in the world, World of Warcraft by Blizzard Entertainment, has over 8 million users, each of whom pay $14 per month for the privilege of playing. Analysts estimate the gaming market will reach $12 billion by 2009.

Inside the virtual worlds created by MMORPGs, simple data structures come to have value, mostly a reflection of the time gamers spend playing the game. Players accumulate and trade virtual wealth (or play money). Many of these virtual economies have per capita GDPs greater than most small nations. Not surprisingly, direct connections between the virtual economies of games and the real economy exist all over the place. Until recently, it was possible to buy in-game play money with real dollars on eBay; now many other well-developed middle markets exist. And the reverse is possible, too. This has led to the emergence of a class of players more interested in wringing virtual wealth out of the game than playing the game itself.

Wherever money is at stake, criminals gather and linger. Cheating happens. In the case of MMORPGs, cheaters have real economic incentive to break the security of the game in order to accumulate virtual items and experience points for their characters. Many of these items and even the characters themselves are then sold off to the highest bidder.

Sophisticated hackers have been working the fertile fields of MMORPGs for years, some of them making a living directly from gaming (or cheating at gaming). This book describes explicitly and in a technical way the kinds of attacks and techniques used by hackers who target games.

Why Are We Doing This?

As you can imagine, game companies take a dim view of cheating in their games. If cheating becomes rampant in a game, unsatisfied noncheating players will simply move on to another. Game developers have taken a number of steps to improve security in their games, some of them controversial (monitoring game players’ PCs behind the scenes), others legalistic (imposing strict software license agreements and terms of use), and some of them trivial to break (using symmetric cryptography but including the secret key in the game client code). Our hope is that by understanding the kinds of attacks and hacking techniques described in this book, game developers will do a better job with online game security.

We think our topic is important for several reasons: First, real money is at stake; second, many players are completely unaware of what is going on; and third, online game software security has many critical lessons that we can directly apply to other, more important software. Plus, it’s fun and controversial.

For example, some game companies have been known to use stealthy techniques most often seen in rootkits to monitor gamers’ PCs. They have also been known to resort to strong-arm tactics to suppress hackers, even those not attempting in any way to be malicious or to make money. Will manufacturers of other software or digital content adopt these techniques for themselves?

Not only are the technical issues captivating, the legal issues surrounding online games and their creative software license terms are also a harbinger of things to come. The legal battles between game companies, academics, and users are by no means over—in fact, they have just begun.

In the end, the topic of online game security poses a number of interesting questions, the most pressing one being this: How do you balance gamers’ privacy rights against game developers’ desires to prevent their games from being hacked?

Where Do We Draw the Line?

For the record, we do not condone cheating, malicious hacking, or any other game-related shenanigans. We are most interested in deeply understanding and discussing what’s going on in online game security. As practical security experts, we believe that only by gaining direct technical understanding of what happens when games are exploited can we begin to build systems that can withstand real attacks. Because in this situation money is at stake, you can be sure that attacks and exploits today are both concerted and organized.

We think it is acceptable and necessary to understand both how games really work and how they fail. The only way to do this is to study them carefully. We pull no punches technically in this book, showing you how online game clients fail from a security perspective in living detail. We also explicitly describe techniques that can be used to exploit online games. We don’t do this to create an army of online game hackers—that army is already brimming in numbers, and those already enlisted in it are unlikely to learn much from this book. We do this so that the good guys will know what they are really up against. Our main objective is to describe the kinds of weapons the existing active army of game attackers has.

In our research for this book, we have broken no laws. We expect our readers likewise not to break the law using the techniques we describe.

What’s In the book?

Like most books, this book starts out at a high level and becomes progressively more technical as it goes on.

Chapter 1, Why Games?, poses and answers some simple questions. How big are online games? How many people play? Why would anyone want to exploit them? What motivation is there to cheat in an online game? The answers to these questions will likely surprise you. Believe it or not, 10 million people play online games, billions of dollars are at stake, and some people even cheat for a living. We also provide a gentle introduction to game architecture in Chapter 1, describing the classic client-server model that most games use.

Things get more technical beginning in Chapter 2, Game Hacking 101, where we describe the very basics of game hacking. The chapter is organized around describing six basic techniques: (1) building a bot, (2) using the user interface, (3) operating a proxy, (4) manipulating memory, (5) drawing on a debugger, and (6) finding the future. We pay special attention to the topic of bots since most game exploits exist to create and operate them. Late in the chapter, we even show a very simple bot that we built so you can see exactly what bot software looks like. We then describe controversial moves taken by one game maker to thwart cheating—installing rootkit-like spyware on a gamer’s PC to keep track of what’s going on. We hold this approach in low opinion and have written a program to help you know what’s going on with these monitoring programs on your own machine. We believe game makers would be better off spending their resources to build games that were less broken than to build monitoring technology.

The next two chapters take a break from technical material to cover money and the law. In particular, Chapter 3, Money, helps us understand why some players might want to cheat. The recent book Play Money by Julian Dibbell (Basic Books, 2006) describes one (pathetic) man’s foray into professional game farming, something that a number of people actively pursue. There is enough money in play here that entire enterprises have grown up around providing middleman services for gamers, buying and selling virtual items in a marketplace. The biggest and most interesting company, Internet Gaming Entertainment, known as IGE to most people, deserves and gets a treatment of its own in this chapter.

Chapter 4, Enter the Lawyers, is about the law. Game companies (and indeed a whole host of other software makers) have created a licensing jungle in the form of end user license agreements (EULAs) and terms of use (TOU) documents. Though we are not lawyers, and by no means should you rely on our advice, we provide a brief description of U.S. copyright law and the Digital Millennium Copyright Act (DMCA). Then we go through an entertaining (and somewhat scary) parade of EULAs gone bad—from Sony’s rootkit debacle to viruses protected by EULAs. We end up with a discussion of your rights as a software user and gamer.

Technical aspects of online game security begin to pick back up in Chapter 5, Infested with Bugs. We spend this chapter talking about the kinds of vulnerabilities found in many games, explaining how attackers usethem to build working exploits. We pay particular attention to bugs involving time and state, which, as we alluded to earlier, are the kinds of bugs we can expect to see much more of as other software evolves to become more like game software.

Chapter 6, Hacking Game Clients, really digs in and gets technical. As we move more deeply into games, we are forced to use a game or two as a particular target. We don’t do this to single out any one game; instead, we do this to make our examples salient and technical. We have chosen to concentrate on World of Warcraft (WoW), a game produced by Blizzard Entertainment, mostly because it is the number one online game in the world. The kinds of techniques we demonstrate using WoW as an example can be applied by analogy to almost any online game (and to modern Web 2.0 software, for that matter). Chapter 6 begins with a discussion of the attacker’s toolkit (many of the tools we describe are standard software testing tools). We then organize our discussion of game hacking techniques into four equally important areas: (1) getting over the game by using its user interface directly, (2) getting inside the game by manipulating memory, (3) getting under the game by interposing on services like video drivers, and (4) getting way outside the game by intercepting and manipulating network traffic. Chapter 6 is in some sense the heart of the book, introducing a large number of techniques commonly used by game attackers.

Chapter 6 has lots of data in it, probably too much. Sometimes it is easier to understand these kinds of techniques by seeing how they are put into practice. Toward that end, in Chapter 7, Building a Bot, we put together all of the lessons of Chapter 6. Chapter 7 is technical, with plenty of example code showing how the ideas in Chapter 6 work in concert to exploit a game.

Chapter 8, Reversing, is even more technical, focusing its attention on reverse engineering techniques that many attackers use to exploit software. Though the techniques we describe in this chapter are intense, they are by no means new. As we describe in our book Exploiting Software (Addison-Wesley, 2004), disassemblers and decompilers are used in computer security every day, both by good guys and by bad guys.

The final technical topic in our book is presented in Chapter 9, Advanced Game Hacking Fu. This chapter discusses what is known in the trade as total conversions and game mods. We describe the process by which some people take apart game data files in order to build their own games or combine different aspects of games in interesting ways. Though some game companies try hard to quash all discussion of total conversion, it happens anyway. We describe how.

Of course, our purpose in this book is to help those who build games understand how to do a better job with security. Chapter 10, Software Security Über Alles, provides a flyover of the new field of software security. Game developers would do well to adopt some of the best practices in common use in the financial vertical today. We also describe a set of questions that everyday gamers can ask their game companies about security. Our fervent hope is that this book will lead to more secure software—both in the game community and beyond.

The Software Security Series

This book is part of the Addison-Wesley Software Security Series of software security books for professional software developers. The series includes the following titles:

• Exploiting Online Games: Cheating Massively Distributed Systems
• Secure Programming with Static Analysis
• Software Security: Building Security In
• Rootkits: Subverting the Windows Kernel
• Exploiting Software: How to Break Code
• Building Secure Software: How to Avoid Security Problems the Right Way

More books in this series are planned for the future. Contact Addison-Wesley or Gary McGraw for more information. Also see the Web site.

This is the foreword that I wrote for Brian Chess and Jacob West’s excellent new book Secure Programming with Static Analysis. I recommend this book for all software security practitioners. Developers, in particular, will find the book extremely helpful. For more on other books in the software security series, see http://www.buildingsecurityin.com.

On the first day of class, mechanical engineers learn a critical lesson—pay attention and learn this stuff or the bridge you build could fall down. This lesson is most powerfully illustrated by a video of the Tacoma Narrows Bridge shaking itself to death. Figure 0-1 shows a 600-foot section of the bridge falling into the water in 1940. By contrast, on the first day of software engineering class, budding developers are taught that they can build anything that they can dream of. They usually start with “hello world.”

Figure 0-1: A 600-foot section of the Tacoma Narrows bridge crashes into Puget Sound as the bridge twists and torques itself to death. Mechanical engineers are warned early on that this can happen if they don’t practice good engineering.

An overly-optimistic approach to software development has certainly led to the creation of some mind-boggling stuff, but it has likewise allowed us to paint ourselves into the corner from a security perspective. Simply put, we neglected to think about what would happen to our software if it were intentionally and maliciously attacked.

Much of today’s software is so fragile that it barely functions properly when its environment is pristine and predictable. If the environment that our fragile software runs in turns out to be pugnacious and pernicious (as much of the Internet environment turns out to be) software fails spectacularly, splashing into the metaphorical Puget Sound.

The biggest problem in computer security today is that most systems aren’t constructed with security in mind. Reactive network technologies such as firewalls can help alleviate obvious script kiddie attacks on servers, but they do nothing to address the real security problem—bad software. If we want to solve the computer security problem, we need to do more to build secure software.

Software security is the practice of building software to be secure and function properly under malicious attack. This book is about one of software security’s most important practices—code review with a static analysis tool.

As practitioners become aware of software security’s importance, they are increasingly adopting and evolving a set of best practices to address the problem. Microsoft has carried out a noteworthy effort under its Trustworthy Computing Initiative. Many Cigital customers are in the midst of enterprise scale software security initiatives. Most approaches in practice today encompass training for developers, testers, and architects; analysis and auditing of software artifacts; and security engineering. There’s no substitute for working software security as deeply into the development process as possible and taking advantage of the engineering lessons software practitioners have learned over the years.

In my book Software Security, I introduce a set of seven best practices called touchpoints. Putting software security into practice requires making some changes to the way most organizations build software. The good news is that these changes don’t need to be fundamental, earth shattering, or cost prohibitive. In fact, adopting a straightforward set of engineering best practices, designed in such a way that security can be interleaved into existing development processes, is often all it takes.

Figure 0-2 specifies the software security touchpoints and shows how software practitioners can apply them to the various software artifacts produced during software development. This means understanding how to work security engineering into requirements, architecture, design, coding, testing, validation, measurement, and maintenance.

Figure 0-2: The software security touchpoints as introduced and fleshed out in Software Security: Building Security In.

Some touchpoints are by their very nature more powerful than others. Adopting the most powerful ones first is only prudent. The top two touchpoints are Code review with a static analysis tool, and Architectural risk analysis. This book is all about the first.

All software projects produce at least one artifact—code. This fact moves code review to the number one slot on our list. At the code level, the focus is on implementation bugs, especially those that static analysis tools that scan source code for common vulnerabilities can discover. Several tools vendors now address this space, including Fortify Software the company that Brian and Jacob work for.

Implementation bugs are both numerous and common (just like real bugs in the Virginia countryside) and include nasty creatures like the notorious buffer overflow, which owes its existence to the use (or misuse) of vulnerable APIs (e.g., gets(), strcpy(), and so on in C). Code review processes, both manual and (even more important) automated with a static analysis tool, attempt to identify security bugs prior to the software’s release.

Of course, no single technique is a silver bullet. Code review is a necessary but not sufficient practice for achieving secure software. Security bugs (especially in C and C++) are a real problem, but architectural flaws are just as big a problem. Doing code review alone is an extremely useful activity, but given that this kind of review can only identify bugs, the best a code review can uncover is around 50% of the security problems. Architectural problems are very difficult (and mostly impossible) to find by staring at code. This is especially true for modern systems made of hundreds of thousands of lines of code. A comprehensive approach to software security involves holistically combining both code review and architectural analysis.

By its very nature, code review requires knowledge of code. An infosec practitioner with little experience writing and compiling software is going to be of little use during a code review. The code review step is best left in the hands of the members of the development organization, especially if they are armed with a modern source code analysis tool. With the exception of information security people who are highly experienced in programming languages and code-level vulnerability resolution, there is no natural fit for network security expertise during the code review phase. This may come as a great surprise to those organizations currently attempting to impose software security on their enterprises through the infosec division. Even though the idea of security enforcement is solid, making enforcement at the code level successful when it comes to code review requires real hands-on experience with code.

The problem is that most developers have little idea what bugs to look for, or what to do about bugs if they do happen find them. That’s where this book, Secure Programming with Static Analysis, comes in. The book that you have in your hands is the most advanced work on static analysis and code review for security ever released. It teaches you not only what the bugs are (what I sometimes call the “bug parade” approach to software security), but how to find them with modern static analysis tools, and more importantly what to do to correct them. By putting the lessons in this book into practice, you can go a long way toward helping to solve the software security problem.

You can purchase a copy of Secure Programming with Static Analysis from Amazon.

Technorati Tags: touchpoints, software security, static anlaysis, Brian Chess, Jacob West

(This is part three of a three part series. Part one is here and part two is here.)

G.I. Joe and his friends (e.g., http://en.wikipedia.org/wiki/G.I._Joe) gave cartoon addicts in the mid-1980s United States a series of television public service announcements. The cartoon kids being given the lesson invariably responded with “Now we know!” to which the G.I. response was always “And knowing is half the battle.” Unfortunately, we often forget that knowing is, in fact, only half the battle.

So, let’s you say have some awesome training materials (slides, eLearning module, video, or whatever) and training day goes beautifully. Everyone now knows exactly what they’re supposed to do. How are you going to get them to do it? Periodic and direct reinforcement are key to real behavior modification.

Start by ensuring that reinforcement messages are pitched at the right level. If you want people to learn the theme, then repeat the theme, act out the theme, and make sure the theme is in their words. Don’t start with “It was the best of times, it was the worst of times” to explain the quarterly software bug report trends to executives (although that would be pretty cool) and don’t tell testers to ensure that the “trend in bug reports must have a positive impact on EBITDA.” For example, tell people who work with software bugs about bugs, and their bosses about trends in bugs, and their bosses about trends in application quality, and their bosses about trends in the application portfolio, and so on.

Don’t start from nowhere with your students. I believe it will be somewhat uncommon for you to be teaching people something entirely new (imagine the blank stares in the first calculus class sometime around 1704 or the first Fortran class in 1954!). Even after training, continue to work diligently to identify the gap between what the students are doing and what you need them to do. Then, use that as a stepping stone to build upon your audience’s existing knowledge and skill. Continue to remind them of what they already know and explain to them what you want them to do that’s extra or different. If you make it sound like they must start from scratch, or make them cover old ground, or if you omit some key knowledge, you’re unlikely to get any positive behavior change at all.

Make sure your learners continue to be motivated to learn and change. Make it part of their objectives. If you took the time to train someone on how to do something, you should check to see that it’s getting done. If the students know you’re going to check, they’ll be a little more motivated to learn it and do it. There are few activities more frustrating than standing in front of a class all day, teaching important stuff that the students really need to know, and having them stare vacantly ahead (remember, you can make them put away their PDAs, but you can’t make them like it). A little positive motivation goes a long way.

Unfortunately, so does a little negative motivation. Just as organizations get peeved when some new law generates an unfunded mandate that’s all stick and no carrot, employees get a little perturbed when any kind of “do it or else” message is sent their way. While this kind of motivation may work in the short term, it’s just not worth it. In many organizations, the employees are the company. For consultancies, independent software vendors, application houses, and so on, the biggest corporate asset—the employees—go home every night. In a job market that offers some easy mobility, don’t give them an unnecessary reason to go somewhere else.

The best way I’ve found to ensure ongoing performance is to institutionalize the activity as the way things are done. Don’t make people have to go out of their way to do the right thing. If you’re training software architects to do threat models and architecture risk assessments, then account for that time in development projects, give them the tools they need, and make it’s part of the software development lifecycle. Ensure there’s not only prescriptive guidance for doing it, but that the results of these activities are part of someone’s metrics and the software governance dashboard. It’s the same for teaching developers to improve their development patterns, teaching testers to perform non-functional security tests, or teaching executives how to institute software governance.

Let me mention metrics once again. If you took the time to train someone how to do it, then “it” should probably be part of the metrics that someone tracks. If you taught a roomful of testers how to do some inside→out white-box security testing, then you should probably keep track of how many software flaws they catch as part of this process, along with some trends. You might even want to, for example, track which code bases are exhibiting the most problems and track that back to the composition, training, and automation support for the various development teams. If you can discover which teams are routinely making similar errors due to, say, having a broken unit testing process, everyone wins.

And, by all means, find and encourage the stars. The people who really get it should be groomed appropriately. They may be the next crop of team leaders or managers. If nothing else, get one of those individuals to improve the course material or even to teach the next class.

I certainly don’t mean that you should ignore the stragglers. If some folks have attended training and have received some follow-up attention and still don’t get it, they may simply be in the wrong profession. That’s always a tough decision, but I find it’s usually best for everyone. If someone has arrived at a position where they are failing, even with training and attention, kindly help them find a place where they can be effective. And I do mean kindly; that person may be your boss someday.

Someone must follow up with the instructors as well. Do post mortems after each class and give constructive feedback. Ensure each student completes a feedback form. And don’t ask those silly questions on the form like “Did you enjoy this class?”; that’s just useless. Ask things like “Did you understand why you were here and the problem we’re addressing?” and something like “Will the information given to you today actually improve your day-to-day performance?”. Of course, you also want to ask things such as “Did the instructor communicate in a clear manner?” and “Do you feel this instructor truly understands the material?” and stuff like that.

As some parting advice, keep in mind that it’s not just the individual student that must get something out of the class. For every course you give, someone is paying a hefty bill. Whether you’re a consultant charging by the hour or an internal expert delivering some training to your friends, someone’s paying the price for you and for those students to not be working. That economic buyer needs to get something for spending that money (real money, opportunity cost, or whatever) if you expect it to happen again. Ensure someone comes up with a way to measure training success, that they actually measure it, and that they let the stakeholders know how it went.

“Now we know!”

(This is part two of a three part series. Part one is here and part three will be posted on Friday.)

“Keep away from people who try to belittle your ambitions. Small people always do that, but the really great make you feel that you, too, can become great.” - Mark Twain

My guess is that, on any given day, thousands in the world-wide work force attend some form of corporate training. It could be software governance training for the pointy-haired bosses, how to use a juicer for department store workers, or keeping fingers away from the business end of a chainsaw for lumberjacks, but it’s training nonetheless.

The training might be delivered in the form of a short video, some eLearning module, or a class led by an instructor. It may have an integrated quiz, or require you to go do some additional exercises, or be taught one day per week for some period of time, or whatever. The possibilities aren’t endless, but there are a fair number of them.

Beyond the training requirements discussed previously, you also need to figure out what training styles works for your people in your environment. Having said that, I think there are a few rules that are fairly universal:

• Be as formal in your presentation method as you need to and then just a touch more. I find that too little formality in training is far worse than too much. Too little formality is often perceived by students as “We had to do this training to check the box, so let’s get through it.”
• Stick to your schedule. If it says “Start at 9am,” then start at 9am. If you’re breaking for 10 minutes, get started in 10 minutes. To some degree, we all want to wait for that last straggler. Don’t do it; you’re just penalizing everyone else who had the professionalism to be punctual. Class management is always hard. You may in fact be the tech dweeb presenting to a room full of executives. On that day and at that time, however, you are in charge of that class; act like it. Professionally. Doing it unprofessionally is definitely a career-limiting move.
• Use eLearning (or computer-based training or whatever you call it) for awareness and to give basic skills to the masses. It’s very important, however, to use instructor-led training (ILT) for teaching important execution skills to your practitioners. It doesn’t matter how many times you watch the video of the expert performing the trick, even in slow motion with arrows and explanations, most people still won’t learn a new skill that way.
• Make sure you can actually teach the material in the time allotted. That usually means you must ban the typical distractions. That includes Blackberries, laptops, various PDAs, and so on. Yes, even for the executives. Stop the side conversations, keep everyone on topic, and tell people who have long, rambling, just-don’t-get-it questions that you’ll get back to them after class. When you rush through the last 20% of the material to stay on schedule, students feel cheated. On the other hand, if you scheduled two hours for a thirty minute class, that’s a completely different screw up and your students will likely be just as irritated.
• Give people sufficient breaks so that they don’t have to violate your other rules. I find that most people, including the instructor, can concentrate only for about 90 minutes at a time. For really dense stuff, that time might be reduced significantly so partition your material appropriately. Also, ensure you provide what people typically want during breaks: snacks, beverages, easy access to a restroom, and a network connection. I imagine millions of person-minutes of training are lost every year just because someone didn’t think of spending $20 to set out some mid-morning and mid-afternoon snacks.

I argue with myself, sometimes successfully and sometimes not, over whether the material or the instructor is the most important part of technical training. Today, I believe that it’s far more likely that a good instructor can save truly awful material than it is likely that a bad instructor will successfully deliver even the best of material. If you’re good at getting a class excited about a topic and you can talk about the topic authoritatively, you’ll rise above the material.

Let me emphasize that more. If you’re the instructor, you must know more than what’s in the training materials. If you can’t go off-script even a little, you’re the wrong person for the job. Excuse yourself graciously rather than wasting everyone’s time. The amount by which you need to go off-script will depend on the skill level of your students, but you certainly should be able to answer questions about rationale, goals, why another way isn’t the right answer, and how it would be done in a scenario somewhat different from the one presented.

When going off-script, it would be great if you can do so in three ways:

• Visual stuff, for the people who learn from what they see. For these people, you must be able to provide visual aids. Draw a picture on a white board, or make a hand-out, or have an exercise where people draw out the process, or whatever. Always give the visual people something to imprint on.
• Auditory stuff, for the people who learn from what they hear. For these people, it’s not just what you say, but it’s also how you say it. They’ll pick up on subtle clues in your delivery that can sabotage your purpose. In other words, be able to be excited about digging deeper. If you think Section 4 is boring, you’ll present it that way, and they’ll believe it. Be interested, not just interesting, and give the auditory people something to pick up on.
• Kinesthetic stuff, for the people who learn from what they feel, physically or emotionally. For these people, you have to get them involved in the problem and the solution. For some, that will be an emotional connection and for others that will be some manner of hands-on demo. Whenever possible, let the kinesthetic folks touch something.

Good speakers always have good notes from which to speak. Don’t be afraid to have a cheat sheet of important points. And don’t be afraid to lecture. Just don’t do it the whole time. Give people the motive, the rationale, the philosophy, or whatever it takes to get them involved in the problem. Then switch to some practical visual, auditory, or kinesthetic stuff to show them how to execute the solution.

Now, I’m not talking about any of those group hug things, because there are also introverts and extroverts in the world. Making an introverted, visually-oriented person do role playing as part of training is just torture. They’ll hate you forever. On the other hand, not only are the extroverted kinesthetic folks excited at the prospect, they’ll show up with streamers and brownies and hold annual reunions. Sigh.

When you’re presenting technical training material, whether as part of ILT or an eLearning quiz, make sure you ask some open-ended questions. This is a great opportunity to let the students use their own words to tell you what they’re thinking. By listening carefully, you’ll be able to tell who really gets it and who struggling to keep up. That will help out a lot in an ILT setting when you’re making teams for group exercises. (Hint: Don’t let all those struggling students end up in one group.)

Training creators and instructors must remember that there will be students with whom you have no shared experiences. They don’t know Babe Ruth, don’t get your Simpson’s references, and are quite alarmed at your claims that sometimes you just need to “kick some ass,” “gore some oxen,” or “kill some sacred cows” to get what you want. This is not my soapbox for political correctness; this is just my experience telling you that teaching via analogy, colloquialism, and local cultural reference often results in a lot of blank stares (like with my upcoming G.I. Joe reference). Life may be like “a box of chocolates” for millions of people, but there may be billions who’ve never even seen a box of chocolates, much less a Tom Hanks movie.

Take the time to tell students what you want them to do when they leave the class. Don’t just talk around it. Really lay out the activities they should undertake and the changes that should occur when they do. Think of it like writing a software requirement. You can write “The system shall…” all day long, but until you actually show the person who has to do it a use case or a scenario or a drawing or something, there’s virtually no chance of getting the behavior (or code or process or whatever) you intended.

Remember, our brain is likely to simply reject any message that it can’t process. Why might that processing not happen? In my experience, the top causes are:

• For new skills, you’ve provided no frame of reference for the student. There no place to latch onto and say “Oh, this is like that other thing I know how to do, only different.”
• For extensions to existing skills, there’s too big a gap between the student’s skill and what you’re teaching, and you’re not providing enough framework for the student to make it over. Either you jumped right into the guts of the material without adequate preamble or someone did a poor job of choosing the students.
• One or more of the material, the presentation method, or the instructor is simply lousy. If the materials seem good and the method is well thought-out and constructed, then it’s probably the instructor. It’s impossible to describe all the ways training material can be lousy, but one of the most prevalent is that it’s just stale. Teaching the same old stuff year after year without ever updating it is just silly. As for the instructor, not everyone is cut out for teaching. I’ve seen some real world-class technical experts do indescribably horrible jobs in front of a class, just as I’ve seen non-practitioners (never actually did this work for money in their lives, but truly understand the topic) teach wonderful classes.

By the way, don’t send the wrong subliminal message with your training. If you’re having a session on the importance of corporate intellectual property, ethics, and how file sharing is wrong, you probably shouldn’t sprinkle some “borrowed” Dilbert, Calvin and Hobbes, and Far Side cartoons in your presentation, while playing a downloaded Van Halen soundtrack in the background. Duh.

Trust me, if you really care about the training you’re giving, as well as the behavior you’re trying to encourage, it will show in the material and in your delivery. This energy is contagious. Chances are that you want these students to stretch themselves in ways that are new and possibly even undesired. Give them every opportunity to succeed. If the training appeals to their senses, they’ll be more engaged and they’ll retain more. If the training appeals to their sensibilities, they’ll be more engaged and they’ll actually follow through.

(This is part one of a three part series. Parts two and three will appear on Wednesday and Friday.)

The beautiful thing about learning is that no one can take it away from you. -B. B. King

We’ve all been to training. More specifically, we’ve all been in training where we were bored to tears, not because we were already experts, but because there was just no connection between our individual learning style, base knowledge, interest level, or job goals and whatever it was the instructor was doing. Invariably, however, there will be a few people in the same class who feel this training was the best they’ve ever attended. Two days later, you can’t even remember what the course was about and two months later others are still quoting the instructor. Why does this seem to happen so often? In my experience, that it was effective even for those few people was probably just luck.

When you’re thinking of making or commissioning some technical training, stop. Completely. Not one of those rolling stops you do at a stop sign in the middle of nowhere. Sit down and take a deep breath. Now, write three complete sentences:

• “This training is appropriate for people who…”
• “In this training, you will learn how to…”
• “After this training, you will be able to…”

If you cannot specifically and succinctly define your audience, tell them what they will learn, and describe how their new ability will benefit the organization, you’re definitely not ready to start building training. This would be just like writing code without requirements and you’re likely to achieve the same spectacular lack of success.

If you need to do a formal performance needs assessment to get requirements because you’re teaching someone a life-or-death skill (e.g., how to disarm landmines in the field), then do it. On the other hand, if you’re teaching employees the basics of physical security for the average office building, you can probably just survey the existing literature to get your main points.

I understand that there is some training that must be given to everyone regardless of their interest, skill level, learning style, or anything else. Examples might include employee orientation training, sexual harassment training, security procedures training, and so on. I feel it’s easy to show how poorly the average one-size-fits-all training approach has worked in these areas, especially when reduced to a poorly-scripted eLearning module or abysmally-acted short film.

Even after you get a good course description and, presumably, the right audience, tell people right up front what they’re going to learn in your course. Don’t make it something cheesy like “Today we’re going to learn about software security.” That’s just silly, even if it’s for a general software security course. Tell them something useful like “Today, I’m going to show you how to generate crypto seed values correctly in various languages and technology stacks. By doing this correctly, you will help the company meet ongoing regulatory compliance requirements.” If it is the general course, tell them something like “Today, I’m going to give 10 reasons to care about software security and help save millions of dollars.”

Well, that’s different. Now I know why I’m here and how what I’m going to learn will really help; it’s not just some flavor-of-the-month exercise because it’s “time to shake things up around here.” Continually reinforcing why this training is important, both during the class and afterwards, means you’ll end up not only with increased competence in the target audience, but, in this case, you’ll also end up with practitioners who are more business-risk aware. That’s a good thing. Why? Because now, they’ll ask themselves questions about “How does this affect the business?” and, when they don’t know the answer, they’ll go ask someone rather than just putting some code together. Give them the chance to do the right thing.

Remember, it is a significant task to get a diverse group of people to leave a room with the basics of a new skill (e.g., developers using certificates for mutual application authentication), with necessary knowledge (rand() is not cryptographically secure), or even with just a common awareness of a given problem (e.g., web application input validation is tricky). This is compounded when we let training creation wait until the last minute and then “throw something together” or we “just use last year’s slides.”

The biggest evidence of such procrastination and lack of planning is ending up with training materials (e.g., PowerPoint slides) that are simply about the topic, not on how to do the topic. How often have you gone to a class guaranteeing to turn you into a practitioner, only to spend the majority of the time on vocabulary and concepts and fluffy cloud process diagrams! What a waste of time. Training about test process automation, for example, is something that you give to people making budgeting and resourcing decisions. Training detailing a method for doing test process automation is what you deliver to practitioners.

Speaking of planning, in my experience you have to budget about 100 minutes per technical information slide to create useful material that focuses on imparting a single important idea in a memorable fashion. This includes sufficient time for instructor and student notes. The instructor notes should be what the instructor says while the slide is being projected, along with any prompts for what to do, what to hand out, and so on. The student notes should include specific reinforcements, references, and related materials applicable to the information on the slide. This 100-minute block also includes internal review, discussion, and bootstrapping the first instructor.

I know 100 minutes per slide sounds like a lot, but if you’re spending less than three days to plan, draft, and complete one hour (about 15-20 information slides) of technical training (or any training that is telling someone how to do something), you’re probably messing it up. In particular, take the time to look for every group of words, bullets, or slides that you can turn into a real, accurate picture. Decorate the picture appropriately, but don’t make it so busy that it becomes an eye chart. Animate it whenever necessary to show how and when things actually happen. People will remember and refer to these pictures (because they can cut and paste them), but they probably won’t ever reuse some random set of bullet points.

This reuse habit, by the way, is also why it is universally idiotic to include on-the-fly sample code that someone just banged out for the slide; it’s almost certainly wrong, not secure, and against some policy and someone will be embedding it in some critical code module tomorrow. If they do that, it’s just as much your fault as it is theirs.

Another word of advice: Don’t have technical experts create the final training product. No, really. Even if you’re the technical expert and you think you know better. Trust me. The only exception to this might be when a technical expert is creating slides for very similar technical experts. Even then, run the slides by someone a level or two below and a level or two above “technical expert,” as well as someone more skilled in training presentation, and heed their feedback. It’ll be a valuable experience and you’ll end up with better material that is less bogged down with jargon, with incomplete thoughts that presume too much historical knowledge, with acronyms that are never explained, and with words that apparently have a different definition on the author’s home planet. One hopes you’ll also end up with something that isn’t 100 slides of wall-to-wall words.

The long-term benefits derived from putting the required time and skill into building good training will far exceed the initial investment. Spend a little extra time on it and make your students happy to be there. Everyone will benefit.

Technorati Tags: training

Benjamin Tomhave wrote about “tiering” security guidance when I cross-posted a comment to my last blog entry on the SC-L mailing list. Quoting him:

The higher up you are in the policy framework, the more general and time-enduring the content should be. The farther you progress down the framework to a more detailed level, the more perishable the content will be, out of necessity.

Later he continues:

…is because implementers need it. They’re not security experts (usually) and do not necessarily grok security the same way a seasoned (salty?) security person might.because “implementers need it”.

This tiering was implicit to my first post. In fact your most senior security resources can probably use nothing but Security Principals (as described by McGraw’s BSS Book and the famous Saltzer paper) and find both insidious vulnerabilities as well as brand-new “Game over” architectural flaws with new development technologies they aren’t familiar with. But, the more junior (inexperienced) or development-oriented (constructive) the person being targeted, the more specific the guidance must be in order to be valuable without requiring inordinate effort.

Because we’re trying to change the behavior of the majority of our Developers–who range in skill from OK to Hero and whom may have never had even a security awareness class–I find “technology-specific” guidance moves the ball the furthest.

In my previous two posts I talk about forms various levels of standards take, and the way in which one might create it. It occurs to me that I all but showed the bigger picture and might as well follow up to do so. Below, you’ll find a map of how I show security guidance flowing throw and effecting a software development team (click-through for full detail):

As information moves from top to bottom and from left to right it becomes more specific and actionable, but also more perishable (as has been said). To build security in, one must think about security’s implications throughout the lifecycle, so I see no reason why security knowledge (regardless of how specific) shouldn’t mirror artifacts used to construct the application itself: software requirements, design, and the code itself.

Though not central to this discussion, the diagram has been annotated to indicate who should produce and consume this information. Here, I’ll point out that your centralized Application Security Resources can probably most effectively and efficiently create the generic security guidance, but will need help of Security Architects to create the more technology-specific guidance and garner broad buy-in.

My last post presented a brief model of how one might organize and fund this in practice.
-jOHN

Technorati Tags: knowledge architecture, software security

Software security veterans have all certainly thought about the idea of ‘securing the SDLC’… I can tell because every consulting firm’s collateral that I’ve seen in the past year has a new bullet under their ‘services’ section referring to something like ‘Secure development process integration’ or ‘Secure SDLC services’. That being said, let’s talk about what this means for a second. Fundamentally, there are a few ‘different’ schools of thought out there (and as it’ll turn out, they’re not all that different at all).

I know of three popular ways of looking at the problem, 1) Microsoft SDL 3.0 (with a recent book by Howard and Lipner to codify the subject), 2) Software Security Touchpoints from Gary’s book Software Security, and 3) CLASP (originally developed by Secure Software, Inc, and now an open project through OWASP). BTW, if anyone knows of other publicly usable process methodologies, by all means email me since I’d love to read about them.

After spending a bit of time thinking through all these different ideas, a few interesting points emerge. First, there’s not much difference between SDL, the Touchpoints, and CLASP. There’s just about nothing I can see where these processes fundamentally disagree. The differences are really only in the timing and the extent of the prescribed activities (i.e. they each cover the bases of what you should be doing, some just give different orderings to the activities and talk about the sub-steps in different ways). My personal opinion is that SDL is particularly suited for companies like MS (large ISVs with large user populations) and process like the Touchpoints and CLASP are a bit more flexible and widely applicable.

So what’s the deal? Do we have the problem of dev process augmentation solved and put to bed? Heck no. Consider the following quote that popped up in a discussion my buddy Gunnar Peterson and I had at the recent OWASP conference in Milan: “Amateurs talk about tactics, debutants talk about strategy, but professionals talk about logistics.” (this quote has many variations and is hard to find a definitive source, but it’s likely from a US military officer many years ago). As the software security space was emerging, you bet we had to crawl from the primordial ooze by figuring out some tactics to stop the bleeding. Logically following, lots of smart folks sat down and figured out the right way (via experimentation, mostly) to look at the problem from a high-level. Hence, strategy for software security was born. Now, the proverbial last mile is the logistics of how you get the job done within an organization that’s got 50,000 real-world constraints that complicate everything.

Regardless of your favorite security-enhanced SDLC method, you’ll notice that they really are, at their core, a collection of activities, procedures and artifacts (tactics). Don’t get me wrong, it’s great stuff in terms of what’s needed to do the job well and it’s generally assembled and presented in a full-blown, whole-hog, flying-car way (strategy). If you’re in the shoes of the person in charge of augmenting your company’s dev processes, you’re handed a large collection of great things to think about, but little that’s directly actionable in terms of answering ‘what do I do tomorrow?’ (logistics).

What I’m getting at is that I think we’ve gotten to the point where if you’re still debating tactics of what to do or the strategic vision of what needs done for process integration, you’re solving the wrong problem. It’s about rubber-to-the-road logistics. We need to build on the work that’s been done already and come up with plans that make it accessible and usable for an average human that hasn’t made a career on thinking about these things. That’s a serious challenge, but not an impossible one. At Cigital, that’s what our SDLC process gigs are all about (providing the company a detailed plan of how to get it done). What’s needed now is to get a more abstract way of looking at the various factors that contribute to logistical differences (e.g. type of business, market vertical, organizational hierarchy, regulatory constraints, etc.). I strongly believe that we can formalize these factors and I think that goes a long way to breaking the back of the problem. I fact, I’ve been working with folks in the OWASP community on this very problem (and would love to get anyone else with field experience involved). Much of that work will be released in a new version of CLASP in the next week or so, so stay tuned if you’re interested (I’ll post another entry here announcing it).

Technorati Tags: software security, sdlc

The process of writing security guidance is just as important to the quality of the resulting standards as is the target: technology-specific, code-centric constructive statements. How do you succeed? By using the same muscles you exercise when you conduct secure design.

When I write Security guidance, such as the technology-specific standards I blogged about last week, I start with a threat model for the system (or design paradigm) I’m trying to protect. This could be as specific as a single system or as generic as all my client’s customer-facing 3-tier J EE Apps. I aim to remove attack vectors with sets of standards. I don’t write standards that don’t directly prevent a particular attack I’ve already documented in the same way Agile Developers ignore abstraction not mandated by the current user stories. Eventually though, removing a subsequent attack vector will require a smidge of refactoring or (more likely) augmentation.

Thinking about each attack vector individually gives necessary focus to the process and will help get past hapless statements of context-free specificity. Years ago I saw, “All data must be protected with 128-bit encryption”, within a customer’s standards. Admirable though the specificity was, the limitations of this as a standard become immediately obvious. Would 128-bit encryption suffice for archival-grade needs—such as log files? What does the plain-text or cipher-text data get used for? If we’re looking at attacks on session management within our imaginary web-app, simply encrypting a session token doesn’t effectively protect against theft and replay does it? Remember last week’s entry. There, in my example, I sought to protect against the circumstance in which (due to omission or later maintenance) a Developer forgot to explicitly demand authentication prior to accessing a piece of functionality.

Think of security standards as requirements driving a design that resists the attacks you outlined in your model. This focuses authoring and often alleviates the writer’s block a blank page can impose. Improving and completing guidance becomes tractable. In my crypto example above, the client was attempting to secure exchange of data between two systems (thick Java clients) over the Internet in a circumstance in which they could not rely on SSL being present. Data from each transition persisted beyond a single session, but had a short span of value—say a day, or week (depending on the volatility present in pricing). You’re probably already conjuring ideas about what you’d expect to see in a secure implementation. With as little context as this, my explanation of the design begins to illuminate attack vectors. From these, you can begin to answer the questions I posed above about ciphers, structure of data, and from there secure design becomes an exercise in trading off difficulties of key maintenance, performance/overhead of the encryption, and others with resisting the attacks you’d enumerate.

Not shockingly, HOW you build this guidance is as important as ending up in courier font. And, (again) it shouldn’t surprise you that I’ve chosen to get end up with code (courier) through design (via threat modeling). This is how software is built—and security should be no different. This does mean you’ll need security architects (who can code—not the corporate librarian types) to write good, detailed security standards though.

Technorati Tags: software security