Sylvan: For the cases where you want users to be able to submit ‘rich content’, I generally recommend NOT allowing the users to do so using HTML. It’s far cleaner to allow them to input their content with some kind of markup language and then have the application parse and validate according to the markup language on input, and translate appropriately on output (to HTML, for instance, by HTML encoding everything that wasn’t a markup control sequence for, say, links, images, etc.)

Edward: I agree with your three step process, with one minor exception… For the second step where you store the data in a database, don’t bother with manually implementing SQL escaping. Instead, use a properly formed parameterized query (prepared statement) and have the underlying framework take care of it for you. That way, there’s a smaller chance of omission or error, not to mention cleaner code that’s easy to audit for compliance.

Rod: I think you misunderstood my comments related to the HTML/URL encoding. I was talking about using those encoding mechanisms as a part of input validation, which is absolutely a misguided strategy. Generally, people do this to avoid having to think about the various types of outputs that their application makes (HTML responses, log messages, SQL queries, etc.), thus it’s indicative of having not thought through the problem.
As for a priori knowledge of output control characters, what I meant was that in a scenario where you still decided to encode on input, you would need to know all the ways in which the app (and database) would eventually use the data and then pick an encoding scheme that was universally safe. This is very hard for any app that’s built to be extensible (web services) since it can’t be known a priori all the ways in which caller’s might use the data. It’s much simpler for all data consumers (including your own app) to treat all the data as potentially hazardous and always encode appropriately on output to ensure whatever it was, it will do no harm. If the data was already encoded before the caller gets it, they’d either a) use it without further through, potentially leading to vulns (obviously bad for security), or b) be forced to unencode and then re-encode appropriate to how they’d like to use it (bad for maintainability and simplicity). Using the validate-on-input/store-pure-data/encode-on-output paradigm, the only thing you need to know a priori is all the ways in which your application will output data and assign an encoding scheme for each. The contract with other callers is clean: don’t trust that the data is safe for any output channel.
As for input validation itself, I did state in the last para (if a little cryptically) that people should do it because it is good for usability and can prevent impact of other programming mistakes. Perhaps it’s just a terminology difference, but I was using the term “input validation” for validation that occurs at the application scope (external data entrypoints and such). If an internal function in your program has a buffer that can be overflowed and it’s relying on entrypoint input validation to keep it safe, it’s a poorly designed function. Keeping a dependency tree of which fragile internal functions are protected by given entrypoint validation methods is error-prone and terrible for maintainability/extensibility. It’s better to design internal objects in such a way that they cannot, under any input/usage, lead to security problems. I suppose that’s a kind of input validation in itself, but like I said, it might be a terminology difference.

As a final note, to everyone, thanks very much for taking the time to leave feedback. I think we’re all more in agreement than disagreement ;)

You wrote:

“augment the send_message() function with logic to zip through the pending message and escape anything that was a control code and then you’d do the normal stuff of writing it to the wire.”

and

“You’ve gotta have output encoding to truly solve it right.”

then you objected to “HTML/URL encod[ing]” which I agree in context you meant using a perfectly valid HTML or URL “output” encoding as an improper “output” encoding for the database.

If your output is going to be sent to HTML the HTML encoding is the correct output encoding, and if is being sent as data on a URL the URL encoding is correct. Those are however, not the correct output encodings if it is being sent to a database.

More to the point; is your function which “zip[s] through the pending message and escape[s] anything that was a control code [before] writing it to the wire;” not in fact a priori? You must know the control characters or how to recognize them in advance of escaping them to perform the specific output to that channel of communication.

At the point where input is received, you must encode or escape it with knowledge of where it is going so it is still a priori.

Furthermore, as Edward Z. Yang correctly pointed out, it is ridiculous to accept characters outside the scope, range or domain of the expected input. To do nothing opens the opportunity for other attacks where perfectly valid data creates an undesired fault, such as a buffer overflow.

You need both input and output operations.

While I don’t disagree with the assertion that: “We need to ensure that architects & developers have a deeper understanding of what the problem really is in order for them to naturally build systems to these types of attacks.”

It would be far better to modify the underlying architecture to make it impossible (or very difficult) to output to any particular “channel” potentially tainted data.

But your argument does not lead me to conclude that validating input warrants the B.S. card. It is indeed the first, but not the only step towards robust security.

When you see someone encoding data for a projected output format (not the one immediately relevant, such as the database), it’s usually an ill-advised optimization. The sub-conscious rationale is if you encode it in-bound, you don’t have to worry about it when it’s out-bound. It can be a perfectly reasonable optimization, especially if encoding the data in a safe manner is particularly resource intensive (the canonic case is user-submitted HTML in HTML, for which I’ve written a library for). But if you’re encoding willy-nilly without understanding the ramifications, you get problems. The perfect example is PHP’s magic_quotes. The functionality “encodes” your data so that it can be safely included in SQL statements. There are two problems with this approach:

1. It assumes that the data is actually going to be used in an SQL context (which, many times, it’s not), and
2. It performs the encoding improperly!

In summary, the proper sequence of events is:

1. Inbound: validate the data
2. Store the pure, un-encoded/un-escaped data (to do this, you’ll probably need to perform SQL escaping on it to store in a database)
3. Output: retrieve the data, and transform it into a form safe for the output format (usually HTML)

If three proves to be too costly to be done on each page request, a cache of the pre-encoded data can be added.

You might want to keep in mind that for sites like MySpace, input validation, manipulation, and rejection are the only way to really deal with their stuff, but how many websites do you really want for the user to be in control of HTML? The cases where output filtering does a bad thing is a *really* small set, but the approach by many other security experts is to treat that case as the norm.