The PCAOB is introducing new guidance to help lower the overall cost and, presumably, increase the effectiveness of SOX 404 audits. It needs to use this opportunity to help fix some root causes, not just tell us how to find more symptoms.

This past December, the PCAOB announced that it would propose for public comment a “new standard on auditing internal control…designed to focus the auditor on the most important matters, increasing the likelihood that material weaknesses will be found…” The proposal itself can be found at http://www.pcaobus.org/Rules/Docket_021/2006-12-19_Release_No._2006-007.pdf.

Starting on page 93 of the document, there is a section on “Benchmarking of Automated Controls.” It includes guidance like “Entirely automated application controls are generally not subject to breakdowns due to human failure,” which is clearly not true since bad input makes functional application security controls fail all the time. Telling auditors that automated application controls automatically get a gold star is not a step in the right direction.

It does go on to suggest that the benchmarking strategy take into account the importance of the effect of related files, tables, data, and parameters on the consistent and effective functioning of the automated application. That’s a good thing.

The document then suggests that “If general controls over program changes, access to programs, and computer operations are effective and continue to be tested, and if the auditor verifies that the automated application control has not changed since the auditor established a baseline (i.e., last tested the application control), the auditor may conclude that the automated application control continues to be effective without repeating the prior year’s specific tests of the operation of the automated application control.”

So, never mind that new attacks are discovered with unnerving frequency and that perfectly good code can suddenly become not so great (think crypto algorithms), the apparent recommendation is that if you didn’t or couldn’t poke hard enough to break it over some time period, then it’s okay to skip it later. Approaches like this where we’re considering only functional changes and not testing skill and depth can’t be effective.

How many times has someone walked up and spotted a problem you failed to notice time and time again. As organizations periodically change auditing firms, expect huge increases in reported problems.

The proposed guidance gives the following factors to use when deciding to use benchmarking:

• The extent to which the application control can be matched to a defined program within an application;
• The extent to which the application is stable (i.e., there are few changes from period to period); and
• The availability and reliability of a report of the compilation dates of the programs placed in production. (This information may be used as evidence that controls within the program have not changed.)

This wording is still neglecting changes on the threat side of the equation. Just as many castles were considered impregnable until about five seconds before the first cannonball hit, many lines of code were considered secure right up to the point where the breach story appeared in the newspaper.

The guidance gives the following factors to use when deciding whether to reestablish the benchmarking baseline:

• The effectiveness of the IT control environment, including controls over application and system software acquisition and maintenance, access controls and computer operations;
• The auditor’s understanding of the nature of changes, if any, on the specific programs that contain the controls;
• The nature and timing of other related tests;
• The consequences of errors associated with the application control that was benchmarked; and
• Whether the control is sensitive to other business factors that may have changed. For example, an automated control may have been designed with the assumption that only positive amounts will exist in a file. Such a control would no longer be effective if negative amounts (credits) begin to be posted to the account.

Okay, so if I know that most auditors and organizations use COSO as a governance model, and COBIT 4 to interpret COSO to arrive some IT control objectives, and I consider AI2, Acquire and maintain application software, important to my environment, then I might understand what “controls over application and system software acquisition and maintenance” means above. And, I might even read COBIT and see AI2.4, Application Security and Availability, which states “Address application security and availability requirements in response to identified risks, in line with data classification, the organization’s information security architecture and risk profile. Issues to consider include access rights and privilege management, protection of sensitive information at all stages, authentication and transaction integrity, and automatic recovery.” Oh wait, I only need security software, not software security. Sigh. Another opportunity missed.

There is a need here for any words that introduce something like the following:

The customer may have written their own software that is directly used in the financial reporting process and you, the auditor, should be aware not only of the software’s functional controls (e.g., I&A, encryption, entitlements), but you must also accrue confidence that their internal development practices and testing are sufficient to produce quality software that has at least some capability to protect itself from attack even in the event of catastrophic failures other general and IT security controls being considered as part of this SOX audit. While security software may comprise the majority of IT security controls, software security is the property that gives us confidence in their continued successful operation even in the certainty of ongoing attack. You must use a risk-based approach to accruing confidence, focusing on relevant factors that have a material effect on the software’s ability to meet business objectives even in an overtly hostile environment.

Now we’re thinking more about the problem and less about the symptoms.

Technorati Tags: software security

Frequently, those directing software security initiatives ask what others in their space are doing. I believed this was a perfectly reasonable question and answered, dutifully protecting each side’s confidentiality as best as humanly possible. Indeed, this kind of perspective represents one key value Cigital provides to our clients.

Over time my relationship with clients deepened, as did their maturity in software security. Their questions also deepened, getting more specific: “How far down the static analysis tool adoption path are my competitors?” I can’t see any way of answering questions this specific without giving away others’ competitive advantage, potentially exposing them to risk, or violating their trust (not to mention NDAs). Stuck wondering if I would be unable to provide further perspective, I began to question this perspective’s real value:

“Is the Jones family really the goal?” I asked myself. Actually, I’m pretty sure it isn’t. Each organization’s security efforts should grow very differently from one and other. They’ll start at different places, sure. Not only that, but even if you tackle the same problem as your competitor chooses to tackle, the ‘optimal’ approach for each organization differs. Why? Because each IT shop grew up to support their business differently. Metaphorically both you and the Joneses have children—but both sets of children have very different special needs.

Certainly, hallmarks of a good program exist. At this point, I consider organizations behind unless they:

• Conduct source code and architectural reviews of software produce and acquire;
• Leverage a static analysis tool suite in reviews;
• Train developers using hands-on instructor-lead courses;
• Augment training with continuing education;
• Distribute technology-specific security guidance, showing developers proper use of open source toolkits, frameworks, and COTS components;
• Subject applications to penetration testing in production;
• Measure both the vulnerability of software as well as the effectiveness of means by which software is secured.

While not exhaustive, the above list indicates a firm commitment to whitebox analysis. Cigital helps companies deploy the above means, in lock-step with each other, in order to get maximum benefit. Without help though, which measure you focus on next depends your organization. But additionally, how you make each effective will also depend on specifics of your organization. I’ve listed several factors that play into software security initiative decision making:

• Do you build, acquire, or buy most of your deployed software?
• Is your company an IT shop, supporting a business, or a product company?
• How do you characterize the bulk of your developers: Off-shore, out-sourced, contracted, seasoned, new to the field?
• What background do your Security resources possess: Network security, Development, Architecture, Management?
• What languages do you use, primarily, for software development?
• How varied is your development platform and build environment?
• Where do security resources report in the organization?
• Is the organizational culture centralized, one of strong governance? Or, does the organization rely on engineers to direct its efforts?
• What architectural paradigms does your application suite conform to? Are they primarily n-tier, monolithic, client/server, or federated? What messaging, transaction, structural, and behavioral patterns do they implement?
• What assets do the applications protect: functionality, data?

Obviously, when we conduct an organizational assessment we augment the above questions with a wealth of others.

Having explored data the above prompts gather, I suggest organizations build on their strengths rather than attacking weakness head-on. For instance, rather than foisting security analysts on development to do architectural analysis, I suggest teams subscribing to an agile methodology begin by augmenting use cases with misuse/abuse cases. Later, those teams can build towards threat modeling and begin to add elements of architectural analysis into their process of re-factoring. Augmenting a familiar activity always proceeds more smoothly than adding another (perhaps foreign) step. This advice stands in complete opposition to what I commonly give to IT shops with strong central governance: begin conducting architectural risk analysis immediately.

Over time, your security efforts will achieve coverage over prescribed best practices. Playing off your organization’s strengths can keep development more comfortable as it maneuvers to deal with security, often a new objective. If your security group succeeds in rolling out security practices synergistically development will view them as resources, not annoying cops. Ideally development begins to pull security into their efforts—requesting their time proactively because they see the value.

Remember, there’s no one-size-fits-all solution, especially in a discipline as young as software security. What will work best for you depends on the specifics of your organization. So rather than worrying about the Joneses look to understand your own situation better and respond to it in turn.

-jOHN

Technorati Tags: software security