Key issues raised by Workshop attendees that need to be considered by 15288 Committee (Note: These issues are in no particular order; they were cataloged as they were spoken by the participants. Therefore some items were repeated at different times by different people.)
J. Voas (scribe)
1. 15288 defines a fixed set of development processes. 15288 should not mandate a fixed way to develop software. That restricts the value of 15288 too much.
2. What is a COTS software component? The definition that was proposed: a software component that has been obtained from a third party and that the developer uses on an "as is" basis. 15288 needs to address what is OTS or COTS software in more precise detail.
3. Is freezing a COTS component a reasonable way to keep systems long-lived if the COTS component receives extremely thorough certification before the system is initially deployed? 15288 should discuss this "unique" design strategy and whether it advocates it. Or at least 15288 can mention it as one approach to reducing maintenance costs in that life-cycle.
4. Is the TCSEC a reasonable way to certify COTS software for security? Since it appears not to be, 15288 can step in a play a role here and will need to for security-critical applications to consider using 15288.
5. 15288 must consider different viewpoints:
6. 15288 defines system as always having humans, software, hardware, and computers. What is the difference between hardware and computers?
7. Too much functionality in a COTS product may cause more problems than too little functionality. What do we do here?
8. How best to evaluate and update COTS components? This refers to the problems of COTS software certification. Almost every speaker in the workshop mentioned this and wanted to see something in 15288 that addresses it.
9. Vendors that adopt 15288 must expect that it may be necessary to test and possibly rewrite glue and wrappers many times in the system life cycle when the system includes COTS software. This should be mentioned so that vendors do not think they have errored if they wind up in this situation.
10. Must have a contingency plan for system downgrading if a COTS component must be removed. 15288 should make mention for how this is to be done.
11. COTS software. vs. CAS (Commercially Acquired Software): the issue here is one commercial customized version vs. multiple generic commercial versions. 15288 needs to make a distinction.
12. Configuration management: COTS components must be tracked by version number otherwise chaos will result during maintenance. The general feeling of the group was that 15288 does not address maintenance issues nearly thoroughly enough.
13. All COTS component failures must be detected, isolated, and logged. 15288 must require this. None in the group knew whether it did or not.
14. Instrumentation must be built into the custom-code since it cannot be built into the COTS software. 15288 should mention this and the advantages of doing so.
15. Software maintenance must be upgrade driven and not user requirements driven. This should be mentioned in 15288.
16. We must have ways to maintain components during system development. This should be a requirement in 15288. 15288 need not state how this is done, but instead must anticipate that a way for doing it will exist.
17. How do you coordinate the management of systems when multiple products are all upgraded at different times? One approach is to totally upgrade all components at one time and only at that time.
18. 15288 cannot separate system management from software management. Several participants felt strongly about this.
19. The term "system" must be defined at different levels in 15288, e.g., software system, the system that the software is embedded into, etc.
20. 15288 must address the idea of dormant software. Dormant software is the extra functionality in a COTS product that is never used: Here are some issues that were raised in the session:
22. How do you certify that COTS software does not have any malicious or Trojan Horse functionality? 15288 must address security.
23. Analyses should be taken to qualify and quantify the effect of faulty hardware on the COTS software. The reverse analysis should also be considered.
24. E-commerce systems have serious potential COTS problems because of bad implementations in Java Virtual Machines. 15288 must be broad enough in scope to consider these new types of information systems and how mobile code and mobile objects greatly change the software development community.
25. 15288 must address the ilities: "safety", "reliability", "dependability", "security" and it should address ways to qualify or quantify these ilities with respect to the entire system. Each of these must be separated out in 15288, and different system life-cycle techniques that address these must be enumerated.
26. 15288 should consider the capabilities of Java JDK 1.2 when addressing COTS security since Java is probably the most widely used COTS component in use today.
27. 15288 needs to address security in more detail
28. 15288 needs to address certification of COTS software.
29. 15288 needs to address CORBA and other brokering technologies between software components and / or hardware components. Object brokering is becoming a commonplace technology, and it clearly affects 15288.
30. Stan Magee mentioned that maybe there needs to be an ISO standard for OTS software. This may have nothing to do with 15288 though, so this point might be mute.
31. How do we estimate the reliability of COTS when the vendor is mute about its quality? Or should we even attempt to? 15288 should address the reliability of software components issue. Maybe it does, but no one in attendance really knew.
32. How do we estimate the reliability of COTS software and/or hardware when it is embedded in a larger system? This must be addressed so models such as fault trees can be used to predict overall system reliability.
33. How do we revise our reliability estimates for a system once COTS components are swapped out and newer upgraded components are swapped in? This is vital for efficiency during maintenance. 34. How should required enhancements to a system be handled when talking to a COTS vendor? That is, how do you inform a COTS vendor that they might need to make changes to their offering when they did not have plans to?
35. Networks add additional problems. How best can we handle the interoperability issues of different network COTS products? Almost all systems today rely to some degree on network protocols. How can we assess a system's reliability if we do not know the reliability of the network protocols that it depends on?
36. Organizational Change Management (OCM) must be addressed in 15288. This refers to managing the change in people, jobs, structure, work processes and operating procedures resulting from the implementation of any number of strategic decisions. To address OCM, 15288 must understand the target organization of the COTS software.
37. Current 15288 does not easily support the OCM life cycle model
38. 15288 focuses too much on technical issues and not enough on change management.
39. OCM can contribute up to 50% of project success for a business information system and therefore must 15288 must address OCM.
40. Contact Doug Thiele for ideas on what to add to 15288 that satisfy his 15288 concerns on OCM.
41. Fault injection is a means for simulating COTS failures to determine the effect of COTS failures on a system that contains COTS components.
42. Wrappers on COTS components can be more defective than the COTS components themselves. 15288 needs to define the role of middleware such as wrappers that are built by the system integrators to protect themselves against bad software components.
43. Liability for defective COTS software must be explicitly discussed in 15288. I'm not sure why since liability is a country by country issue, but someone raised this point.
44. Does a system integrator have a claim against a COTS vendor when the COTS vendor supplies defective software components? 15288 should address this. This is related to #43.
45. Because 80% of the total costs of an information system can go into operations and maintenance, 15288 needs to address that phase particularly carefully.
46. Fault tree analysis might be a way for a COTS vendor to show that certain undesirable output states are not possible. This could be guidance in 15288.
47. Slicing and testing in combination might be a substitute for fault tree analysis (see #46).
48. System level testing by the system integrator can give some information about how the system will behave if the COTS software fails but if the COTS software fails rarely, this information may be unattainable.
49. What effect will all of the different brokering architectures (e.g., COM, DCOM, CORBA, etc) have on 15288. These brokering architectures define the manner by which objects communicate. 15288 needs to consider that those may be commercial industry's de facto standard by the time 15288 is finalized. This was mentioned also in #29.
Jeffrey Voas
Cigital
21351 Ridgetop Circle
Suite 400
Sterling, VA 20166
jmvoas@cigital.com
Phone: 703-404-9293
Fax: 703-404-9295