Informal techniques are among the most commonly used. They are called informal because they rely heavily on human reasoning and subjectivity without stringent mathematical formalism. The informal label should not imply, however, a lack of structure or formal guidelines in their use. In fact, these techniques should be applied using well-structured approaches under formal guidelines. They can be very effective if employed properly.
The following techniques are discussed in the paragraphs below:
An audit is a verification technique performed throughout the development life cycle of a new model or simulation or during modification made to legacy models and simulations. An audit is a staff function that serves as the "eyes and ears of management" [Perry, 1995, p. 26]. An audit is undertaken to assess how adequately a model or simulation is used with respect to established plans, policies, procedures, standards, and guidelines. Auditing is carried out by holding meetings and conducting observations and examinations [Hollocker, 1987]. The process of documenting and retaining sufficient evidence about the substantiation of accuracy is called an audit trail [Perry, 1995]. Auditing can be used to establish traceability within the simulation. When an error is identified, it should be traceable to its source via its audit trail.
Desk checking, or self-inspection, is an intense examination of a working product or document to ensure its correctness, completeness, consistency, and clarity. It is particularly useful during requirements verification, design verification, and code verification. Desk checking can involve a number of different tasks, such as those listed in the table below [Beizer, 1990].
Typical Desk Checking Activities
· syntax review
· cross-reference examination
· convention violation assessment
· detailed comparison to specifications
· code reading
· control flowgraph analysis
· path sensitizing
To be effective, desk checking should be conducted carefully and thoroughly, preferably by someone not involved in the actual development of the product or document, because it is usually difficult to see one’s own errors [Adrion et al., 1982].
The project team members, potential users of the model, and subject matter experts (SMEs) review simulation output (e.g., numerical results, animations, etc.) for reasonableness. They use their estimates and intuition to compare model and system behaviors subjectively under identical input conditions and judge whether the model and its results are reasonable [Hermann, 1967].
Informal Techniques Example 1:
Face validation was used in the development of a simulation of the U.S. Air Force (AF) manpower and personnel system to ensure it provided an adequate representation. The simulation was designed to provide AF policy analysts with a system-wide view of the effects of various proposed personnel policies. The simulation was executed under the baseline personnel policy and results shown to AF analysts and decision-makers who subsequently identified some discrepancies between the simulation results and perceived system behavior. Corrections were made and additional face validation evaluations were conducted until the simulation appeared to closely approximate current AF policy. The face validation exercise both demonstrated the validity of the simulation and improved its perceived credibility [Morrison]
Face validation is regularly cited in V&V efforts within the Department of Defense (DoD) M&S community. However, the term is commonly misused as a more general term and misapplied to other techniques involving visual reviews (e.g., inspection, desk check, review). Face validation is useful mostly as a preliminary approach to validation in the early stages of development. When a model is not mature or lacks a well-documented VV&A history, additional validation techniques may be required.
Inspection is normally performed by a team that examines the product of a particular simulation development phase (e.g., M&S requirements definition, conceptual model development, M&S design). A team normally consists of four or five members, including a moderator or leader, a recorder, a reader (i.e., a representative of the Developer) who presents the material being inspected, the V&V Agent; and one or more appropriate subject matter experts (SMEs). Normally, an inspection consists of five phases: overview, preparation, inspection, rework, and follow-up [Schach, 1996].
Informal Techniques Example 2:
The team inspecting a simulation design might include a moderator; a recorder; a reader from the simulation design team who will explain the design process and answer questions about the design; a representative of the Developer who will be translating the design into an executable form; SMEs familiar with the requirements of the application, and the V&V Agent.
· Overview -- The simulation design team prepares a synopsis of the design. This and related documentation (e.g., problem definition and objectives, M&S requirements, inspection agenda) is distributed to all members of the inspection team.
· Preparation --The inspection team members individually review all the documentation provided. The success of the inspection rests heavily on the conscientiousness of the team members in their preparation.
· Inspection -- The moderator plans and chairs the inspection meeting. The reader presents the product and leads the team through the inspection process. The inspection team can be aided during the faultfinding process by a checklist of queries. The objective is to identify problems, not to correct them. At the end of the inspection the recorder prepares a report of the problems detected and submits it to the design team.
· Rework --The design team addresses each problem identified in the report, documenting all responses and corrections.
· Follow-up -- The moderator ensures that all faults and problems have been resolved satisfactorily. All changes should be examined carefully to ensure that no new problems have been introduced as a result of a correction.
A review is intended to evaluate the simulation in light of development standards, guidelines, and specifications and to provide management, such as the User or M&S PM, with evidence that the simulation development process is being carried out according to the stated objectives. A review is similar to an inspection or walkthrough, except that the review team also includes management. As such, it is considered a higher-level technique than inspection or walkthrough.
A review team is generally comprised of management-level representatives of the User and M&S PM. Review agendas should focus less on technical issues and more on oversight than an inspection. The purpose is to evaluate the model or simulation relative to specifications and standards, recording defects and deficiencies. The V&V Agent should gather and distribute the documentation to all team members for examination before the review. The V&V Agent should also prepare a set of indicators to measure such as those listed in the table below.
· appropriateness of the problem definition and M&S requirements
· adequacy of all underlying assumptions
· adherence to standards
· modeling methodology
· quality of simulation representations
· model structure
· model consistency
· model completeness
The V&V Agent may also prepare a checklist to help the team focus on the key points. The result of the review should be a document recording the events of the meeting, deficiencies identified, and review team recommendations. Appropriate actions should then be taken to correct any deficiencies and address all recommendations.
The Turing test is used to verify the accuracy of a simulation by focusing on differences between the system being simulated and the simulation of that system. System experts are presented with two blind sets of output data, one obtained from the model representing the system and one from the system, created under the same input conditions and are asked to differentiate between the two. If they cannot differentiate between the two, confidence in the model’s validity is increased [Schruben, 1980; Turing, 1963; Van Horn, 1971]. If they can differentiate between them, they are asked to describe the differences. Their responses provide valuable feedback regarding the accuracy and appropriateness of the system representation.
The main thrust of the walkthrough is to detect and document faults; it is not a performance appraisal of the Developer. This point must be made to everyone involved so that full cooperation is achieved in discovering errors. A typical structured walkthrough team consists of
· Coordinator, often the V&V Agent, who organizes, moderates, and follows up the walkthrough activities
· Presenter, usually the Developer
· Maintenance oracle, who focuses on long-term implications
· Standards bearer, who assesses adherence to standards
· Accreditation Agent, who reflects the needs and concerns of the User
· Additional reviewers such as the M&S PM and auditors
Except for the Developer, none of the team members should be involved directly in the development effort. [Adrion et al., 1982; Deutsch, 1982; Myers, 1978, 1979; Yourdon, 1985].
Inspections differ significantly from walkthroughs. An inspection is a five-step, formalized process. The inspection team uses the checklist approach for uncovering errors. A walkthrough is less formal, has fewer steps, and does not use a checklist to guide or a written report to document the team’s work. Although the inspection process takes much longer than a walkthrough, the extra time is justified because an inspection is extremely effective for detecting faults early in the development process when they are easiest and least costly to correct [Ackerman et al., 1983; Beizer, 1990; Dobbins, 1987; Knight and Myers, 1993; Perry, 1995; Schach, 1996].
Inspections and walkthroughs concentrate on assessing correctness. Reviews seek to ascertain that tolerable levels of quality are being attained. The review team is more concerned with design deficiencies and deviations from the conceptual model and M&S requirements than it is with the intricate line-by-line details of the implementation. The focus of a review is not on discovering technical flaws but on ensuring that the design and development fully and accurately address the needs of the application. For this reason, the review process is effective early on during requirements verification and conceptual model validation. [Hollocker, 1987; Perry, 1995; Sommerville, 1996; Whitner and Balci, 1989].
Ackerman, A.F., Fowler, P.J., & Ebenau, R.G., “Software inspections and the industrial production of software,” in Hans-Ludwig Hausen (Ed.), Software validation: Inspection, testing, verification, alternatives, Proceedings of the Symposium on Software Validation, pp. 13–40, Darmstadt, FRG, 1983.
Adrion, W.R., Branstad, M.A., & Cherniavsky, J.C., “Validation, verification, and testing of computer software,” Computing Surveys, 14 (2), pp. 159–192, 1982.
Beizer, B., Software testing techniques (2nd ed.), Van Nostrand Reinhold, New York, 1990.
Deutsch, M.S., Software verification and validation: Realistic project approaches, Prentice-Hall, Englewood Cliffs, NJ, 1982.
Dobbins, J.H., “Inspections as an up-front quality technique,” in G.G. Schulmeyer & J.I. McManus (Eds.), Handbook of Software Quality Assurance, pp. 137–177, Van Nostrand-Reinhold Company, New York, 1987.
Hermann, C.F., “Validation problems in games and simulations with special reference to models of international politics,” Behavioral Science, 12 (3), pp. 216–231, 1967.
Hollocker, C.P., “The standardization of software reviews and audits,” in G.G. Schulmeyer & J.I. McManus (Eds.), Handbook of Software Quality Assurance, pp. 211–266, Van Nostrand-Reinhold Company, NY, 1987.
Knight, J.C. & Myers, E.A., “An improved inspection technique,” Communications of the ACM, 36 (11), pp. 51–61, 1993.
Myers, G.J., “A controlled experiment in program testing and code walkthroughs/inspections,” Communications of the ACM, 21 (9), pp. 760–768, 1978.
Myers, G.J., The art of software testing, John Wiley & Sons, NY, 1979.
Perry, W., Effective methods for software testing, John Wiley & Sons, NY, 1995.
Schach, S.R., “Software Engineering (3rd ed.), Irwin, Homewood, IL, 1996.
Schruben, L.W. “Establishing the credibility of simulations,” Simulation, 34 (3), pp. 101–105, 1980.
Sommerville, I., Software engineering (5th ed.), Addison-Wesley, Reading, MA, 1996.
Turing, A.M., “Computing machinery and intelligence,” in E.A. Feigenbaum & J. Feldman (Eds.), Computers and thought, pp. 11–15,McGraw-Hill, NY, 1963.
Van Horn, R.L., “Validation of simulation results,” Management Science, 17 (5), pp. 247–258, 1971.
Whitner, R.B. & Balci, O., “Guidelines for selecting and using simulation model verification techniques,” in E.A. MacNair, K.J. Musselman, & P. Heidelberger (Eds.), Proceedings of the 1989 Winter Simulation Conference, pp. 559–568, IEEE, Piscataway, NJ, 1989.
Yourdon, E., Structured Walkthroughs (3rd ed.), Yourdon Press, NY, 1985.
RPG Links in this Document
RPG Special Topic: “Subject Matter Experts and VV&A”
§ § § § § § §