For over 100 years, the evolution of modern survey methodologyâ€"using the theory of representative sampling to make interferences from a part of the population to the wholeâ€"has been paralleled by a drive toward automation, harnessing technology and computerization to make parts of the survey process easier, faster, and better. The availability of portable computers in the late 1980s ushered in computer-assisted personal interviewing (CAPI), in which interviewers administer a survey instrument to respondents using a computerized version of the questionnaire on a portable laptop computer. Computer assisted interviewing (CAI) methods have proven to be extremely useful and beneficial in survey administration. However, the practical problems encountered in documentation and testing CAI instruments suggest that this is an opportune time to reexamine not only the process of developing CAI instruments but also the future directions of survey automation writ large.
Harry Robinson leads software test productivity and model-based testing
initiatives as part of Microsoft's Six Sigma Team. ... Art. Susan Schechter is senior
statistician in the Statistical Policy Office of the U.S. Office of Management and
Budget.
The Department of Defense (DoD) recently adopted evolutionary acquisition, a dynamic strategy for the development and acquisition of its defense systems. Evolutionary defense systems are planned, in advance, to be developed through several stages in a single procurement program. Each stage is planned to produce a viable system which could be fielded. The system requirements for each stage of development may be specified in advance of a given stage or may be decided at the outset of that stage's development. Due to the different stages that comprise an evolutionary system, there exists a need for careful reexamination of current testing and evaluation policies and processes, which were designed for single-stage developments. The Office of the Under Secretary of Defense for Acquisition, Technology and Logistics (USD-AT&L) and the Director of Operational Testing and Evaluation (DOT&E) asked the Committee on National Statistics (CNSTAT) of the National Academies to examine the key issues and implications for defense testing from the introduction of evolutionary acquisition. The CNSTAT was charged with planning and conducting a workshop to study test strategies for the evolutionary acquisition. The committee reviewed defense materials defining evolutionary acquisition and interviewed test officials from the three major test service agencies to understand the current approaches used in testing systems procured through evolutionary acquisition. The committee also examined possible alternatives to identify problems in implementation. At the workshop that took place on December 13-14, 2004, the committee tried to answer many questions including: What are the appropriate roles and objectives for testing in an evolutionary environment?, Can a systematic, disciplined process be developed for testing and evaluation in such a fluid and flexible environment?, and Is there adequate technical expertise within the acquisition community to fully exploit data gathered from previous stages to effectively combine information from various sources for test design and analysis?. Testing of Defense Systems in an Evolutionary Acquisition Environment provides the conclusions and recommendations of the CNSTAT following the workshop and its other investigations.
Oversight Committee for the Workshop on Testing for Dynamic Acquisition of
Defense Systems, Committee on ... As Art Koehler described in a presentation at
the workshop, a particular application of modeling and simulation that is ... SOFTWARE DEVELOPMENT IN AN EVOLUTIONARY CONTEXT Software
development is one area in which evolutionary acquisition is expected to play a
major role.
In 2009, the Government Accountability Office (GAO) released the report Warfighter Support: Independent Expert Assessment of Army Body Armor Test Results and Procedures Needed Before Fielding, which commented on the conduct of the test procedures governing acceptance of body armor vest-plate inserts worn by military service members. This GAO report, as well as other observations, led the Department of Defense Director, Operational Test & Evaluation, to request that the National Research Council (NRC) Division on Engineering and Physical Sciences conduct a three-phase study to investigate issues related to the testing of body armor materials for use by the U.S. Army and other military departments. Phase I and II resulted in two NRC letter reports: one in 2009 and one in 2010. This report is Phase III in the study. Testing of Body Armor Materials: Phase III provides a roadmap to reduce the variability of clay processes and shows how to migrate from clay to future solutions, as well as considers the use of statistics to permit a more scientific determination of sample sizes to be used in body armor testing. This report also develops ideas for revising or replacing the Prather study methodology, as well as reviews comments on methodologies and technical approaches to military helmet testing. Testing of Body Armor Materials: Phase III also considers the possibility of combining various national body armor testing standards.
This section describes the “art of measurement” in testing procedures as
observed by the committee. Human ... These include subjective differences in
human handling, process transparency, and the selection and settings of software.
