Automation of the Operational Test Data Process
Frontier Technology, Inc (FTI) received a contract award to assist Army ATEC Mission with developing the capability to automate the operational test data process through design, demonstration, and implementation of a unified automated process related to system-of-systems test data with the objective of providing a draft product to the analyst.
The test data process consists of many time-related steps, most of which are conducted separately with significant intervening delays. The following steps are generally recognized, but may be varied in order or selected as needed, to be part of the test data process:
a) Collection of data from the system(s) under test
b) Transfer of selected data from the units(s) to a central or distributed collection point(s)
c) On-line monitoring of selected data during the test
d) Real-time test control
e) Conversion of data to a common format
f) Transfer and storage of the raw (unprocessed) data in a central or distributed database
g) Processing
h) Authentication
i) Warehousing of processed data in a central or distributed database
j) Restricting access
k) Retrieval of the processed data
l) Mining of the processed data for trends, relationships, and statistics
m) Generation of the Test Report
The end product of the test and evaluation process is a Test Report on the battlefield effectiveness and suitability of the system(s) or system-of-systems under test. This report must be definitive and timely to support major acquisition decisions. In particular, evaluators and analysts must be able to have draft test results shortly after the conclusion of a test to allow sufficient time for analysis and final editing of the report. The criticality of acquisition decision timelines and the reduced resources to support analysis and evaluation require an increased emphasis on use of automation to augment and accelerate the operational test data process. During post-event analysis, the evaluator and analyst will want to assess performance from a wide range of aspects – from purely technical (speed and quality of service, throughput, timeliness of system responses) to purely operational (what were the impacts on command and control decisions, were fire missions executed in a timely manner if fire support messages were re-routed due to network turbulence, was situational awareness adversely impacted). In either case, the information required is exponentially more complex than that normally provided or assessed during traditional acquisition efforts.
It is envisioned that tools from many large-scale operations may provide some of the potential solutions, for example, the Versatile Information Systems Integrated On-Line Nationwide (VISION) technology, will make unified collection possible. Software tools such as VeriFIDES can be used to restrict access of data to authorized users. Other potential sources of solutions may come from the rapidly evolving fields of visual analysis and data mining and from previous government efforts towards an automated process. Applications for monitoring and analysis must be integrated with a variety of data types in real-time, application interfaces and data collection instrumentation to perform these functions.
PHASE I: The goal of Phase I is to perform a study to identify various automation technologies to meet the objectives outlined above and provide a general approach to their integration into a unified process. This study will include identification of technology alternatives for each test data process step listed above which may be suitable for process integration and sources of these technologies. Reuse of existing technology with appropriate tailoring is encouraged as is identification of technologies needed which do not exist and possible avenues of development. The report should compare the technology alternatives and recommend a course of action.
| 
