When it comes to making a critical decision, it’s essential to have the data to back it up. This is especially true in operational test and evaluation where analysts apply advanced analytical methods to deliver relevant, accurate and timely information to senior leaders responsible for major acquisition decisions.
The Air Force Operational Test and Evaluation Center trains its military and civilian analysts on these methods and their application in a focused five-day course known as AFOTEC 401. In the most recent iteration of the course, the course cadre introduced a new Capstone project which provides practical application of the advanced methods. This project builds upon the classroom-based sections of the course which introduce the test and evaluation concepts of:
- outlining the design space
- designing the test
- collecting test data
- applying analytic methods to test results.
In outlining the design space, students are taught the operational test and evaluation construct which includes battlespace development, test goals, design of experiments, and test measures development.
In designing the test, students are introduced to concepts such as evaluation of designs, advanced and augmented designs, as well as planning and execution of data collection. They are then introduced to analysis methods for modeling and graphing results.
The cadre has also introduced the concepts of Bayesian statistics into the course in anticipation of future application of that statistical approach to AFOTEC-led testing. Bayesian statistical methods use Bayes' theorem to compute and update probabilities after obtaining new data. Bayes' theorem describes the conditional probability of an event based on data as well as prior information or beliefs about the event or conditions related to the event.
For the Capstone project, students apply the techniques learned in the classroom to plan and conduct a notional operational test. The project provides a realistic venue for the application of analytic methods; with the course cadre providing real time oversight of the students’ progress. The new Capstone project includes modeling and simulation, larger test data sets for processing, integrated test conditions, and active troubleshooting to better prepare students for actual operational test conditions.
The cadre of analysts and engineers accomplished this by introducing a manned drone system in the form of a surveillance and reconnaissance system.
“The course cadre exhibit the ingenuity of our Air Force analysts and engineers. They worked through many rounds of prototyping, building custom electronics, writing code, and integrating feedback from other members of the team,” said Col. Travis Herbranson, AFOTEC’s Director of Intelligence, Analyses, and Assessments. “Mr. Vance Oas and Ms. Alex Kozai were pivotal in this effort to deliver relevant and realistic training to the test and evaluation community. The team’s efforts epitomize the ingenuity our analysts and engineers bring to the fight.”
In the Capstone project, the notional “system under test” is a surveillance and reconnaissance system undergoing operational testing. In this case, the system is represented by a Brookstone Rover Revolution vehicle, a wirelessly controlled vehicle that was modernized with a new operator interface. The system has onboard sensors to determine distance from rover, provide visual identification, detect ambient sound, determine 3-D space positioning, temperature, and humidity. It connects to remote sensors for motion detection, sensor to target range, data storage, and real-time clock capabilities.
“By implementing a more realistic Capstone, students get hands-on experience with the process and tools used in operational test before the first time they need to execute on a real program. They can troubleshoot real, rather than simulated, issues and work with data sets representative of actual programs,” said Colonel Herbranson. “This evolution in training is a first step towards implementing the AFOTEC Commander’s vision and addressing Secretary Frank Kendell’s Global Power Competition imperative to restructure key processes related to weapons systems to be data-driven and risk-informed.”