Learning from other Domains to Advance AI Evaluation and Testing: Testing in Aircraft Design and Manufacturing

In aviation design and manufacturing, testing can generally be described as serving the following
purposes: (i) facilitating development of a product design; (ii) verifying its functionality and
robustness; (iii) demonstrating compliance of the design with regulatory standards in order to
obtain approval from a civil aviation authority such as the U.S. Federal Aviation Administration;
and (iv) verifying that the as-built product conforms to its design, complies with applicable
regulations, and is in a condition for safe operation.
Testing is, therefore, closely linked to showing compliance with regulatory requirements, which
forms the predicate for obtaining approvals from aviation authorities that allow aircraft to be
operated. To be sure, some design features do not require testing to show compliance, and
compliance showings may also be made based on analysis, historical findings of compliance,
inspection, or system architecture. But, testing remains a centerpiece of compliance, and the
regulations to which compliance must be shown fundamentally define the nature of aircraft
products and the processes for creating them.
For these reasons, testing with respect to aviation products must be considered in the context of
the overarching regulatory framework for standards and how compliance to standards must be
shown.