Define and Verify AI System Requirements for ACAS Xu Neural Networks Integrated Into Simulink

Since R2026a

This example uses:

Open Live Script

This example shows how to define, implement, and verify requirements for a Simulink® model that integrates the ACAS Xu family of neural networks. This example is step 9 in a series of examples that take you through formally verifying a set of neural networks that output steering advisories to aircraft to prevent them from colliding with other aircraft.

To run this example, open Verify and Deploy Airborne Collision Avoidance System (ACAS) Xu Neural Networks and navigate to Step9DefineAndVerifyAISystemRequirementsForACASXuNeuralNetworksIntegratedIntoSimulinkExample.m. This project contains all of the steps for this workflow. You can run the scripts in order or run each one independently.

The Simulink model used in this example simulates the movements of two unmanned aerial vehicles (UAV). The ACAS Xu neural networks control the behavior of the ownship based on the actions of the intruder ship. For more information on the Simulink model, see Integrate ACAS Xu Neural Networks into Simulink.

Types of Requirements

Requirements are a collection of statements describing the desired behavior and characteristics of a system. Requirements ensure system design integrity and are achievable, verifiable, unambiguous, and consistent with each other. Each stage of design should have appropriate requirements.

This end-to-end example has two types of requirements:

  • AI Constituent requirements — A set of requirements for the neural networks that issue collision avoidance advisories. You can split these requirements into two subsets: requirements for the model architecture and requirements for the model performance.

  • AI System requirements — A set of requirements for the system that integrates the neural networks.

This step of the end-to-end example shows how to define, implement, and verify AI system requirements. For AI constituent requirements, see Define and Verify AI Constituent Requirements for ACAS Xu Neural Networks.

Define Requirements

This example includes a small set of realistic requirements as illustration. If a system needs to meet specific safety criteria, then it will likely have many more requirements.

A requirement describes behavior that a design must satisfy. Requirements typically take the form of natural language statements, formulas, tables, and graphics. In Requirements Toolbox™, you use the Requirements Editor (Requirements Toolbox) to author requirements and create requirement sets to contain the requirements. By linking one requirement set to another, you can trace each high-level requirement to its implementation. As the system design evolves, you can use iterative requirements analysis to enhance requirement traceability and coverage. You can use a traceability diagram to visualize requirement traceability.

In this example, the model must satisfy these requirements. the requirements the model must satisfy are that the two aircraft must not collide, starting from each of the configurations described in the Explore ACAS Xu Neural Networks example:

  • Left Abeam

  • Intruder Tail Chase

  • Right Gaining

  • Left Gaining

  • Left Closing

  • Right Abeam

  • Right Isosceles

  • Right Closing

  • Ownship Tail Chase

For information on how to add and manage requirements, see Use Requirements to Develop and Verify MATLAB Functions (Requirements Toolbox). The requirements for the AI constituents are attached to this example as a supporting file. Inspect the requirements in the Requirements Editor. Opening the file may take several minutes.

slreq.open(fullfile(matlab.project.rootProject().RootFolder,"requirements","AISystem.slreqx"));

AI system requirements in the Requirements Editor.

Load ACAS Xu Neural Networks

Load and save the networks. The example prepares the networks as shown in Explore ACAS Xu Neural Networks.

acasXuNetworkFolder = fullfile(matlab.project.rootProject().RootFolder,".");
zipFile = matlab.internal.examples.downloadSupportFile("nnet","data/acas-xu-neural-network-dataset.zip");
filepath = fileparts(zipFile);
unzip(zipFile,acasXuNetworkFolder);
matFolder = fullfile(matlab.project.rootProject().RootFolder,"acas-xu-neural-network-dataset","networks-mat");
helpers.convertACASXuFromONNXAndSave(matFolder);

Track and Implement Requirements Implementation

Each requirements has two properties that you can track:

  • Implemented — Indicates whether the requirement has been incorporated into the system or software design and development.

  • Verified — Indicates whether the implementation of the requirement passes the test and meets the specified criteria.

For example, the ACAS Xu system requirements are implemented if the model allows the ownship to follow steering advisory provided by the ACAS Xu neural networks.

The model is set up inside ACAS_Scenario.slxc, which is attached to this example as a supporting file. In the Requirements Editor, you can link each requirement to the part of the model that implements it. The ACAS Xu system requirements are implemented by the ACAS Xu subsystem inside the ACAS_Scenario Simulink model.

Requirement implementation link in the Requirements Editor.

For more information, see View and Link Requirements in Simulink.

Verify AI System Requirements

To verify requirements, create tests, link the tests to the requirements, then run the tests.

In Simulink, create tests using Simulink Test™.

In this example, write tests to check that, for each initial configuration, the separation between the ownship and intruder never falls below 500 ft. Falling below that threshold is known as a near mid-air collision (NMAC). You can find the tests for this example in the tests file, which is attached to this example as a supporting file.

Load all the ACAS Xu neural networks to the MATLAB base workspace by using the loadAllACASXuNetworks function, which is defined at the bottom of this example.

loadAllACASXuNetworks

Open the tests using Simulink Test Manager.

sltest.testmanager.view;
sltest.testmanager.load(fullfile(matlab.project.rootProject().RootFolder,"tests","AISystem.mldatx"));

Run the tests.

testResultsObj = sltest.testmanager.run;

AI system tests in Simulink Test Manager.

All tests pass.

To inspect the simulation output of the Left Abeam test scenario, in the Results and Artifacts panel, go to AISystem > Closed-Loop Tests > LeftAbeam and select Sim Output (ACAS_Scenario : ). To plot the distance between the two aircraft in this scenario, in the Data Inspector pane, under Sim Output (ACAS_Scenario : ), select ExtractRho:1. The distance between the aircraft never falls below 6000 ft, well above the threshold of 500 ft.

AI system test results in Simulink Test Manager.

In the Requirements Editor, select Refresh to view the updated verification status of the requirements.

Updated verification status of the requirements in Requirements Editor.

Generate Test Report

Generate a test report result by using the sltest.testmanager.report function.

sltest.testmanager.report(testResultsObj,...
    fullfile(matlab.project.rootProject().RootFolder,"artifacts","AISystemReport.pdf"),...
    IncludeTestResults=0,...
    IncludeSimulationSignalPlots=true,...
    NumPlotRowsPerPage=2);

The test result report includes an overview of the test results, as well as information about each individual test.

AI system requirements test result report.

Run Individual Test Scenario

You can run and replay individual test scenarios.

First, load the test file.

tf = sltest.testmanager.TestFile(fullfile(matlab.project.rootProject().RootFolder,"tests","AISystem.mldatx"));
ts = getTestSuites(tf);
tc = getTestCases(ts);

Select a test scenario from the drop down menu.

testSelector = "RightClosing";

Run the test.

tcNames = {tc(:).Name};
idx = tcNames == testSelector;
resultObj = run(tc(idx));

Replay the scenario by using the replayTestScenario function, which is defined at the bottom of this example.

replayTestScenario(resultObj)

Supporting Functions

loadAllACASXuNetworks

The loadAllACASXuNetworks function loads all 45 ACAS Xu neural networks into the workspace.

function loadAllACASXuNetworks
for previousAdvisory = 1:5
    for timeBeforeLossOfVerticalSeparation = 1:9
        net = helpers.loadACASNetwork(previousAdvisory,timeBeforeLossOfVerticalSeparation);
        varname = sprintf('net%d_%d',previousAdvisory,timeBeforeLossOfVerticalSeparation);
        assignin('base',varname,net);
    end
end
end

replaySimScenario

The replaySimScenario function creates an animation visualizing the interaction between the two aircraft.

function replayTestScenario(resultObj)
% Extract timetable from the test result object.
tcr = getTestCaseResults(resultObj);
runDataset = getOutputRuns(tcr);
dataset = export(runDataset);
data = extractTimetable(dataset);
data = fillmissing(data,"previous");

% Adjust video frames every second.
data = retime(data,"secondly");

% Retrieve waypoints from the test.
ownshipVarIdx = cellfun(@(x)isequal(x,'ownshipWpts'),{tcr.ParameterSet.ParameterOverrides.Variable});
intruderVarIdx = cellfun(@(x)isequal(x,'intruderWpts'),{tcr.ParameterSet.ParameterOverrides.Variable});
ownshipPth  = tcr.ParameterSet.ParameterOverrides(ownshipVarIdx).Value;
intruderPth  = tcr.ParameterSet.ParameterOverrides(intruderVarIdx).Value;

% Create replay scene.
helpers.replayScenario(data,ownshipPth,intruderPth)
end

References

[1] Manzanas Lopez, Diego, et al. “Evaluation of Neural Network Verification Methods for Air-to-Air Collision Avoidance.” Journal of Air Transportation, vol. 31, no. 1, Jan. 2023, pp. 1–17. DOI.org (Crossref), https://doi.org/10.2514/1.D0255.