Classify Images Using ONNX Model Predict Block

This example shows how to use the ONNX Model Predict block for image classification in Simulink®. The block accepts an image and passes it to a pretrained ONNX™ neural network classification model that is executed in Python®. The predicted response for the image is then output to Simulink.

MATLAB® supports the reference implementation of Python, often called CPython. If you use a Mac or Linux® platform, you already have Python installed. If you use Windows®, you need to install a distribution, such as those found at https://www.python.org/downloads/. For more information, see Configure Your System to Use Python. Your MATLAB Python environment must have the onnxruntime module installed.

The ONNX Model Predict block requires a pretrained ONNX model file that you saved in Python. This example provides the saved model digitsdlnetworkwithnoise.onnx, which was saved using onnxruntime version 1.15.0.

Open Provided Simulink Model

This example provides the Simulink model slexDigitsONNXModelPredictExample.slx, which includes the ONNX Model Predict block. You can open the Simulink model or create a new model as described in the next section.

Open the Simulink model slexDigitsONNXModelPredictExample.slx.

open_system("slexDigitsONNXModelPredictExample");

When you open the Simulink model, the software runs the code in the PreLoadFcn callback function before loading the Simulink model. The PreLoadFcn callback function of slexDigitsONNXModelPredictExample includes code to check if your workspace contains the inputIms variable for the trained model. If the workspace does not contain the variable, PreLoadFcn loads the image data for the Simulink model. To view the callback function, in the Setup section on the Modeling tab, click Model Settings and select Model Properties. Then, on the Callbacks tab, select the PreLoadFcn callback function in the Model callbacks pane.

Create Simulink Model

To create a new Simulink model, open the Blank Model template and add the ONNX Model Predict block from the Deep Learning Toolbox™ library. The block can be configured to accept input data in the form of an h-by-w-by-c-by-N numeric array, where h, w, and c are the height, width, and number of channels of the images, respectively, and N is the number of images.

Insert a Video From Workspace block from the Computer Vision Toolbox™ library and connect it to the input port of the ONNX Model Predict block. Double-click the Video From Workspace block to open the Block Parameters dialog box. Specify Signal as inputIms and Form output after final value by as Holding final value. Click OK.

Double-click the ONNX Model Predict block to open the Block Parameters dialog box. On the Specify model file tab, specify digitsdlnetworkwithnoise.onnx. Under Execution Providers, specify CPUExecutionProvider.

The input signal inputIms is a sequence of eight grayscale (one-channel) images that are 28-by-28 pixels in size. The saved digitsdlnetworkwithnoise.onnx model expects a four-dimensional signal with dimensions in a different order. On the Inputs tab, under Permutation to Python, specify [4 1 2 3].

Add a To Workspace block to the model and connect it to the output port of the ONNX Model Predict block. Double-click the To Workspace block to open the Block Parameters dialog box. Change the Variable name to yPred.

Open the Configuration Parameters dialog box. On the Modeling tab, click Model Settings to open the Configuration Parameters dialog box. On the left of the dialog box, click Solver. Under Simulation time, set Stop time to 7. Under Solver selection, set Type to Fixed-step, and set Solver to discrete (no continuous states). Click OK.

Load the digit sample data as an image datastore. The imageDatastore function automatically labels the images based on the folder names, and stores the data as an ImageDatastore object.

digitDatasetPath = fullfile(matlabroot,"toolbox","nnet","nndemos", ...
    "nndatasets","DigitDataset");
imds = imageDatastore(digitDatasetPath, ...
    IncludeSubfolders=true,LabelSource="foldernames");

Randomly select eight images from the image datastore, create the array of images displayIms, and display the selected images by using the function montage (Image Processing Toolbox).

rng(0,"twister") % For reproducibility
perm = randperm(10000,8);
for i = 1:8
    displayIms(:,:,:,i) = imread(imds.Files{perm(i)});
end
montage(displayIms,size=[1 NaN]);

Convert the images in displayIms to single data type.

inputIms = single(displayIms);

Specify the class names.

classNames = {'0','1','2','3','4','5','6','7','8','9'};

Save the model as slexDigitsONNXModelPredictExample.slx in Simulink.

Predict Using Simulink Model

Simulate the model and save the simulation output to modelOutput. The field modelOutput.yPred.Data contains the classification results. You might receive a warning message if your Python installation uses an onnxruntime version prior to 1.15.0.

modelOutput = sim("slexDigitsONNXModelPredictExample");

Display the sequence of images and the classification results.

tiledlayout(1,12,TileSpacing="None");
for i = 1:size(inputIms,4)
   nexttile
   imshow(displayIms(:,:,:,i))
   label = modelOutput.yPred.Data(:,:,i)==1;
   title([classNames{label}],FontSize=20)
end