Technical Article

Implementing a Workflow for Deploying and Integrating Deep Learning Networks on PLCs for Industrial Automation

By Dr. Fabian Bause and Nicolas Camargo Torres, Beckhoff Automation GmbH & Co. KG


“A key advantage of the workflow described here is that it enables teams to evaluate tradeoff decisions via rapid iterations. For example, in our quality control applications we traded some classification accuracy for reduced network size and execution times.”

In industrial automation, machine learning has many potential applications. A vision-based classifier, for example, can assess the quality of finished products, identify potentially faulty components before they are added to an assembly, or sort fruits and vegetables that tend to have significant natural variations. Although building some of these applications with traditional computer vision techniques might be possible, the emergence of AI, such as deep learning, is driving the automation of visual inspection.

However, engineering teams often encounter difficulties deploying AI models to a PLC or Industrial PC. Thus, instead of integrating control and deep learning algorithms in a single system, they use separate systems, leading to higher latency and deployment and maintenance costs. Another challenge that teams encounter stems from the disparate skillsets required: Data science and industrial control workflows have, to date, shared relatively little common ground.

Our team at Beckhoff Automation has implemented a new workflow that combines MATLAB® tools and Beckhoff Automation products to enable low-code design and AI model training—and simplifies the deployment and integration of those models on industrial targets. Working with MathWorks engineers, we developed this workflow and demonstrated it on an example quality control application that involved the visual inspection of hex nuts (Figure 1). While this simple application classifies hex nuts as either defective or not, demonstrating a straightforward use case, the steps in the workflow can be applied to accelerate the development and deployment of much more sophisticated and complex applications.

Video length is 0:22

Figure 1. An application of quality control in which AI models are used to inspect the quality of hex nuts. 

These steps are:

  1. Design, train, and optimize a deep learning model in MATLAB with Deep Learning Toolbox™—or import one from another machine learning framework like PyTorch® or TensorFlow™.
  2. Create a compiled TwinCAT® object from the model using MATLAB Coder™ and TwinCAT Target for MATLAB (the compiled TwinCAT object can contain both pre- and postprocessing code in addition to the deep learning model).
  3. Invoke the TwinCAT object in TwinCAT 3 engineering to perform classifications or make predictions. Integrate it with other capabilities or components running on the same PLC.

In addition to these core steps, our demonstration workflow included two additional complementary steps:

  1. Design a control algorithm in Simulink® and Stateflow®. Create a TwinCAT object from this control model using Simulink Coder™ and TwinCAT 3 Target for Simulink.
  2. Use MATLAB App Designer to design a human-machine interface (HMI) that uses TwinCAT 3 Interface for MATLAB and Simulink to exchange data between MATLAB and the TwinCAT run-time environment.

Transfer Learning with a Pretrained Network

After collecting and preparing the data to be used in a deep learning application, the workflow’s first step is training a deep learning model. With MATLAB and Deep Learning Toolbox, there are several ways to do this, including training a network from scratch with Deep Network Designer app, defining a deep learning model as a function and using a custom training loop, or retraining a pretrained model with new data, also known as transfer learning. If there is a small amount of abnormal data, anomaly detection methods such as FCDD and PatchCore, which are included in the Automated Visual Inspection Library for Computer Vision Toolbox™, are also effective.

For the hex nut example application, we chose to use transfer learning, retraining convolutional neural networks to classify a set of hex nut images. Specifically, we loaded pretrained ResNet-18 and SqueezeNet networks into MATLAB and then retrained them to classify hex nut images (Figure 2).

Video length is 1:33

Figure 2. Training a deep learning network to classify new images with Deep Learning Toolbox.

Although the ResNet-18 network was highly accurate, it was an order of magnitude larger and slower than the SqueezeNet network, which was somewhat less accurate, but still accurate enough for the use case at hand. In addition, given the PLC timing budget of 300 ms, we see ResNet-18 would not satisfy our requirements, hence we pivoted to SqueezeNet. We further improved the performance of this network—and reduced its size—by pruning filters of convolution layers. This pruning resulted in just a 7% decrease in accuracy but a twofold increase in speed. The ability to quickly evaluate different networks and network compression options is a significant advantage for teams that need to make tradeoff decisions involving accuracy, speed, and size.

Building a Compiled TwinCAT Object and Integrating It in TwinCAT 3 Engineering

Creating a TwinCAT object from a MATLAB function (in this case a function that invokes a deep learning model) is a two-step process. The first step is to generate C/C++ code for the function using MATLAB Coder. The second step is to use TwinCAT Target for MATLAB to compile the generated code into a TwinCAT object. Writing a script to execute these two steps makes it possible to automate this part of the workflow (Figure 3). When any changes are made to the deep learning network, the team can simply rerun this script and immediately generate an updated TwinCAT object.

Video length is 0:20

Figure 3. Generating C/C++ code that will be used to build a TwinCAT object.

It’s important to note here that there is another avenue for deploying a deep learning network in MATLAB to a Beckhoff PLC. With this approach, a team would export an ONNX file from MATLAB and load this file into TwinCAT 3, where it can be used with an inference engine (TwinCAT Machine Learning Server) to perform classification or regression tasks—even with the option to compute the model on a GPU. One advantage that the approach in our workflow (based on MATLAB Coder and TwinCAT Target for MATLAB) has over the alternative is that it can be used to include additional preprocessing and postprocessing functionality implemented in MATLAB. With the ONNX export approach, only the network itself is deployed.

Once the TwinCAT object has been compiled, it can be used in TwinCAT engineering just like any other object. An automation engineer can integrate the deep learning model, now implemented as a TwinCAT object, with other PLC code, including structured text. In our sample application, for example, we wrote code that invoked the compiled object, passing it an image (resized and converted to a matrix of pixels) captured from a camera. The code then processed the classifier’s output, which includes both the prediction result (the hex nut is OK or defective) and a probability score that reflects the confidence in the result (Figure 4).

A screenshot showing system integration of a deep learning model with TwinCAT Vision.

Figure 4. System integration of a deep learning model with TwinCAT Vision.

Motion Control Design and Deployment

To complete our example quality control application design, we needed a control system for a servo motor used to position hex nuts within the field of view of a camera. Although we could have implemented this control system directly in TwinCAT engineering, we took this opportunity to use Model-Based Design while targeting the same Beckhoff PLC used in our classifier deployment. We began by modeling the motion controller in Simulink with Stateflow. Next, after running simulations to verify the design, we used Simulink Coder to generate C/C++ code from our model, and then used TwinCAT 3 Target for Simulink to compile the code into a TwinCAT object. We could then visualize this object in TwinCAT engineering and integrate it with other elements of our design (Figure 5).

A Simulink model showing a motion control system with inputs, logic gates, and outputs for drive control and position feedback.

Figure 5. A Simulink model for motion control.

HMI Design with MATLAB App Designer and TwinCAT Interface for MATLAB

In most industrial automation use cases, an HMI is provided to enable operators to monitor and control equipment, view real-time data, receive alerts, and manage production processes. To create the HMI for our example quality control application, we started by laying out the design of the user interface using MATLAB App Designer. The operator can control the motion of the hex nuts via the interface, as well as view the hex nut currently positioned under the camera, see the results of the deep learning network (including the classification and the probability score), and monitor benchmarks, including the classifier’s execution time (Figure 6). All the data exchange between the user interface and the PLC run-time environment is conducted via the TwinCAT 3 Interface for MATLAB and Simulink, which uses the Automation Device Specification (ADS) communication protocol.

A screenshot showing the hex nut quality control HMI.

Figure 6. The hex nut quality control HMI, built with MATLAB App Designer and the TwinCAT 3 Interface for MATLAB and Simulink.

Future Iterations

A key advantage of the workflow described here is that it enables teams to evaluate tradeoff decisions via rapid iterations. For example, in our quality control applications we traded some classification accuracy for reduced network size and execution times. If the design requirements called for still lower latency, we could have sought further improvements through modifications to the network, or perhaps chosen a PLC with components and processors from a higher performance class.

It is also possible to extend this workflow to meet specific project requirements. One such extension would be to use the classification result of the deep neural network to influence the control system. While not needed in the specific quality control demonstrator we built, the ability for a control system to take some immediate action based on the classification result is widely applicable to a broad range of industrial automation use cases and would be a natural next step for many teams adopting this workflow.

Published 2024