A MATLAB crash while using a custom C++ library could be caused by a number of issues, including an issue with the custom library (for example, coding error) or a version conflict between the custom library, or its dependencies, and third-party C++ libraries that are shipped with MATLAB.
For both of these types of causes, it might be beneficial to use out-of-process execution mode (available in R2023a, and later releases).
If a library conflict is the cause of the crash, working in out-of-process execution mode may resolve the issue, since out-of-process execution mode has fewer dependencies on third-party libraries. Also, if the library conflict is not resolved by using out-of-process execution mode, the process will simply terminate and MATLAB will not crash.
Similarly, if the crash is due to a coding issue, then debugging and testing the code will be easier in out-of-process execution mode, since errors will simply cause the process to terminate and MATLAB will not crash. In addition, after making code changes, the library can be unloaded and reloaded without having to restart MATLAB.
