The recommended and common workflow for calling MATLAB Engine in C++ application consists of the following steps :
1. Setup the supported compiler for the 'mex' command. You can find the supported compiler for you version of MATLAB by using the page, System Requirements and Supported Compilers. Use the following command in MATLAB Command Window to setup the complier.
2. Open the engdemo.cpp file using the following command and copy the file to the current directory.
3. Build the application using the 'mex' command with the compiler we setup in 1.
mex -v -client engine engdemo.cpp
4. Before you can run the compiled application, you will need to modify the 'PATH' and 'LD_LIBRARY_PATH' environment variables.
a) Set the run-time library path. This command replaces the value, if any, in LD_LIBRARY_PATH.
setenv LD_LIBRARY_PATH matlabroot/bin/glnxa64:matlabroot/sys/os/glnxa64
b) Set the path. Make sure that you include the : path terminator character.
setenv PATH matlabroot/bin:$PATH
NOTE: Here matlabroot is the path of the directory where MATLAB is installed. It can be found by executing the command 'matlabroot' on MATLAB Command Window.
5. Run the compiled application.
I would recommend that you refer the following documentation page for more details.