How can I run Python Functions in MATLAB?

I have created a Neural Network in Python but would like to analyze and test it in MATLAB. I have a funtion in Python 3.7 which uses TensorFlow and Keras to run the model using saved training weights. Additionally, it is on a separate GPU environment from the root. I am trying to be able to send inputs and see outputs from this model using MATLAB 2018a. I have scoured the web, but nothing (that I was able to gather) quite fit my situation.

My current solution, which produces errors, is this:

% Load pyversion in PythonGPU environment
pyversion('C:\Users\mitch\Anaconda3\envs\PythonGPU\python.exe')

This seems to work so far:

>> pyversion

       version: '3.7'
    executable: 'C:\Users\mitch\Anaconda3\envs\PythonGPU\python.exe'
       library: 'C:\Users\mitch\Anaconda3\envs\PythonGPU\python37.dll'
          home: 'C:\Users\mitch\Anaconda3\envs\PythonGPU'
      isloaded: 1

Then I add a path for the Neural Net:

pathToNN = 'C:\Users\mitch\Documents\...\NNTestingFolder';
if count(py.sys.path, pathToNN) == 0
    insert(py.sys.path, int64(0), pathToNN)
end

This also seems to work with no errors. But next is where I am getting stuck/ confused.

A lot of forums seem to suggest to run the funciton "runCarNN" in the Python file "NNtest.py", it is as simple as "py.NNtest.runCarNN". I get an error here...

>> py.NNtest.runCarNN
Undefined variable "py" or class "py.NNtest.runCarNN".

I have tried an alternative of using "system" to run the file, but I am met with errors there too

>> system('python C:\Users\mitch\Documents\...\NNtest.py')
2020-08-24 15:32:53.987079: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'cudart64_101.dll'; dlerror: cudart64_101.dll not found
2020-08-24 15:32:53.987742: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
Traceback (most recent call last): 
  File "C:\Users\mitch\Documents\...\NNtest.py", line 9, in <module> 
    import runCarNN 
  File "C:\Users\mitch\Documents\...\runCarNN.py", line 9, in <module> 
    from keras.models import Sequential 
  ...
  File "C:\Users\mitch\Anaconda3\lib\site-packages\h5py\__init__.py", line 34, in <module> 
    from . import version 
  File "C:\Users\mitch\Anaconda3\lib\site-packages\h5py\version.py", line 17, in <module> 
    from . import h5 as _h5 
  File "h5py\h5.pyx", line 41, in init h5py.h5 
AttributeError: type object 'h5py.h5.H5PYConfig' has no attribute '__reduce_cython__' 

I have omitted some errors, as they all trace back to the original error.

Since I am on 2018, pyenv is not implemented yet and most solutions I have found either use pyenv or py.ScriptName

Any help is appreciated! Thanks!

[Sergio E. Obando]

Hi Mitchell, according to the release notes, Python 3.7 was not supported until R2019a. Not sure if that is part of the issue but if possible might be worth testing the code in that version.

[Mitchell Metzger]

Thanks for the insight Sergio!

I have downgraded my Python version to 3.6 and it will run seemingly everything except TensorFlow... I have tested some other scripts that do not use TensorFlow and they work seeminly fine with

>> system('python scriptPath')

For testing, I decided to try finding breaking points:

print('----- Starting numpy Import! -----')
import numpy as np
print('----- Finished numpy Import! -----')

print('----- Starting pandas Import! -----')
import pandas as pd
print('----- Finished pandas Import! -----')

print('----- Starting tensorflow Import! -----')
import tensorflow
print('----- Finished tensorflow Import! -----')

Which in Python outputs as expected

----- Starting numpy Import! -----
----- Finished numpy Import! -----
----- Starting pandas Import! -----
----- Finished pandas Import! -----
----- Starting tensorflow Import! -----
----- Finished tensorflow Import! -----

Running this in MATLAB gives the folowing

2020-08-25 10:55:26.059626: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'cudart64_101.dll'; dlerror: cudart64_101.dll not found
2020-08-25 10:55:26.060129: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
----- Starting numpy Import! -----
----- Finished numpy Import! -----
----- Starting pandas Import! -----
----- Finished pandas Import! -----
----- Starting tensorflow Import! -----
Traceback (most recent call last): 
  File "C:\Users\mitch\Documents\...\NNtest.py", line 29, in <module> 
    import tensorflow 
    ...

and follows the same error traceback as before.

This seems to maybe be a MATLAB/ tensorflow issue rather than a MATLAB/ Python issue...

[Mohammad Sami]

Can you try outofprocess execution ?

pyenv("ExecutionMode","OutOfProcess");

[Mitchell Metzger]

No luck with the out of process execution... I ran the function above at startup and proceeded as I had before. If I made an error in using the out of process execution, please let me know.

[Peter O]

Try launching MATLAB from a Conda Powershell prompt using your configured environment? With some of the DLLs (numpy for sure, haven't played with cuda stuff), the Anaconda distributions place some of the compiled libraries in non-standard locations.

1. Launch Anaconda Navigator
2. Powershell (or command) Prompt
3. >> conda activate PythonGPU (or whatever your environment is named)
4. >> matlab

Does that do anything?

[Rik]

Comment posted as flag by sadik satir:

This works for me. Thanks.

Answer 1

[Mohammad Sami]

I keep getting errors when using InProcess mode. However managed to succesfully load the modules using OutOfProcess mode, with python 3.7 on Matlab R2020a.

Supported Python release was installed from https://www.python.org/downloads/

And I used pip install to install tensorflow.

To test if python is working, try to run the following imports

% check the python version supported by your release of MATLAB
pyenv('Version','3.7','ExecutionMode','OutOfProcess');
tfmod = py.importlib.import_module('tensorflow');

If they manage to load without error, your python function should be able to run.

Entire test setup is as follows.

pyenv('Version','3.7','ExecutionMode','OutOfProcess');
tfmod = py.importlib.import_module('tensorflow'); % seems without importing first, the following does notwork
testmod = py.importlib.import_module('mypythonpackage.test');
testmod.test();

I had the following version of the package installed.

pip list
Package                Version
---------------------- ---------
absl-py                0.10.0
astor                  0.8.1
astunparse             1.6.3
cachetools             4.1.1
certifi                2020.6.20
chardet                3.0.4
gast                   0.3.3
google-auth            1.21.0
google-auth-oauthlib   0.4.1
google-pasta           0.2.0
grpcio                 1.31.0
h5py                   2.10.0
idna                   2.10
importlib-metadata     1.7.0
Keras-Preprocessing    1.1.2
Markdown               3.2.2
numpy                  1.19.1
oauthlib               3.1.0
opt-einsum             3.3.0
pip                    20.2.2
protobuf               3.13.0
pyasn1                 0.4.8
pyasn1-modules         0.2.8
PyYAML                 5.3.1
requests               2.24.0
requests-oauthlib      1.3.0
rsa                    4.6
scipy                  1.4.1
setuptools             50.0.0
six                    1.15.0
tensorboard            2.2.2
tensorboard-plugin-wit 1.7.0
tensorflow             2.2.0
tensorflow-estimator   2.2.0
termcolor              1.1.0
urllib3                1.25.10
Werkzeug               1.0.1
wheel                  0.35.1
wrapt                  1.12.1
zipp                   3.1.0

[Mohammad Sami]

Test code in the attached zip file was copied from : https://github.com/keras-team/keras-io/blob/master/guides/sequential_model.py

[Aaruththiran Manoharan]

I am able to run my neural network model in matlab using this method...looks like we need to include this:

py.importlib.import_module('tensorflow');

before calling the python files....thanks!