I was able to reproduce this issue if vgg.m exists on somewhere in MATLAB search path.
Could you check you have custom vgg16.m?
If it returns only the following,
MATLAB_INSTALL\toolbox\nnet\cnn\spkgs\vgg16.m % Shadowed
it will work well.
But if there is another vgg16.m and it is a script not a function, it will occur "Execution of script vgg16 as a function is not supported:" error.
Try renaming you custom vgg16.m to vgg16_custom.m and then, compiling your code again.