Hello,
You can right click Flowmaster interface block->Properties->Callbacks->OpenFcn, you will find this OpenFcn callback will invoke FM_setup. This error message is originated from incorrectly invoking FM_setup, that means there is something wrong with Simulink & Flowmaster cosimulation environment setup.