It depends on what "the function does not respond" exactly means. If it stops with an error message, use try catch to catch the error and the corresponding message. If a specific output is replied, simply obtain this output, when you call the function from the callback of the GUI. Afterwards you can display the results how ever you want.
If is exactly the same situation, when you call a built-in function like e.g. sin(). When the input is a string, an error is thrown. When invalid input values are provided, e.g. NaN, NaN is replied also, such that you can catch this by checking the output.
Finally I assume, the actual question is not clear yet.