Write Plugin to Save Diagnostic Details

This example shows how to create a custom plugin to save diagnostic details. The plugin listens for test failures and saves diagnostic information so you can access it after the framework completes the tests.

Create Plugin

In a file in your working folder, create a class, myPlugin, that inherits from the matlab.unittest.plugins.TestRunnerPlugin class. In the plugin class:

  • Define a FailedTestData property on the plugin that stores information from failed tests.

  • Override the default createTestMethodInstance method of TestRunnerPlugin to listen for assertion, fatal assertion, and verification failures, and to record relevant information.

  • Override the default runTestSuite method of TestRunnerPlugin to initialize the FailedTestData property value. If you do not initialize value of the property, each time you run the tests using the same test runner, failed test information is appended to the FailedTestData property.

  • Define a helper function, recordData, to save information about the test failure as a table.

The plugin saves information contained in the PluginData and QualificationEventData objects. It also saves the type of failure and timestamp.

classdef DiagnosticRecorderPlugin < matlab.unittest.plugins.TestRunnerPlugin
    
    properties
        FailedTestData
    end
    
    methods (Access = protected)
        function runTestSuite(plugin, pluginData)
            plugin.FailedTestData = [];
            runTestSuite@...
                matlab.unittest.plugins.TestRunnerPlugin(plugin, pluginData);
        end
        
        function testCase = createTestMethodInstance(plugin, pluginData)
            testCase = createTestMethodInstance@...
                matlab.unittest.plugins.TestRunnerPlugin(plugin, pluginData);
            
            testName = pluginData.Name;
            testCase.addlistener('AssertionFailed', ...
                @(~,event)plugin.recordData(event,testName, 'Assertion'));
            testCase.addlistener('FatalAssertionFailed', ...
                @(~,event)plugin.recordData(event,testName, 'Fatal Assertion'));
            testCase.addlistener('VerificationFailed', ...
                @(~,event)plugin.recordData(event,testName, 'Verification'));
        end
    end
    
    methods (Access = private)
        function recordData(plugin,eventData,name,failureType)
            s.Name = {name};
            s.Type = {failureType};
            if isempty(eventData.TestDiagnosticResult)
                s.TestDiagnostics = 'TestDiagnostics not provided';
            else
                s.TestDiagnostics = eventData.TestDiagnosticResult;
            end
            s.FrameworkDiagnostics = eventData.FrameworkDiagnosticResult;
            s.Stack = eventData.Stack;
            s.Timestamp = datetime;
            
            plugin.FailedTestData = [plugin.FailedTestData; struct2table(s)];
        end
    end
end

Create Test Class

In your working folder, create the file ExampleTest.m containing the following test class.

classdef ExampleTest < matlab.unittest.TestCase
    methods(Test)
        function testOne(testCase)
            testCase.assertGreaterThan(5,10)
        end
        function testTwo(testCase)
            wrongAnswer = 'wrong';
            testCase.verifyEmpty(wrongAnswer,'Not Empty')
            testCase.verifyClass(wrongAnswer,'double','Not double')
        end
        
        function testThree(testCase)
            testCase.assertEqual(7*2,13,'Values not equal')
        end
        function testFour(testCase)
            testCase.fatalAssertEqual(3+2,6);
        end
    end
end

The fatal assertion failure in testFour causes the framework to halt and throw an error. In this example, there are no subsequent tests. If there was a subsequent test, the framework would not run it.

Add Plugin to Test Runner and Run Tests

At the command prompt, create a test suite from the ExampleTest class, and create a test runner.

import matlab.unittest.TestSuite
import matlab.unittest.TestRunner

suite = TestSuite.fromClass(?ExampleTest);
runner = TestRunner.withNoPlugins;

Create an instance of myPlugin and add it to the test runner. Run the tests.

p = DiagnosticRecorderPlugin;
runner.addPlugin(p)
result = runner.run(suite);
Error using ExampleTest/testFour (line 16)
Fatal assertion failed.

With the failed fatal assertion, the framework throws an error, and the test runner does not return a TestResult object. However, the DiagnosticRecorderPlugin stores information about the tests preceding and including the test with the failed assertion.

Inspect Diagnostic Information

At the command prompt, view information about the failed tests. The information is saved in the FailedTestData property of the plugin.

T = p.FailedTestData
T =

  5×6 table

             Name                    Type                  TestDiagnostics                                                                                                                                                                FrameworkDiagnostics                                                                                                                                                           Stack             Timestamp      
    _______________________    _________________    ______________________________    ____________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________    ____________    ____________________

    'ExampleTest/testOne'      'Assertion'          'TestDiagnostics not provided'    'assertGreaterThan failed.↵--> The value must be greater than the minimum value.↵↵Actual Value:↵     5↵Minimum Value (Exclusive):↵    10'                                                                                                                                                                                       [1x1 struct]    17-Jul-2017 12:41:18
    'ExampleTest/testTwo'      'Verification'       'Not Empty'                       'verifyEmpty failed.↵--> The value must be empty.↵--> The value has a size of [1  5].↵↵Actual char:↵    wrong'                                                                                                                                                                                                                  [1x1 struct]    17-Jul-2017 12:41:18
    'ExampleTest/testTwo'      'Verification'       'Not double'                      'verifyClass failed.↵--> The value's class is incorrect.↵    ↵    Actual Class:↵        char↵    Expected Class:↵        double↵↵Actual char:↵    wrong'                                                                                                                                                                        [1x1 struct]    17-Jul-2017 12:41:18
    'ExampleTest/testThree'    'Assertion'          'Values not equal'                'assertEqual failed.↵--> The values are not equal using "isequaln".↵--> Failure table:↵        Actual    Expected    Error      RelativeError   ↵        ______    ________    _____    __________________↵    ↵        14        13          1        0.0769230769230769↵↵Actual Value:↵    14↵Expected Value:↵    13'         [1x1 struct]    17-Jul-2017 12:41:18
    'ExampleTest/testFour'     'Fatal Assertion'    'TestDiagnostics not provided'    'fatalAssertEqual failed.↵--> The values are not equal using "isequaln".↵--> Failure table:↵        Actual    Expected    Error      RelativeError   ↵        ______    ________    _____    __________________↵    ↵        5         6           -1       -0.166666666666667↵↵Actual Value:↵     5↵Expected Value:↵     6'    [1x1 struct]    17-Jul-2017 12:41:18

There are many options to archive or post-process this information. For example, you can save the variable as a MAT-file or use writetable to write the table to various file types, such as .txt, .csv, or .xls.

View the stack information for the third test failure

T.Stack(3)
ans = 

  struct with fields:

    file: 'C:\Work\ExampleTest.m'
    name: 'ExampleTest.testTwo'
    line: 9

Display the diagnostics that the framework displayed for the fifth test failure.

celldisp(T.FrameworkDiagnostics(5))
ans{1} =
 
fatalAssertEqual failed.
--> The values are not equal using "isequaln".
--> Failure table:
        Actual    Expected    Error      RelativeError   
        ______    ________    _____    __________________
    
        5         6           -1       -0.166666666666667

Actual Value:
     5
Expected Value:
     6

See Also

| | |

Topics