How do I use matlab.uitest.TestCase to open my app once and perform multple function tests?

I'm using matlab.uitest.TestCase to test my app. I'm looking for a way to launch my app only once and then perform a variety of tests defined under methods (Test)

Answer 1

[Steven Lord]

You could do this by opening your app in a TestClassSetup method and storing a handle to it in a property of your test class, but this could lead to your tests violating the "Keep Tests Independent" principle.

Suppose you had two buttons, A and B. You want to unit test both buttons. So you write one test method named testA that clicks button A and another named testB that clicks button B. But the code that executes when you click button A disables button B. Depending on the order in which testA and testB run and whether or not they restore the state of the app back to how it was before they ran, testB may or may not fail. And if you tried to reproduce the failure (to investigate the root cause) by just running testB it would pass.

Heisenbugs are highly annoying to investigate.

[Tria Technologies]

Excellent! Thank you @Steven Lord

I somehow missed the TestClassSetup method, which is a "do once" method at the start of test, verses TestMethodSetup which runs for every function included in methods (Test)

You highlight exactly what I need to test. Complex apps often get "tangled" the more you use them, so I need to hit it with a sequence of tests to get it spun up.

[Steven Lord]

I would advise you to avoid using other test methods as the Setup phase of your four-phase test. There is no guarantee that test method testA will run before test method testB or vice versa. There's no guarantee that both testA and testB will run in the same execution of the test either; if you call runtests with the ProcedureName name-value argument you can tell MATLAB to run just testA or just testB.

If you need to run through the same set of steps to Setup the software under test (your app) for multiple test methods you could write a non-Test method or even a local function inside your test class file to perform those Setup steps and call the method or function at the start of your test. Alternately you could put those steps in a TestMethodSetup method. That ensures that the software under test is in a known good state when it comes to the Exercise phase of the Four-Phase Test.

[Tria Technologies]

Ah hah! I made the (wrong) assumption that my method tests would be run top-to-bottom. They have so far, but I'm glad you pointed out that sequential execution is not guaranteed. Thanks for clarifying! That probably saved me a few weeks of pain.

[Steven Lord]

The (very, very basic) test file tplus.m has two Test methods, onePlusOne and twoPlusTwo, but as you can see from the detailed output and the test results only one Test method ran. If twoPlusTwo had depended on some setup from onePlusOne it would have failed. But since they're independent I can run either or both with no problem.

runtests("tplus.m", ...
    ProcedureName="twoPlusTwo", ...
    OutputDetail="Detailed", ...
    LoggingLevel="Detailed")

 Running tplus
  Setting up tplus
  Done setting up tplus in 0 seconds
   Running tplus/twoPlusTwo
   Done tplus/twoPlusTwo in 0.011796 seconds
  Tearing down tplus
  Done tearing down tplus in 0 seconds
 Done tplus in 0.011796 seconds
__________

ans = 
  TestResult with properties:

          Name: 'tplus/twoPlusTwo'
        Passed: 1
        Failed: 0
    Incomplete: 0
      Duration: 0.0118
       Details: [1×1 struct]

Totals:
   1 Passed, 0 Failed, 0 Incomplete.
   0.011796 seconds testing time.

If I omit the ProcedureName argument you'll see both Test methods run.

runtests("tplus.m", ...
    OutputDetail="Detailed", ...
    LoggingLevel="Detailed")

 Running tplus
  Setting up tplus
  Done setting up tplus in 0 seconds
   Running tplus/onePlusOne
   Done tplus/onePlusOne in 0.003262 seconds
   Running tplus/twoPlusTwo
   Done tplus/twoPlusTwo in 0.002463 seconds
  Tearing down tplus
  Done tearing down tplus in 0 seconds
 Done tplus in 0.005725 seconds
__________

ans = 
  1×2 TestResult array with properties:

    Name
    Passed
    Failed
    Incomplete
    Duration
    Details

Totals:
   2 Passed, 0 Failed, 0 Incomplete.
   0.005725 seconds testing time.

[Tria Technologies]

Very helpful. I'm new to runtests, so the examples are great!

The help for runtests has this tip: "... use shared test fixtures in your tests and specify the input to the runtests function as a string array or cell array of character vectors, "

Do you have an example of this?

I foresee creating multiple test fixtures that share a common setup stage. Above, you recommended creating a non-test setup function that is called from each test fixture. That's pretty straightforward. Could the concept of "shared test fixtures" also solve the problem?

[Steven Lord]

This documentation page has an example of using shared test fixtures. The example runs two tests that each need to have a certain directory on the MATLAB search path. Instead of having each test add the directory to the path and restore the path afterwards they use a shared test fixture to add it to the path before either test runs and restore the path after both tests have run.

Another example where you might want to use a shared test fixture is to create or read a large data set to be used in multiple test files. Imagine the example on this documentation page but with a large matrix instead of a short user name.

You could have a shared test fixture that opens your app for multiple test files to use, but remember to Keep Tests Independent.

[Tria Technologies]

Great. Thanks again Steven!