Group C/C++ Tests into Suites with Common Setup and Teardown Code

You can group multiple tests that you write with the Polyspace^® Test™ xUnit API into a suite of tests. You can group all tests on the same function together, or use another criteria for grouping.

A test suite allows you to:

• Group tests that are semantically related, for example, all tests for a given source code component.

• Define a setup function that runs once before any test in the suite executes, and a teardown function that runs once after all tests complete execution.

  For example, you can run a function that reads the contents of a file into dynamically allocated memory before any test in a suite executes, and release the memory after the tests complete.

• Define a setup function that runs before each test in the suite executes, and a teardown function that runs after each test.

  For example, you can set global variables before executing each test and reset them after each test.

By grouping tests into suites and writing common setup and teardown functions, you avoid duplicating setup and teardown code. You can also ensure that steps that are common to a group of tests are executed consistently.

This example shows how to write a suite of tests with common setup and teardown functions.

Prerequisites

This topic describes test authoring using the Polyspace Test xUnit API. To compile these tests, you are required to know some file paths in advance. For your convenience, you can define environment variables to stand for the file paths, or otherwise include the file paths in your build. For more information, see Set Up C/C++ Testing and Code Profiling Using Self-Managed Builds.

Workflow

If you define a test as part of a test suite, use the macros PST_TEST_CONFIG and PST_TEST_BODY to define the test and PST_ADD_TEST to register the test. (A test that is not part of a test suite is defined using the macro PST_SIMPLE_TEST and registered using the macro PST_ADD_SIMPLE_TEST.)

To define a test suite with a configuration (that is, a setup and teardown function):

1. Write the definitions of the setup and teardown functions.

   Setup and teardown functions must have the signature:

   void funcName(void);

2. Specify these setup and teardown functions in the test suite configuration using the PST_SUITE_CONFIG macro:

   PST_SUITE_CONFIG(suiteName) {
       PST_SUITE_SETUP(setupFuncName);
       PST_SUITE_TEARDOWN(teardownFuncName);
   }
   

   Note that:

   • You can define a test suite without a configuration using the PST_SUITE macro instead of the PST_SUITE_CONFIG macro, for example:

     PST_SUITE(NewSuite);

   • The macro PST_SUITE_SETUP specifies a setup function that runs once before all tests in a suite. To specify a setup function that runs before each test in a suite, use the macro PST_SUITE_TEST_SETUP. Likewise, to specify a teardown function that runs after each test in a suite, use the macro PST_SUITE_TEST_TEARDOWN. For example:

     PST_SUITE_CONFIG(suiteName) {
         PST_SUITE_TEST_SETUP(setupFuncName);
         PST_SUITE_TEST_TEARDOWN(teardownFuncName);
     }
     

   • You can define tests as part of a suite that you define in another file. Before defining the tests, declare an external suite using the macro PST_SUITE_EXTERN, for example:

     PST_SUITE_EXTERN(suiteName);

3. Define tests that are part of the suite by using the macros PST_TEST_CONFIG and PST_TEST_BODY (or simply the macro PST_TEST if you do not require test-specific configuration). Both macros take the name of the test suite as the first argument and the name of the test as the second:

   PST_TEST_CONFIG(suiteName, testName) {
     ...
   }
   PST_TEST_BODY(suiteName, testName) {
     ...
   }

   If you require test-specific setup and teardown functions, you can specify them in the PST_TEST_CONFIG macro using the macros PST_SETUP and PST_TEARDOWN. For example, to specify the setup function setupFuncName and teardown function teardownFuncName, define the test configuration as:

   PST_TEST_CONFIG(suiteName, testName) {
       PST_SETUP(setupFuncName);
       PST_TEARDOWN(teardownFuncName);
   }

4. Register the tests by using the macros PST_ADD_TEST inside a PST_REGFCN macro, for example:

   PST_REGFCN(regFuncName) {
       PST_ADD_TEST(suiteName, testName);
   }

Example

Example Files

Find the files for this tutorial in the folder polyspaceroot\polyspace\examples\doc_pstest\test_suites_setup_teardown. Copy these files to a writable location and continue the tutorial. Here, polyspaceroot is the Polyspace installation folder, for example, C:\Program Files\Polyspace\R2026a.

Inspect Function Under Test and Requirements

The function globalSum adds its argument to the previous value of a global variable gSum. The sum is saturated at UINT_MAX.

#include "example.h"

unsigned gSum;

SUM_STATUS globalSum(unsigned x) {
    if (x > UINT_MAX - gSum) {
        gSum = UINT_MAX;
        return SATURATED;
    }
    gSum += x;
    return NOT_SATURATED;
}

Save the function in a file example.c.

The enumeration SUM_STATUS with two values, SATURATED and NOT_SATURATED, is defined in a header file named example.h.

#include <limits.h>

extern unsigned gSum;

typedef enum {
    NOT_SATURATED = 0,
    SATURATED = 1
} SUM_STATUS;

SUM_STATUS globalSum(unsigned x);

Suppose that you want to test if the global variable gSum is updated correctly after the function globalSum is called a certain number of times. You can start from a known value of gSum before running each test and reset the value afterward.

In this case, it is convenient to create a suite of tests with a common setup and teardown function.

Write Test

The following test defines a test suite gSumTests with three test cases:

• TestCaseOne, which calls the function globalSum once with an argument that does not result in saturation of gSum.

• TestCaseTwo, which calls the function globalSum twice. The second call results in saturation of gSum.

• TestCaseThree, which calls the function globalSum three times. The second call results in saturation of gSum.

All three test cases start with a value of gSum equal to 0 and reset the value of gSum to zero afterward. The test cases use a setup and teardown function setgSum and resetgSum that you specify in the test suite configuration.

#include <pstunit.h>
#include "example.h"

//Test utilities
void setgSum (void) {
    gSum = 0;
}

void resetgSum (void) {
    gSum = 0;
}


PST_SUITE_CONFIG(gSumTests) {
    PST_SUITE_TEST_SETUP(setgSum);
    PST_SUITE_TEST_TEARDOWN(resetgSum);
}

PST_TEST(gSumTests, TestCaseOne) {
    unsigned test_data = UINT_MAX/2 + 1;
    unsigned expected_gSum = UINT_MAX/2 + 1;
    
     for(int i = 1; i<= 1; i++)    
            globalSum(test_data);
    
    PST_VERIFY_EQ_INT_MSG(gSum, expected_gSum, "Issue in nonsaturating sum.");
      
}

PST_TEST(gSumTests, TestCaseTwo) {
    unsigned test_data = UINT_MAX/2 + 1;
    unsigned expected_gSum = UINT_MAX;
    
    for(int i = 1; i<= 2; i++)    
            globalSum(test_data);
    
    PST_VERIFY_EQ_INT_MSG(gSum, expected_gSum, "Issue in sum that is just above limit.");

}

PST_TEST(gSumTests, TestCaseThree) {
    unsigned test_data = UINT_MAX/2 + 1;
    unsigned expected_gSum = UINT_MAX;
        
    for(int i = 1; i<= 3; i++)    
            globalSum(test_data);
    
    PST_VERIFY_EQ_INT_MSG(gSum, expected_gSum, "Issue in sum that is well above limit.");
}


PST_REGFCN(myRegFcn) {
    PST_ADD_TEST(gSumTests, TestCaseOne);
    PST_ADD_TEST(gSumTests, TestCaseTwo);
    PST_ADD_TEST(gSumTests, TestCaseThree);
}

#ifndef PSTEST_BUILD
int main(int argc, char *argv[]) {
    PST_REGFCN_CALL(myRegFcn);
    return PST_MAIN(argc, argv);
}
#endif

Save this code in a file test.c.

The test consists of these macros from the Polyspace Test xUnit API:

• PST_SUITE_CONFIG – This macro defines the test suite gSumTests and its configuration.

• PST_SUITE_TEST_SETUP and PST_SUITE_TEST_TEARDOWN – These macros specify a common setup and teardown function to be run before and after every test case in the suite.

• PST_TEST: – This macro defines the test body.

• PST_ADD_TEST – This macro registers a test that is part of the suite gSumTests.

Execute Test and Inspect Results

Compile the files example.c and test.c along with files that ship with Polyspace:

gcc example.c test.c <PSTUNIT_SOURCE> -I <PSTUNIT_INCLUDE> -o testrunner

For more information on the file paths <PSTUNIT_SOURCE> and <PSTUNIT_INCLUDE>, see Set Up C/C++ Testing and Code Profiling Using Self-Managed Builds.

Run the test executable:

testrunner.exe

(or ./testrunner in Linux^®).

You see that all three tests in the suite pass.

| Running tests   |            |
|-----------------|------------|
| Running         | suite      | gSumTests | test_with_suite_global_setup.c
|-----------------|------------|
| Running         | test       | gSumTests/TestCaseOne
|            PASS | test       | gSumTests/TestCaseOne
| Running         | test       | gSumTests/TestCaseTwo
|            PASS | test       | gSumTests/TestCaseTwo
| Running         | test       | gSumTests/TestCaseThree
|            PASS | test       | gSumTests/TestCaseThree
|-----------------|------------|
|            PASS | suite      | gSumTests

|        |  Total | Passed | Failed | Incomplete
|--------|--------|--------|--------|------------
| Suites |      1 |      1 |      0 |          0
|  Tests |      3 |      3 |      0 |          0

Modify the line:

unsigned expected_gSum = UINT_MAX;

in test TestCaseThree to:

unsigned expected_gSum = 0;

(If you copied the files that ship with Polyspace Test, you might have to enable write permissions to the file.)

When you recompile and rerun the tests, you see that the test TestCaseThree, and therefore the suite gSumTests fails:

| Running tests   |            |
|-----------------|------------|
| Running         | suite      | gSumTests | test_with_suite_global_setup.c
|-----------------|------------|
| Running         | test       | gSumTests/TestCaseOne
|            PASS | test       | gSumTests/TestCaseOne
| Running         | test       | gSumTests/TestCaseTwo
|            PASS | test       | gSumTests/TestCaseTwo
| Running         | test       | gSumTests/TestCaseThree
|     VERIFY FAIL |            |
test_with_suite_global_setup.c:56: Verify failed
Issue in sum that is well above limit.
Expression '(pst_long_long_t)(gSum) == (pst_long_long_t)(expected_gSum)' evaluated to false
lhs="4294967295" rhs="0"
|            FAIL | test       | gSumTests/TestCaseThree
|-----------------|------------|
|            FAIL | suite      | gSumTests

|        |  Total | Passed | Failed | Incomplete
|--------|--------|--------|--------|------------
| Suites |      1 |      0 |      1 |          0
|  Tests |      3 |      2 |      1 |          0

Instead of running all tests in a suite, you can run a subset of tests. To run only TestCaseOne in suite gSumTests, for example, enter:

testrunner.exe -test gSumTests/TestCaseOne

(or ./testrunner -test gSumTests/TestCaseOne in Linux). To see all tests that can be run using the test executable, enter:

testrunner.exe -list

(or ./testrunner -list in Linux).

Further Exploration: Organizing Tests

The preceding example defines a test suite with several tests in one file. You can also define a test suite in several files.

1. In one of the files, define the test suite configuration. You can optionally add some tests to the suite in the file itself.

2. In the remaining files, declare the test suite as external and add more tests to the suite.

In a new file, add a test TestCaseFour to the previously defined suite gSumTests,. You can also move the main function into a separate file. That way, each time you create a new file for adding tests, you have to modify only the file containing the main function.

The modifications you have to make to the preceding example are:

• Add a new test TestCaseFour in a separate file. In this file, declare the suite gSumTests as an external test suite using the PST_SUITE_EXTERN macro. Define the test TestCaseFour as part of this suite.

  The contents of the file with the new test are shown below. The test also uses a test-specific configuration that runs after the suite configuration and, in this case, overrides the initial value set in the suite configuration. A new registration function myRegFcn2 registers the new test.

  #include <pstunit.h>
  #include "example.h"
  
  void setgSumNonZero (void) {
      gSum = UINT_MAX/2 + 1;
  }
  void resetgSum (void); //Defined in other test file
  
  PST_SUITE_EXTERN(gSumTests);
  
  PST_TEST_CONFIG(gSumTests, TestCaseFour){
      PST_SETUP(setgSumNonZero);
      PST_TEARDOWN(resetgSum);
  }
  
  PST_TEST_BODY(gSumTests, TestCaseFour) {
      unsigned test_data = UINT_MAX/2 + 1;
      unsigned expected_gSum = UINT_MAX;
      
       for(int i = 1; i<= 1; i++)    
              globalSum(test_data);
      
      PST_VERIFY_EQ_INT_MSG(gSum, expected_gSum, "Issue in sum if not starting from zero.");
        
  }
  
  PST_REGFCN(myRegFcn2) {
      PST_ADD_TEST(gSumTests, TestCaseFour);
  }

  Save this code in a file test2.c.

• Move the main function from the file test.c into a new file (and rename the file test.c to test1.c). The new main function calls the previous registration function myRegFcn from the file test.c (now test1.c) and the new registration function myRegFcn2 from the file test2.c.

  The file with the new main looks like this:

  #include <pstunit.h>
  #include "example.h"
  
  #ifndef PSTEST_BUILD
  int main(int argc, char *argv[]) {
      PST_REGFCN_CALL(myRegFcn);
      PST_REGFCN_CALL(myRegFcn2);
      return PST_MAIN(argc, argv);
  }
  #endif
  

  Save this code in a file test_main.c.

Compile the new files, test1.c, test2.c, and test_main.c, along with the source file:

gcc example.c test1.c test2.c test_main.c <PSTUNIT_SOURCE> -I <PSTUNIT_INCLUDE> -o testrunner

For more information on the file paths <PSTUNIT_SOURCE> and <PSTUNIT_INCLUDE>, see Set Up C/C++ Testing and Code Profiling Using Self-Managed Builds.

When you recompile and run the tests, you see that TestCaseFour runs as part of the test suite.