Test coverage#

See also:

Directory structure#

Tests usually grouped by additional folders that groups tests according to their purpose.

Directory

Purpose

benchmarks

Any kind of benchmarks should go here.

component_tests

Any module-level tests should go here.

fixtures

Any fixture that is shared by more that one test suite should go here.

mocks

Any non-public test double should go here.

unit_tests

Any unit-related tests should go here.

See also:

Test suite & Test plan#

Tests can be grouped into test suite. Each root test suite should have test plan. Usually test plan is written for entire test file despite there can be several test suites. Usually test file has single root test suite and several nested ones.

Test plan conventions#

  • Test plan should be located before any test within test suite.

  • Test plan should contain each test case title from test suite.

  • It should be formed with normal English word (without underscores).

  • Each test should be marked as Done or Fail (or other state).

/*----------------------------------------------------------------------------*

Test plan:

Done    1. C-tor
Done    2. Empty object
Done    3. Reserve
Done    4. Add
            4.1. New entity
Fail        4.2. Existing entity

*----------------------------------------------------------------------------*/

Warning

Test plan should be always synchronized with tests inside suite!

Test case#

Structure#

Test case should be well structured to easy read, modify and add new tests. Usually each test case contains four sections:

Section

Purpose

Setup

Here we usually prepare test to run: setup input data, initialize environment, etc.

Action

Here we usually execute target to perform target behavior and capture all outputs.

Check

Here we usually check output values to be correct.

Cleanup

Here we usually restore system to pre-test state to run next tests.

Avoid explicit Cleanup phase. Use RAII to ensure all resources are uninitialized.

Test should not depend on other tests. It should be possible to run each test separately.

Each test case has name and corresponding number as a postfix. It should be well-formed as in test plan. Pascal style should be used for test names. Each logical level should be separated by underscore. Test number is located at the end of test case name. Each test case name should be unique within single test suite.

BOOST_LOCATED_AUTO_TEST_CASE(Add_NewEntity_4_1)

It is not recommended to use other styles for test case name because of poor readability.

Optionally each test case phase can be highlighted by comment header.

BOOST_AUTO_TEST_CASE(Object_Name_1_1)
{
    //--------------------------------- Init ---------------------------------//

    auto pObject = createObject();
    pUpdate->setName("MyObjectName");

    // ...

    //-------------------------------- Check ---------------------------------//

    BOOST_CHECK(pObject->isNameSet());
    BOOST_CHECK_EQUAL(pObject->getName(), "MyObjectName");

    // ...

    BOOST_CHECK(pObject->isValid());
}

Fixtures#

Fixture should be used to avoid code duplication inside each test case. Also any initialization/uninitialized should be performed inside test fixture. Keep test case code clean.

Test assertions#

Single test should check single behavior. Do not add multiply Action and Check sections within single test case. It is recommended to use assertion macros such as BOOST_CHECK, BOOST_REQUIRE only within test case, but not within fixtures. Each test assertion macro setups location for error to easy find what check was failed. If condition is not enough to understand what is wrong, use BOOST_CHECK_MESSAGE (and so on) assertion macros.

Note

Use BOOST_CHECK if concrete assertion do not influence on following checks.

Note

Use BOOST_REQUIRE for critical assertions.

BOOST_REQUIRE_EQUAL(result.size(), 2);
BOOST_CHECK_EQUAL(result.getLibrary(0).getName(), "MySuperLibrary");

You may want to read the documentation before using any test assertions macros.

Temporary failed tests#

Do not comment any tests. It is recommended to change expectation to failed current behavior with proper comment that this expectation is wrong and should be revised.

// BOOST_CHECK_EQUAL(result.getValue(), "a"); // Fix from Ticket-1234 is required
BOOST_CHECK_EQUAL(result.getValue(), "b");

Unit-Testing Private C++ Functions#

Caution

This section has been based on LSST’s C++ testing guide. Please respect the creators and contributor of the original guide as they have done an external work.

Note

This material is still in draft.

Warning

There exists significant debate about whether or not private functions should ever be unit-tested. In most situations, it should be preferable to test the public functions of the class, and to use coverage analysis to make sure that private functions are tested as well.

However, there may exist circumstances in which testing and debugging is considerably simplified by testing a private class. For example, the private functions may be too complex to easily generate thorough coverage without calling them directly. In these cases, you may go ahead and directly test a private function.

The Problem with Boost Unit Testing Macros#

Boost’s unit testing macros create test classes at compile-time. The names of these classes are not known to the programmer, and thus it is not possible to simply declare them as “friend” to the tested class.

Some quick searching on Google will reveal some dubious ideas, such as include guards which re-declare the private functions as public if a “testing” flag is set. This will mean that code compiled for production will not pass tests. This is not an optimal solution.

How to do it, if you must#

It is suggested to create an additional “bridge” class which is declared friend to the tested class. This “bridge” class can then call the private functions of the tested class, and the unit tests should call an instance of the “bridge” class.