Home | Libraries | People | FAQ | More |
While in a perfect world all test assertions should pass in order for a test module to pass, in some situations it is desirable to temporarily allow particular tests to fail. For example, where a particular feature is not implemented yet and one needs to prepare a library for the release or when particular test fails on some platforms. To avoid a nagging red box in regression tests table, you can use the expected failures feature.
This feature allows specifying an expected number of failed assertions per test unit. The value is specified during test tree construction, and can't be updated during test execution.
The feature is not intended to be used to check for expected functionality
failures. To check that a particular input is causing an exception to be
thrown use BOOST_<level>_THROW
family of testing tools.
The usage of this feature should be limited and employed only after careful consideration. In general you should only use this feature when it is necessary to force a test module to pass without actually fixing the problem. Obviously, an excessive usage of expected failures defeats the purpose of the unit test. In most cases it only needs be applied temporarily.
You also need to remember that the expected failure specification is per test case. This means that any failed assertion within that test case can satisfy the expected failures quota. Meaning it is possible for an unexpected failure to occur to satisfy this quota.
Note | |
---|---|
If an assertion at fault is fixed and passed while an expected failures specification still present, the number of failures becomes smaller than expected. The test is going to be reported as passed; instead, a warning message will be issued. |
The decorator expected_failures
defines the number
of assertions that are expected to fail within the corresponding test unit.
It is reported as failure when the number of failed assertions is greater
than the declared expected number of failures. If the number of failed assertions
is less than the number of expected failures a message is reported. The total
number of expected failures for a given test suite S
is the sum of the declared expected failures in S
and the sum of expected failures in all nested test units:
Code |
---|
#define BOOST_TEST_MODULE decorator_10 #include <boost/test/included/unit_test.hpp> namespace utf = boost::unit_test; BOOST_AUTO_TEST_SUITE(suite1, * utf::expected_failures(1)) BOOST_AUTO_TEST_CASE(test1, * utf::expected_failures(2)) { BOOST_TEST(false); BOOST_TEST(false); } BOOST_AUTO_TEST_CASE(test2) { BOOST_TEST(false); BOOST_TEST(false); } BOOST_AUTO_TEST_SUITE_END() |
Output |
---|
> decorator_10 Running 2 test cases... test.cpp(11): error: in "suite1/test1": check false has failed test.cpp(12): error: in "suite1/test1": check false has failed test.cpp(17): error: in "suite1/test2": check false has failed test.cpp(18): error: in "suite1/test2": check false has failed *** 4 failures are detected (3 failures are expected) in the test module "decorator_10" > decorator_10 --run_test=suite1/test1 Running 1 test case... test.cpp(11): error: in "suite1/test1": check false has failed test.cpp(12): error: in "suite1/test1": check false has failed *** No errors detected |
In the above example, we first run all test cases with four failed assertions.
The total number of expected failures is 3: 1 (for test suite1
)
+ 2 (for test1
). Because
the expected failure count is exceeded, the error is reported. In the second
case, we only run test case suite1/test1
:
two failures occur, two failures are expected, therefore no error is reported.
Caution | |
---|---|
this usage is considered as deprecated. Please consider using the |
For backwards compatibility, it is possible to indicate the expected failures
with BOOST_AUTO_TEST_CASE_EXPECTED_FAILURES
[15] before the test case definition.
BOOST_AUTO_TEST_CASE_EXPECTED_FAILURES(test_case_name, number_of_expected_failures);
You can use this macro both on a file scope and inside a test suite. Moreover
you can use it even if name of test units coincide in different test suites.
Expected failures specification applies to the test unit belonging to the
same test suite where BOOST_AUTO_TEST_CASE_EXPECTED_FAILURES
resides.
Code |
---|
#define BOOST_TEST_MODULE example #include <boost/test/included/unit_test.hpp> BOOST_AUTO_TEST_CASE_EXPECTED_FAILURES( my_test1, 1 ) BOOST_AUTO_TEST_CASE( my_test1 ) { BOOST_TEST( 2 == 1 ); } BOOST_AUTO_TEST_SUITE( internal ) BOOST_AUTO_TEST_CASE_EXPECTED_FAILURES( my_test1, 2 ) BOOST_AUTO_TEST_CASE( my_test1 ) { BOOST_CHECK_EQUAL( sizeof(int), sizeof(char) ); BOOST_CHECK_EQUAL( sizeof(int*), sizeof(char) ); } BOOST_AUTO_TEST_SUITE_END() |
Output |
---|
> example --report_level=short Running 2 test cases... test.cpp(10): error in "my_test1": check 2 == 1 has failed test.cpp(21): error in "my_test1": check sizeof(int) == sizeof(char) has failed [4 != 1] test.cpp(22): error in "my_test1": check sizeof(int*) == sizeof(char) has failed [4 != 1] Test suite "example" passed with: 3 assertions out of 3 failed 3 failures expected 2 test cases out of 2 passed |
Caution | |
---|---|
this usage is considered as deprecated. Please consider using the |
To set the value of expected failures for the manually registered test unit
pass it as a second argument for the test_suite::add
call during test unit registration.
Code |
---|
#include <boost/test/included/unit_test.hpp> using namespace boost::unit_test; void free_test_function() { BOOST_TEST( 2 == 1 ); } test_suite* init_unit_test_suite( int, char* [] ) { framework::master_test_suite(). add( BOOST_TEST_CASE( &free_test_function ), 2 ); return 0; } |
Output |
---|
> example --log_level=message Running 1 test case... test.cpp(8): error in "free_test_function": check 2 == 1 has failed Test case has less failures then expected *** No errors detected |