Catch – multi-paradigm automated test framework for C++

When unit testing in C++ I tend to use Google Test/Mock, it’s simple to use, multi-platform solution. In the last few days I have been having second thoughts about it due to a new framework called Catch.

What is (the) Catch?

“Catch” stand for C++ Automated Test Cases in Headers. It’s an open source project (GitHubbed). What I like about it is that instead of implementing the old and trusted “[A..Z]Unit” way its creators have decided to rethink the way tests are written – with a few small changes that make a difference (more on that later).
Getting started with Catch is done by downloading catch.hpp and that’s it – you’re all set to go, no need for linking with a dll/lib which can be quite painful when working in C++ (where you sometimes need a different library for each release/debug, platform, single/multi-thread and day of the week).
But this quick & simple way to get you up and running is not the reason I find myself liking Catch – it’s the way the tests are written.

Taking it for a spin

I’ve wanted to try Catch and see how it rolls. For that I’ve decided on implementing the Bowling kata (if you don’t know what it is – go read about it, I’ll wait).
Writing the first test was simple and looked very similar to the tests of old:

TEST_CASE(Score_NewGameCreated_ScoreIsZero)
{
    Game game;
    int result = game.Score();

    REQUIRE(result == 0);
}

Noting big here – or is it? Have you noticed something different about my assert?
Catch has done something that the unit testing world should have done a long time ago – use code to describe what assert is needed instead of dozens of Assert.X methods (in fact I’ve been doing the same for .NET). When running the test I was pleasantly surprised to find the following error message:

See that – it tells me exactly what went wrong: I have a variable called result which does not equal ‘0’, in fact it’s ‘-1’. With test failures like this I can understand what went wrong – even without debugging my test.
I’ve wanted to see if I can get even better error message from Catch and so I moved the call to the method into the REQUIRE block and got the following error:

Even better – now I can really understand what’s going on – Game.Score is not zero.
I’ve continued my TDD experience and then I found the 2nd feature that makes Catch standout – Sections.
Usually I have a hard time with developers (ab)using Setup methods. I’ve wrote (read: ranted) about this before.
In a nutshell – I do not like setup methods because:

• After a while the setup method becomes hard to understand and tend to initialize fields that are only needed in some of the tests.
• It’s hard to fully understand a test that is split between two places (setup and actual test method)
• As an added bonus you also get shared state (fields again) that can (read: will) cause a lot of damage in the future (test A fail when run after test B)

And so I was happy to find out that Catch has initialization per test case and the actual tests are written in isolated sections:

TEST_CASE("Simple Bowling score -- without score or strike", "[BowlingKata]")
{
    Game game;

    REQUIRE(game.Score() == 0);

    SECTION("All rolls are gutter balls --> score == 0"){
        RollSeveral(game, 20, 0);

        REQUIRE(game.Score() == 0);
    }

    SECTION("All rolls are 1 --> score == 20"){
        RollSeveral(game, 20, 1);

        REQUIRE(game.Score() == 20);
    }

    SECTION("All rolls are 2 --> score == 40"){
        RollSeveral(game, 20, 2);

        REQUIRE(game.Score() == 40);
    }
}

not sure if I like the fact that I’m also asserting something about the initial state of the tests – seems a very un-AAA thing to do and it confuses the hack out of the test results (says it has 2 X 3 == 6 assertions) so I’ll remove it (and create a test just for this scenario).
When I first saw similar code in the tutorial I almost decided not to try Catch at all – it seems like pouring code instead of writing atomic, isolated tests. In order to really appreciate what Catch does I’ll make one of the tests fail:

We’re looking at three different tests – which means that if one of the sections fails – it does not effect the other two. This ability enable us to write common code in the beginning of the test case and then write different sections for each test (case?).
Have you noticed another cool feature on the way? The “tests” names are simple texts – we’re getting to a point in which naming a method with underscores and language trickeries will become obsolete.

Catching a case of BDD

Using macros Catch also cater to the BDD crowd, I’ve decided to write the rest of the tests using BDD style:

SCENARIO("If on his first try in the frame he knocks down all the pins, this is called a strike. His turn is over, and his score for the frame is ten plus the simple total of the pins knocked down in his next two rolls.")
{
    GIVEN("Bowled strike on first turn")
    {
        Game game;
        game.Roll(10);

        WHEN("All rest rolls are gutter balls")
        {
            RollSeveral(game, 18, 0);
            THEN("Total score is 10")
            {
                REQUIRE(game.Score() == 10);
            }
        }

        WHEN("Next two rolls are not spare or strike")
        {
            game.Roll(3);
            game.Roll(4);
            RollSeveral(game, 16, 0);

            THEN("Total score is 10 plus twice the sum of these rolls")
            {
                REQUIRE(game.Score() == 24);
            }
        }        
    }
}

Again We have a common GIVEN step and two isolated When-Then clauses. When one of the behaviors fails we get a nice BDD style error message:

Conclusion

You might have noticed that I like Catch – but it’s not all a garden of roses. I miss having parameterized tests which I need for several of my projects and currently there is no Visual Studio test runner to be seen both of which are a deal breaker for some of the projects I’m involved with.
Since it’s open source there’s a chance that we’ll get an implementation for it in the future.
I know I want to use Catch as much as I can because of features such as sections, test “names”, one assert (require) and error messages.

Happy coding…