Test Drum

14 05 2013

Since I’m working on two projects at the moment (both written in C#), and since both of them will end up as games (hopefully), I’ve decided I needed something else to wash away the taste of gameplay code and graphics code and physics code and whatever else kind of code a single game developer must write. So I’ve decided to write my own testing framework for C++! Why?! why not?!

Actually as far as the “why not” question goes I can give many reasons why not (and you, yes YOU, can probably give some more) but heck, this is my time and I want to try it. Let’s see how far I get into it before my eyes go cross 😛

As a first disclaimer I would like to say that I have already started this project a couple of times before just to test (hee hee, see what I did there) the water. As another disclaimer I would also like to mention that I’ve used and am using the wonderfully wonderful UnitTest++ and so am more than a little influenced by it. Having said that, “Test Drum” will not be a copy/paste affair but another approach to that same basic concept.

As I’ve mentioned just passing the framework’s name is ‘Test Drum’. Don’t ask me why since I’ve got no idea.

With that in mind let’s put down some ground rules;

Test Drum is designed to be a portable, lightweight
and fast framework for writing tests in C++

So, what’s a test?

A test [in the framework] is any function that contains
some expectations in it.

Next, what exactly is an expectation?

Expectations are code slices that test specific conditions
on input data; the expectation code should throw a
specialized exception (ExpectationFailed) if any of its
conditions are not met.

And finally the idea of the TestContext.

A TestContext is a mapping between a name and a 'Test' function.
A TestContext is executed in order to run its testing function.

I think that with these three basic ideas I can start writing the framework and later begin iteration and refinement. Already I know that I’m missing basic concepts such as timed tests, test fixtures and test suites. But those I’m happy to add on later once the groundwork is done.

This post is almost done but before I go I want set the corner stone of this framework. Namely, the actual test function, which is in fact nothing more than a function pointer.

typedef void (*Test)();

Well, that’s it for today.
“I may take a holiday in Spain, leave my wings behind me…”

Advertisements




Unit Tests v2.0

6 11 2012

Not the most profound title to be found out there but I couldn’t think of anything better. I’m a big proponent of testing and test-driven-development in particular. This enthusiasm, unfortunately, does not automatically make me good at writing tests or at driving my designs using tests – but I try. One of the things I find really hard about TDD is how to organize the tests into coherent units and, extrapolating from this, which tests should I write where. I’ve tried a few schemes and didn’t really feel comfortable with any of them (most likely an issue with me rather than the schemes themselves by the way). So lately, and I mean as of the following week or so, I’ve adopted the following pattern.

Quick aside, I’m using C# and the .Net version 4.0 along with NUnit and Visual Studio 2010.

First of I’ve decided to symbolically separate test classes [TestFixture] and methods [Test] from the rest of my code by using, *gasp*, underscores and all lower-case letters. Secondly I’ve started making instances of the class under test so that I can verify actual usage of the class itself and finally I’m making a namespace for each instance and a test fixture for each operation/property that I want to test under that instance.

It’s probably a bit hard to follow that explanation (and I probably will refine it to a set of written standards at some point) so let me give a real world example from my very own code!

/// <summery>
/// Testing the generated target configuration when using the
/// DefaultInternalBuildProperties.
/// </summer>
[TestFixture]
public class target_configuration
{
  [Test]
  public void should_be_release()
  {
    var instance = new ContentForTestProject();
    Assert.That(instance.Factory.GetTargetConfiguration(),
                Is.EqualTo("Release"));
  }
}

That’s the testing piece and it lives under the namespace of:

testing_default_internal_build_factory

Which means that when I’m reading the entire test description it generates the following string:

testing_default_internal_build_factory.target_configuration.should_be_release

Another point should also be made here concerning the following line in the testing code:

 var instance = new ContentForTestProject();

ContentForTestProject is a type of, ah, factory I guess you could call it, that generates for me an instance of the class being tested. The previous line of code will be repeated for each testing method simply because I’ve come to realize that this article is correct when using NUnit.

/// <summary>
/// Generates an instance of the DefaultInternalBuildProperties [Factory]
/// which will in turn fabricate those properties for some fake game
/// project in a folder called 'Test'.
/// </summary>
public class ContentForTestProject
{
  public readonly DefaultInternalBuildProperties Factory;

  public ContentForTestProject()
  {
    Factory = new DefaultInternalBuildProperties();
    Factory.Fabricate();
  }
}

This entire scheme might not be ideal or even used and maybe I’ll find it doesn’t scale very well down the line. But for now it sort of fits the way I view my code and it helps me keep track of the operations that I’m testing (that is the operations that I want my classes to perform).

“And we only smoke when bored, So we do two packs a day”





Mirror, Mirror

18 10 2012

Urgh… spent the last 4 hours tangling and untangling code using reflection on assemblies in C# and I think I’ve gone cross-eyed 😛

I’ve managed to do what I set out to do, which is to load a few assemblies and get all the classes with a certain attribute to them. It isn’t terribly hard to do I agree but the confusing part comes when I figured I can’t, easily, unload those assemblies because of the whole separation by AppDomain and not being able to unload specific assemblies within a single AppDomain without unloading the entire domain. Using Assembly.LoadFrom, by the way, load the input assembly quite well but into the AppDomain.CurrentDomain (I’ve tried to unload that during a test just to see what would happen… it made me laugh).

So, the code itself is fairly simple; load the assembly, get all types from that assembly with the particular attribute and then query the attribute for the required information.

[Test]
public void can_retrieve_attribute_properties()
{
  var assembly = Assembly.LoadFrom(File);
  var types =
    (from type in assembly.GetTypes()
    where Attribute.IsDefined(type, typeof(ContentImporterAttribute))
    select type).ToList();

  var first = types[0];
  var customAttributes =
    (from attribute in first.GetCustomAttributes(true)
    where attribute is ContentImporterAttribute
    select (ContentImporterAttribute)attribute).ToList();

  var importerAttribute = customAttributes[0];
  Assert.IsTrue(importerAttribute.FileExtensions.Contains(".edi"));
  Assert.AreEqual("EDI Importer", importerAttribute.DisplayName);
  Assert.AreEqual("TestProcessor", importerAttribute.DefaultProcessor);
}

There are a few caveats there; the first and most obvious one is that the test implies there is somewhere another assembly that has that particular attribute attached to a class in a very specific way (and in fact there is, I’ve made it myself for this test). The other little worry here is the fact that I’ve got no bound checking or indeed any null-assertions on referencing the first item in each list (I’m basically assuming that the first type is the one I’m looking for and that the first attribute on it is the one I’m looking for as well). But for me the biggest unknown here is the assembly loading behavior in a real-world context (which I shall test later) in regards to the whole AppDomain merry-go-round; after going through MSDN and StackOverflow for a couple of hours I did find some solutions but I’m not sure that the complexity involved with creating specialized domains is worth the trouble.

Anyway, even if this approach fails (can anyone guess what I’m trying to do? :P) I’ve got another way to go about the system which would work for sure but it does mean a bit more manual labor.

“…I cut down trees, I eat my lunch I go to the lavatory…”





TDDXNA

20 03 2011

Lots of letters…

As I’ve said before, using test driven development and unit tests on an established framework (XNA in this case) is fairly futile. I guess it can be done if I put my mind to it and lots of time to figure out how to mock every single aspect of the framework but I don’t think it’s worth it. In the end I have to trust that the framework does what it says it does and have my asserts, preconditions and postconditions act as the live tests for it; after all that’s what they’re there for.

Writting the sprite classes as data stores which describe the drawing/loading states of the actual images did prove to be the right course (so far anyway). It makes the code more natural to read and it allows me to apply these descriptions to multiple textures at once. The next module I’m going to throw myself at is the input and game commands, wonder how that’ll go?

Short post today, I’m sick and so is my little one…

“Lady Startdust sang his songs of darkness and dismay”





And I’m back!

16 03 2011

So, I’m not dead, not yet anyway; let’s just say I took a very looooong vacation. Which is a lie since I never take vacations and I’ve actually started full time work, but hey I’m allowed to dream.

The last time I’ve seen this blog it was actually still 2010 and I was writing tests for a timer class which is long been tested to death and it works… Honest… Trust me… Okay, alright, I’ll post the rest of the in the next couple of days once I reorganize and try to remember what the hell I was talking about. In the meanwhile I’m also glad to report that I’ve picked up two other projects to work on, both sort of have to do with Microsoft’s Dream.Build.Play (rather than Dream.Build.Debug which is a completely different thing altogether) and the upside of that is that I get to work with C# (which I have grown to love in the last few months) and XNA (which I have grown to like in the last two weeks). Well, being the lovable programmer that I am the first thing on my mind is, you guessed it, TDD!

And you know what, XNA is not the friedliest framework for tests as I’ve discovered, at least from what I’ve gathered by trying a few bits and reading up online. See the problem with a framework and the concept of Unit Tests is that I don’t think they mash together quite perfectly. Unit tests rely on nothing but the classes, frameworks force complience on everything they have and the kitchen sink (actually XNA isn’t that bad, I’ve seen worse, much much worse).

Let’s put things in perspective.

One of the projects in question to use XNA is a 2D platformer type thingy, which I won’t go into much detail since I’m not really at liberty to say much at the moment. I’ve been nabbed on Sprite duty. XNA comes with a lot of built in functionality for loading and drawing pretty 2D pictures via the ContentManager and the SpriteBatch objects and having a central object to hold all the nice data and functionailty will be really nifty; enter the mighty Sprite class. Let’s TDD!!

class TestNewSprite
{
  [TestMethod]
  public void name_is_NewSprite()
  {
    Assert.AreEqual("NewSprite", _sprite.Name); 
  }
}

Wooh, so many things that break on that it isn’t even funny. For the rest of it you’ll have to imagine me building the code backwards.

class TestNewSprite
{
  public TestNewSprite()
  { 
    _sprite = new Sprite("NewSprite"); 
  }

  [TestMethod]
  public void name_is_NewSprite()
  { 
    Assert.AreEqual("NewSprite", _sprite.Name); 
  }

  private Sprite _sprite;
}

Alright, I know the drill; create the class, create the fields and watch everything go green. Now what?

My first thought was a Sprite needs an image. Basic right? I mean, the whole point of it is to wrap a texture and describe how to draw it with extra data… Wrong! This is basically where I hit a brick wall. A Texture2D object requires the XNA framework to load, meaning I would need start fiddling around with mocks and manually instantiating the parts of the framework which I need. While these things are possible, and in fact have been done by a couple of other people on the ‘net, it just doesn’t strike me as a unit test, intergration test – yes, but not a unit test. So after some thought and talking to myself while doing the dishes I’ve decided to change my perspective on the problem.

I’ll TDD my Sprite class as a data set which describe the drawing behaviour to apply on a texture. The class will hold a reference to a texture which will be created by a ContentManager object but I’ll not test that particular functionality, I’ve got to trust XNA that it works right?

I’ll most likely wrap XNA’s concept of a ContentManager in my own custom object which will automate the loading of a texture using the Sprite class data. And as for drawing; same thing. I’ll wrap the SpriteBatch in an object which will know how to use the Sprite, SpriteMap and whatever else I’ll think about and joy to the world. Unit tests will ensure not only that I only have the data I need but that all my assumptions about manipulating it [like scaling, moving, etc.] will be correct.

Since I haven’t yet tried it I don’t know if it’ll work but I suspect that it will. I’ll write up a self-review tomorrow after the first attempt.

“Though I do like breaking femurs, You can count me with the dreamers”





Timer – Part 2

2 12 2010

Finally, Timers and all that Jazz. Let’s continue with a quick sneak peek into the previous post and the original requirements.

Another thing I would like to test is whether our timer is running or not.

TEST_FIXTURE(timer_not_running, _check_timer_is_not_running)
{
  CHECK_EQUAL(false, _timer.IsRunning());
}

Add the functionality and remove the duplication.

#ifndef _TDD_TIMER_
#define _TDD_TIMER_

class Timer
{
  public:
    typedef char* name_type;
    typedef double elapsed_type;

    bool IsRunning() const { return _isRunning; }
    name_type Name() const { return _name; }
    elapsed_type TotalElapsed() const { return _totalElapsed; }

    Timer(name_type name)
    : _name(name)
    , _totalElapsed(0)
    , _isRunning(false)
    {}

  private:
    bool _isRunning;
    name_type _name;
    elapsed_type _totalElapsed;
};

#endif/*_TDD_TIMER_*/

Compile, run, works, let’s stop for a moment and think. At the end of the last post I’ve had to make a design decision regarding the Timer and with the boolean test I’ve just written that decision comes into light. But what is it? The answer lies in the fixture itself. By allowing tests to be carried across a Timer object which is not running I’ve actually also stated that the Timer object will have to be started manually by the class users. It may seem to be a small part of the overall design and probably fairly intuitive and/or obvious but we also have to remember that this functionality emerged as a consequence of writing the tests first. Another benefit of these tests which can become apparent is their value as documentation. With just a glance at the name of this fixture we are already shown what can and needs to be done in order to use our Timer.

There’s one last thing I can think of that the Timer needs at the moment and that’s the ability to measure the time elapsed between updates. A timer that hasn’t been started and hasn’t updated should report that value as zero though.

TEST_FIXTURE(timer_not_running, _check_tick_elapsed_is_zero)
{
  CHECK_EQUAL(0, _timer.TickElapsed());
}

And we know the drill by now so let’s just skip and create the property and variable in the same step.

#ifndef _TDD_TIMER_
#define _TDD_TIMER_

class Timer
{
  public:
    typedef char* name_type;
    typedef double elapsed_type;

    bool IsRunning() const { return _isRunning; }    
    name_type Name() const { return _name; }    
    elapsed_type TickElapsed() const { return _tickElapsed; }    
    elapsed_type TotalElapsed() const { return _totalElapsed; }

    Timer(name_type name)
    : _isRunning(false)
    , _name(name)
    , _tickElapsed(0)
    , _totalElapsed(0)
    {}

  private:
    bool _isRunning;
    name_type _name;
    elapsed_type _tickElapsed;
    elapsed_type _totalElapsed;
};

#endif/*_TDD_TIMER_*/

Run the tests, see that everything works. And that’s all the functionality I can think of that our non-running timer should or can expose for us. We need to step it up, go crazy… Let’s make a timer that actually runs!

#include <UnitTests.h>
#include "Timer.h"

struct timer_is_running
{
  Timer::name_type _name;
  Timer _timer;

 ~timer_is_running(){}
   timer_is_running()
   : _name("IsRunning")
   , _timer(_name)
   {}
};

TEST_FIXTURE(timer_is_running, _check_timer_name_is_correct)
{
  CHECK_EQUAL(_name, _timer.Name());
}

Run tests and everything is green. Now comes a little confession. I’m not sure whether I should re-test the name functionality which is immutable in the sense that the name cannot change and has already been proven to be working properly by removing the code duplication from the previous tests. For now I’ll keep the test and the test-code duplication it [sort of] brings, why? I’m not sure…

Let’s look at the running timer concept. I know that the fixture as it stands will not start a timer running because the previous test for IsRunning() passes on false. My assumption though is that the IsRunning() on a running timer should be true. Test time.

TEST_FIXTURE(timer_running, _check_is_running)
{
  CHECK_EQUAL(true, _timer.IsRunning());
}

Run and fail. Clearly we need a way to start the timer and change the value returned by IsRunning(). Unlike the previous tests I can’t just hardcode a return true since the other test will fail. I’ve got to actually add a function to do that and call it before I test, to do this I’ll modify the fixture a bit.

#include <UnitTests.h>
#include "Timer.h"

struct timer_is_running
{
  Timer::name_type _name;
  Timer _timer;

 ~timer_is_running(){}
   timer_is_running()
   : _name("IsRunning")
   , _timer(_name)
   {
      _timer.Start();
   }
};

Compile, run… Hey, it’s not even compiling now; apparently we need to actually write the Start() function.

#ifndef _TDD_TIMER_
#define _TDD_TIMER_

#include <cassert>

class Timer
{
  public:
    typedef char* name_type;
    typedef double elapsed_type;

    bool IsRunning() const { return _isRunning; }    
    name_type Name() const { return _name; }    
    elapsed_type TickElapsed() const { return _tickElapsed; }    
    elapsed_type TotalElapsed() const { return _totalElapsed; }

    void Start();

    Timer(name_type name)
    : _isRunning(false)
    , _name(name)
    , _tickElapsed(0)
    , _totalElapsed(0)
    {}

  private:
    bool _isRunning;
    name_type _name;
    elapsed_type _tickElapsed;
    elapsed_type _totalElapsed;
};

inline void Timer::Start()
{
  assert(_isRunning==false);
  _isRunning=true;
}

#endif/*_TDD_TIMER_*/

I’ve jumped a step in there by already putting in the code to change the _isRunning variable because there’s no point in trying to run the tests for running’s sake. I’ve also added an assert there [I’ll assume anyone who reads this also read my other posts about preconditions/postconditions and writing custom asserts]. The assert is another preliminary test to communicate the internal logic of the Timer. In this case I’ve chosen to say that trying to Start a running timer is cause for alarm because somewhere there’s a logic misstep. Anyway, compile, run, test and pass.

This post is running a bit long and I’m running a bit tired but I’m happy with the direction this is taking…

“I pounded on a farmhouse looking for a place to stay, I was mighty mighty tired I had come a long long way”





Timer – Part 1

27 11 2010

Why part 1? Because arrays are zero-indexed for me.
Lame jokes aside, let’s continue… um… now what?

In the army when we’ve done navigation field exercises they told us that whenever we get confused or lost the best option is to find higher grounds and survey the land. This particular advice is just as relevent for me here as it was back then [although no matter how many times I got up I always got lost again on the way down but that’s another story :D].

The equivalent to taking to the higher grounds here is looking back at the original answer to “what do I want?”. Ok, looked at it, now what’s the simplest state in which a Timer object can exist? The simple answer [and simple is the only answer one should look for when testing first] is a Timer that isn’t running. Sounds good, let’s do that. I can already see that I’m going to end up with some duplicated setup code so I’ll write the fixture code now [but keep the previous test in a different file, call this one timer_not_running_tests.cpp].

#include <UnitTests.h>
#include "Timer.h"

struct timer_not_running
{
  Timer::name_type _name;
   Timer _timer;

 ~timer_not_running(){}
    timer_not_running()
    : _name("NotRunning")
    , _timer(_name)
    {}
};

TEST_FIXTURE(timer_not_running, _check_timer_name_is_correct)
{
  CHECK_EQUAL(_name, _timer.Name());
}

Compile, run and watch it flame. Why? Timer::Name() returns a constant string to satisfy the previous test. Stop, let’s reflect here for a second before going on. If we look at the code from last post we’ll also notice that the first test we wrote suffers from a bad case of code duplication. The “Timer1” string is constantly duplicated, in the test and in the Timer class code. I’m going to take the time now to remove the duplication by refactoring the Name() function.

#ifndef _TDD_TIMER_
#define _TDD_TIMER_

class Timer
{
  public:
    typedef char* name_type;

    name_type Name() const { return _name; }

    Timer(name_type name)
    : _name(name)
    {}

  private:
    name_type _name;
};

#endif/*_TDD_TIMER_*/

Compile, run and watch both tests pass. What have I learnt? Removing code duplication as soon as possible leads to more general solutions. So from now on, obvious code duplication should be removed as soon as possible. The second test should always be written though, to verify that the solution is indeed what we want. The other approach for something like this is using the additional tests to force the functionality to change, in this case I would have had to hold the name string in an actual variable in order to have different names. Using more tests to zoom-in on the correct implementation is called Triangulation within the Agile circles and I usually use it to flesh out complex functionality. Alright, less talk more code, I’m testing a timer that isn’t running. If a timer isn’t running then the total time that it counted should be zero, right? let’s test.

TEST_FIXTURE(timer_not_running, _check_total_elapsed_time_is_zero)
{
  CHECK_EQUAL(0, _timer.TotalElapsed());
}

Compile, run, things explode, let’s add the right function.

#ifndef _TDD_TIMER_
#define _TDD_TIMER_

class Timer
{
  public:
    typedef char* name_type;
    typedef double elapsed_type;

    name_type Name() const { return _name; }
    elapsed_type TotalElapsed() const { return 0; }

    Timer(name_type name)
    : _name(name)
    {}

  private:
    name_type _name;
};

#endif/*_TDD_TIMER_*/

Whoops, code duplication. Let’s refactor.

#ifndef _TDD_TIMER_
#define _TDD_TIMER_

class Timer
{
  public:
    typedef char* name_type;
    typedef double elapsed_type;

    name_type Name() const { return _name; }
    elapsed_type TotalElapsed() const { return _totalElapsed; }

    Timer(name_type name)
    : _name(name)
    , _totalElapsed(0)
    {}

  private:
    name_type _name;
    elapsed_type _totalElapsed;
};

#endif/*_TDD_TIMER_*/

Compile, run, all tests pass and I’m happy. The code itself isn’t very exciting to look at, after all of this and I’ve just got two functions up and running. But the bigger picture is a bit more invloved. I’ve learnt the importance of removing code duplication early, I’ve had some more practice in making assumptions about functionality [a timer counts the total elapsed time therefore a timer that isn’t running should have a zero value for it]. And last but far from least, I’ve started documenting the contract between the client [what is needed] and the developer [what the code does] through the tests; that is, the timer must have a name and will measure a total amount of elapsed time. There is one more design decision I’ve made here, it might not be as obvious though, can anyone spot what it is??

“…Into the half light, Another velvet morning for me, yeah…”