BDD - Available Frameworks

I've been using the Astels style of BDD for a while now but so far I've just done it using MSTest/NUnit and a few custom base classes. I think that's a good way to start out, as with so many good things in life it doesn't require a new whizzy tool/framework.

However I've just joined a new project and we've been looking at different frameworks that are available for unit testing and for BDD so I thought I'd post about what I've seen so far. Any opinions would be gladly received.

MSpec
This project is very cool, not only does it allow you to create superb specifications but you also get some nice reporting. Here's a simple example of a sample spec:

public class When_adding_a_contact_to_a_user_with_no_existing_contacts
{
  private static User _user;
  private static Contact _contact;

  Establish context_once =()=>
  {
      _user = new TestUserBuilder().Build();
      _contact = new ContactBuilder().Build();
  };

  Because a_context_is_added =()=>
      _user.Contacts.Add(_contact);

  private It should_associate_the_contact_with_the_user = () =>
      _user.Contacts.Contains(_contact).ShouldBeTrue();
                                                           
}

One thing to note is if your planning to look at MSpec then you'll probably want to download the Machine codebase since there aren't many examples of using MSpec on the Web the examples with Machine are a good starting point.

So far there's no R# integration but that doesn't worry me at all as if needed it will come and this is still a very early version.

The reporting seems to work well, but we primarily use this style for unit/integration tests and so we are unlikely to present the reports outside the development team. Having said that Aaron pointed out that they can be useful within the development team, which makes a lot of sense.

Overall my main worry is the syntax could be a bit much for some people, in particular if you go for the compact style:

NUnit
I think there's a good argument for just using a base class, especially when you are getting going with the approach:

public class When_adding_a_contact_to_a_user_with_no_existing_contacts : SpecificationBaseNUnit
{
  private User _user;
  private Contact _contact;

  protected override void EstablishContext()
  {
      _user = new TestUserBuilder().Build();
      _contact = new ContactBuilder().Build();
  }

  protected override void Act()
  {
      _user.Contacts.Add(_contact);
  }

  [Test]
  public void should_associate_the_contact_with_the_user()
  {
      _user.Contacts.Contains(_contact).ShouldBeTrue();
  }
}

This is a hopelessly naive example but you get the idea. You lose some of the syntax niceness, suddenly the specs themselves take up multiple lines because of all the curlies. You've also lost reporting, unless you put in some work yourself. However it is a little easier to understand and when introducing TDD/BDD that could be important.

XUnit.net
I'm no XUnit.net expert but Ben Hall convinced us to give it a shot by recommending it and it is very nice. You can read about an approach that works here. If you use the specification base class described in that post you might end up with this:

// Using base class influenced by http://www.bjoernrochel.de/2008/10/04/introducing-xunitbddextensions/
public class When_adding_a_contact_to_a_user_with_no_existing_contacts : SpecificationBase
{
  private User _user;
  private Contact _contact;

  protected override void EstablishContext()
  {
      _user = new TestUserBuilder().Build();
      _contact = new ContactBuilder().Build();
  }

  protected override void Because()
  {
      _user.Contacts.Add(_contact);
  }

  [Observation]
  public void should_associate_the_contact_with_the_user()
  {
      _user.Contacts.Contains(_contact).ShouldBeTrue();
  }
}

One aspect of XUnit that might throw you is how opinionated it is, which could be an advantage or a disadvantage. An example is that it's aiming for each test to run in isolation, so the fixture class is re-created each time and if you really want to reuse the fixture you implement IUseFixture. I guess this is a very safe approach because it means tests/specs are extremely unlikely to affect each other, but it actually seems over-kill if you're using a style where the specification methods only assert (no side-effects).

The lack of messages on assertions seems sensible, and it is for small focused BDD specifications, but if you use it for integration testing you would want the option of adding a message in.

On the syntax front we could always go for more flexibility:

public class When_a_user_has_no_contacts : FlexibileGrammarSpecificationBase
{
  private User _user;
  private Contact _contact;

  protected override void EstablishContext()
  {
      _user = new TestUserBuilder().Build();
      _contact = new ContactBuilder().Build();
  }

  [Because]
  protected void and_we_give_them_a_new_contact()
  {
      _user.Contacts.Add(_contact);
  }

  [Observation]
  public void the_contact_should_be_associated_with_the_user()
  {
      _user.Contacts.Contains(_contact).ShouldBeTrue();
  }
}

However this seems a little pointless to me so I've dumped the idea.

Summary
We decided to go with XUnit.net but we also plan to look at Ruby based solutions including RSpec and Cucumber. Cucumber seems exciting as it lets you specify table based specifications, theoretically allowing us to get the advantages of a FIT style approach without having to use FIT (or SLIM) itself.

Ultimately there is a lot going on in the BDD space in Ruby-land (and a book on the way) and the language does suit it quite well so we intend to do some playing.

Share This - Digg It • Save to del.icio.us • Stumble It! • Kick It • DZone