Saturday, June 07, 2008

Pex - First Impressions

Apparently the issues I was having with Pex (ClrMonitorFail errors) were caused by its interaction with TypeMock. This doesn't shock me as having used TypeMock for a while I've learned that any time anything odd starts happening disabling TypeMock is a good idea.

So I disabled TypeMock and started playing with Pex, first impression is that it looks great but my second impression was that the IDE integration seemed a little flaky. My IDE actually crashed many times whilst using it but after a while I learned what to click and not click :)

Anyway I thought I'd start writing down the little that I've found out about Pex in case it is in any way useful to anyone.

First Feelings

I did notice a few interesting things when working with Pex, the first is that it seemed like it c lull you into a false sense of security.

Seeing a whole loading of auto-generated tests passing is great but I quickly began to notice that I could modify the code under test in inappropriate ways and my tests weren't failing. This wasn't Pex's fault though, I just hadn't been thorough enough in telling it what to expect and once I applied more PexAssume values and a few more assertions I definitely felt safer.

Fix It Or Allow It

I found that I was getting some useful tests generated even if I was quite vague:

[PexMethod]
[PexUseType(typeof(AuthorizationService))]
public void overall_behavior_correct(IAuthorizationService authorizationService,
Account source, Account destination, double amountToTransfer)
{
new AccountTransferService().Transfer(source, destination, amountToTransfer, authorizationService);
}

The tests Pex was generating were to do with null reference exceptions and invariants that the SUT was enforcing. Obviously initially the tests were failing so I had to tell Pex what I expected the SUT to do in each situation. I could do this using either the "Allow It" or "Fix It" options from the "Pex Results" panel.

If I chose "Fix It" then I would tend to get the following:

image

My IDE would then close down, gah. However if I persevered though it would update the test, for example:

[PexMethod]
[PexUseType(typeof(AuthorizationService))]
public void overall_behavior_correct(IAuthorizationService authorizationService,
Account source, Account destination, double amountToTransfer)
{
PexAssume.IsNotNull((object)source, "source");
PexAssume.IsTrue(source != destination, "source == destination");

new AccountTransferService().Transfer(source, destination, amountToTransfer, authorizationService);
}

When I then ran Pex for this method again it wouldn't generate a test that passed in null for source or that passed in source and destination as the same objects.

The alternative to "Fix It" is to select "Allow It" it, when I did that Pex would put this attribute in PexAssemblyInfo.cs

[assembly: PexAllowedExceptionFromAssembly(typeof(ArgumentException), "PexPlay")]

This seems a bit brute force though, I don't want to allow the exception across the entire assembly so I probably need to do a bit more research.

Decimals

One interesting thing that I noticed is that when I setup my methods to take in decimals I got no tests generated, but if I changed the inputs to be doubles I did. Not sure what that's all about but again I need to do some more research.

Mocking

Ignore the layout of this test, its a mess, but I do like the easy way I'm able to specify that I want to use a mock of the authorization service using the PexUseType attribute:

       [PexMethod]
[PexUseType(typeof(MockAuthorizationService))]
public void transfers_correctly_between_accounts(IAuthorizationService authorizationService,
double intialInSource, double initialInDestination, double amountToTransfer)
{
PexAssume.IsNotNull(authorizationService);
PexAssume.IsFalse(amountToTransfer < 0);
PexAssume.IsTrue(intialInSource > amountToTransfer);

Account source = new Account(intialInSource);
Account destination = new Account(initialInDestination);

double fromSourceBefore = source.Balance;
double fromDestinationBefore = destination.Balance;

new AccountTransferService().Transfer(source, destination, amountToTransfer, authorizationService);

Assert.AreEqual(fromSourceBefore - amountToTransfer, source.Balance);
Assert.AreEqual(fromDestinationBefore + amountToTransfer, destination.Balance);
}

The mock service is in this form (see the PDF for more on this but I haven't truly had time to grok it yet):

[PexMock]
public class MockAuthorizationService : IAuthorizationService
{
public bool AllowTransfer(Account source, Account destination, double amountToTransfer)
{
var call = PexOracle.Call(this);
return call.ChooseResult<bool>();
}
}

When I run Pex over transfers_correctly_between_accounts it will actually correctly use an instance of MockAuthorizationService and will run a test where that service returns false, causing the transfer to fail as expected. Nice.

Share This - Digg It Save to del.icio.us Stumble It! Kick It DZone

3 comments:

  1. Unfortunately, the understanding of Pex for decimal is very limited since the Decimal type is mostly implementd in unmanaged code. Pex cannot instrument unmanaged and thus reason about it.

    ReplyDelete
  2. @Peli
    Can I suggest that you might want to raise a more obvious error/warning if someone does use decimal as an input for a Pex test, as at the moment it fails quite quietly.

    ReplyDelete
  3. Good point. We are working on a better feedback mechanism when you use decimal (and other unmanaged APIs).

    ReplyDelete