Pex - "Fix It" or "Allow It"

It seems like when Pex generates failing tests the choice is between "Fix It" and "Allow It" and they do seem to do very different things so I thought it was worth mentioning what little I've found out.

I'll start out with this code in a Pex test:

[PexMethod]
[PexUseType(typeof(AuthorizationService))]
public void overall_behavior_correct(IAuthorizationService authorizationService,
   Account source, Account destination, double amountToTransfer)
{
   new AccountTransferService().Transfer(source, destination, amountToTransfer, authorizationService);
}

When I run Pex with this test it generates one passing test and some failing tests:

The failing tests are showing me useful things, for example the SUT does indeed raise an exception if the source and destination Accounts are the same. If I right click on either of these issues I get two options:

Fix It

If I select "Fix It" on each of the automatically generated tests then Pex ends up updating the original parameterized unit test (PUT) to look like this:

    [PexMethod]
   [PexUseType(typeof(AuthorizationService))]
   public void overall_behavior_correct(IAuthorizationService authorizationService,
       Account source, Account destination, double amountToTransfer)
   {
       // <pex>
       PexAssume.IsNotNull((object)source, "source");
       PexAssume.IsTrue(source != destination, "source == destination");
       PexAssume.IsTrue
           (source.Balance >= amountToTransfer, "source.Balance < amountToTransfer");
       PexAssume.IsNotNull((object)destination, "destination");
       PexAssume.IsNotNull((object)authorizationService, "authorizationService");
       PexAssume.IsTrue(amountToTransfer >= 1.5, "amountToTransfer < 1.5");
       PexAssume.IsTrue(((AuthorizationService)authorizationService).AllowTransfer
           (source, destination, amountToTransfer) != false, "complex reason");
       // </pex>

       new AccountTransferService().Transfer(source, destination, amountToTransfer, authorizationService);
   }

If I now run Pex again it will generate 0 tests from this code. I guess this makes sense, the PexAssumes are presumably telling Pex not to pass in certain values (such as null for source). However in the process I've made my PUT pretty useless and it does make me question the usefulness of the "Fix It" option in these sorts of situations, so "Allow It" must be the more sensible option in this case...

Allow It

If I select "Allow It" for each of the failing generated tests then my Pex test stays as it was originally but the following attributes are put into my assembly:

[assembly: PexAllowedExceptionFromAssembly(typeof(ArgumentException), "PexPlay")]
[assembly: PexAllowedExceptionFromAssembly(typeof(ArgumentNullException), "PexPlay")]
[assembly: PexAllowedExceptionFromAssembly(typeof(ArgumentOutOfRangeException), "PexPlay")]
[assembly: PexAllowedExceptionFromAssembly(typeof(InvalidOperationException), "PexPlay")]

PexPlay is the assembly I'm working in (the assembly that contains the SUT) and this seems to be indicating that if I get any of the specified types of exceptions anywhere in PexPlay then the tests should still pass. This is confirmed if I re-run Pex as it will generate the same set of tests but they now pass:

Problem is that the attributes are at too high a level for me to be happy, I'm not necessarily always happy to see those exceptions so instead of using PexAllowedExceptionFromAssembly I tried using attributes at the Pex test level which seems to work fine:

[PexAllowedException(typeof(ArgumentException)), PexAllowedException(typeof(ArgumentNullException)),
PexAllowedException(typeof(ArgumentOutOfRangeException)), PexAllowedException(typeof(InvalidOperationException))]
[PexMethod]
[PexUseType(typeof(AuthorizationService))]
public void overall_behavior_correct(IAuthorizationService authorizationService,
   Account source, Account destination, double amountToTransfer)
{
   new AccountTransferService().Transfer(source, destination, amountToTransfer, authorizationService);
}

Its probably worth noting how the PexAllowedException have effected the generated tests, here's one of them (MSTest):

[TestMethod]
[ExpectedException(typeof(ArgumentNullException))]
[PexGeneratedBy(typeof(when_account_transfer_occurs))]
public void overall_behavior_correctIAuthorizationServiceAccountAccountDouble_20080607_120018_002()
{
   Account a0;
   a0 = AccountFactory.Create(0);
   AuthorizationService as0 = new AuthorizationService();
   this.overall_behavior_correct((IAuthorizationService)as0, a0, (Account)null, 1);
}

As you can see the test is marked with ExpectedException attribute which correctly specifies the behaviour I expect when I pass in a null Account.

Share This - Digg It • Save to del.icio.us • Stumble It! • Kick It • DZone

Pex - First Impressions

Apparently the issues I was having with Pex (ClrMonitorFail errors) were caused by its interaction with TypeMock. This doesn't shock me as having used TypeMock for a while I've learned that any time anything odd starts happening disabling TypeMock is a good idea.

So I disabled TypeMock and started playing with Pex, first impression is that it looks great but my second impression was that the IDE integration seemed a little flaky. My IDE actually crashed many times whilst using it but after a while I learned what to click and not click :)

Anyway I thought I'd start writing down the little that I've found out about Pex in case it is in any way useful to anyone.

First Feelings

I did notice a few interesting things when working with Pex, the first is that it seemed like it c lull you into a false sense of security.

Seeing a whole loading of auto-generated tests passing is great but I quickly began to notice that I could modify the code under test in inappropriate ways and my tests weren't failing. This wasn't Pex's fault though, I just hadn't been thorough enough in telling it what to expect and once I applied more PexAssume values and a few more assertions I definitely felt safer.

Fix It Or Allow It

I found that I was getting some useful tests generated even if I was quite vague:

[PexMethod]
[PexUseType(typeof(AuthorizationService))]
public void overall_behavior_correct(IAuthorizationService authorizationService,
    Account source, Account destination, double amountToTransfer)
{
    new AccountTransferService().Transfer(source, destination, amountToTransfer, authorizationService);
}

The tests Pex was generating were to do with null reference exceptions and invariants that the SUT was enforcing. Obviously initially the tests were failing so I had to tell Pex what I expected the SUT to do in each situation. I could do this using either the "Allow It" or "Fix It" options from the "Pex Results" panel.

If I chose "Fix It" then I would tend to get the following:

My IDE would then close down, gah. However if I persevered though it would update the test, for example:

[PexMethod]
[PexUseType(typeof(AuthorizationService))]
public void overall_behavior_correct(IAuthorizationService authorizationService,
   Account source, Account destination, double amountToTransfer)
{
   PexAssume.IsNotNull((object)source, "source");
   PexAssume.IsTrue(source != destination, "source == destination");

   new AccountTransferService().Transfer(source, destination, amountToTransfer, authorizationService);
}

When I then ran Pex for this method again it wouldn't generate a test that passed in null for source or that passed in source and destination as the same objects.

The alternative to "Fix It" is to select "Allow It" it, when I did that Pex would put this attribute in PexAssemblyInfo.cs

[assembly: PexAllowedExceptionFromAssembly(typeof(ArgumentException), "PexPlay")]

This seems a bit brute force though, I don't want to allow the exception across the entire assembly so I probably need to do a bit more research.

Decimals

One interesting thing that I noticed is that when I setup my methods to take in decimals I got no tests generated, but if I changed the inputs to be doubles I did. Not sure what that's all about but again I need to do some more research.

Mocking

Ignore the layout of this test, its a mess, but I do like the easy way I'm able to specify that I want to use a mock of the authorization service using the PexUseType attribute:

       [PexMethod]
      [PexUseType(typeof(MockAuthorizationService))]
      public void transfers_correctly_between_accounts(IAuthorizationService authorizationService,
          double intialInSource, double initialInDestination, double amountToTransfer)
      {
          PexAssume.IsNotNull(authorizationService);
          PexAssume.IsFalse(amountToTransfer < 0);
          PexAssume.IsTrue(intialInSource > amountToTransfer);

          Account source = new Account(intialInSource);
          Account destination = new Account(initialInDestination);

          double fromSourceBefore = source.Balance;
          double fromDestinationBefore = destination.Balance;

          new AccountTransferService().Transfer(source, destination, amountToTransfer, authorizationService);

          Assert.AreEqual(fromSourceBefore - amountToTransfer, source.Balance);
          Assert.AreEqual(fromDestinationBefore + amountToTransfer, destination.Balance);
      }

The mock service is in this form (see the PDF for more on this but I haven't truly had time to grok it yet):

[PexMock]
public class MockAuthorizationService : IAuthorizationService
{
   public bool AllowTransfer(Account source, Account destination, double amountToTransfer)
   {
       var call = PexOracle.Call(this);
       return call.ChooseResult<bool>();
   }
}

When I run Pex over transfers_correctly_between_accounts it will actually correctly use an instance of MockAuthorizationService and will run a test where that service returns false, causing the transfer to fail as expected. Nice.

Share This - Digg It • Save to del.icio.us • Stumble It! • Kick It • DZone