Pex - "Fix It" or "Allow It"

It seems like when Pex generates failing tests the choice is between "Fix It" and "Allow It" and they do seem to do very different things so I thought it was worth mentioning what little I've found out.

I'll start out with this code in a Pex test:

[PexMethod]
[PexUseType(typeof(AuthorizationService))]
public void overall_behavior_correct(IAuthorizationService authorizationService,
   Account source, Account destination, double amountToTransfer)
{
   new AccountTransferService().Transfer(source, destination, amountToTransfer, authorizationService);
}

When I run Pex with this test it generates one passing test and some failing tests:

The failing tests are showing me useful things, for example the SUT does indeed raise an exception if the source and destination Accounts are the same. If I right click on either of these issues I get two options:

Fix It

If I select "Fix It" on each of the automatically generated tests then Pex ends up updating the original parameterized unit test (PUT) to look like this:

    [PexMethod]
   [PexUseType(typeof(AuthorizationService))]
   public void overall_behavior_correct(IAuthorizationService authorizationService,
       Account source, Account destination, double amountToTransfer)
   {
       // <pex>
       PexAssume.IsNotNull((object)source, "source");
       PexAssume.IsTrue(source != destination, "source == destination");
       PexAssume.IsTrue
           (source.Balance >= amountToTransfer, "source.Balance < amountToTransfer");
       PexAssume.IsNotNull((object)destination, "destination");
       PexAssume.IsNotNull((object)authorizationService, "authorizationService");
       PexAssume.IsTrue(amountToTransfer >= 1.5, "amountToTransfer < 1.5");
       PexAssume.IsTrue(((AuthorizationService)authorizationService).AllowTransfer
           (source, destination, amountToTransfer) != false, "complex reason");
       // </pex>

       new AccountTransferService().Transfer(source, destination, amountToTransfer, authorizationService);
   }

If I now run Pex again it will generate 0 tests from this code. I guess this makes sense, the PexAssumes are presumably telling Pex not to pass in certain values (such as null for source). However in the process I've made my PUT pretty useless and it does make me question the usefulness of the "Fix It" option in these sorts of situations, so "Allow It" must be the more sensible option in this case...

Allow It

If I select "Allow It" for each of the failing generated tests then my Pex test stays as it was originally but the following attributes are put into my assembly:

[assembly: PexAllowedExceptionFromAssembly(typeof(ArgumentException), "PexPlay")]
[assembly: PexAllowedExceptionFromAssembly(typeof(ArgumentNullException), "PexPlay")]
[assembly: PexAllowedExceptionFromAssembly(typeof(ArgumentOutOfRangeException), "PexPlay")]
[assembly: PexAllowedExceptionFromAssembly(typeof(InvalidOperationException), "PexPlay")]

PexPlay is the assembly I'm working in (the assembly that contains the SUT) and this seems to be indicating that if I get any of the specified types of exceptions anywhere in PexPlay then the tests should still pass. This is confirmed if I re-run Pex as it will generate the same set of tests but they now pass:

Problem is that the attributes are at too high a level for me to be happy, I'm not necessarily always happy to see those exceptions so instead of using PexAllowedExceptionFromAssembly I tried using attributes at the Pex test level which seems to work fine:

[PexAllowedException(typeof(ArgumentException)), PexAllowedException(typeof(ArgumentNullException)),
PexAllowedException(typeof(ArgumentOutOfRangeException)), PexAllowedException(typeof(InvalidOperationException))]
[PexMethod]
[PexUseType(typeof(AuthorizationService))]
public void overall_behavior_correct(IAuthorizationService authorizationService,
   Account source, Account destination, double amountToTransfer)
{
   new AccountTransferService().Transfer(source, destination, amountToTransfer, authorizationService);
}

Its probably worth noting how the PexAllowedException have effected the generated tests, here's one of them (MSTest):

[TestMethod]
[ExpectedException(typeof(ArgumentNullException))]
[PexGeneratedBy(typeof(when_account_transfer_occurs))]
public void overall_behavior_correctIAuthorizationServiceAccountAccountDouble_20080607_120018_002()
{
   Account a0;
   a0 = AccountFactory.Create(0);
   AuthorizationService as0 = new AuthorizationService();
   this.overall_behavior_correct((IAuthorizationService)as0, a0, (Account)null, 1);
}

As you can see the test is marked with ExpectedException attribute which correctly specifies the behaviour I expect when I pass in a null Account.

Share This - Digg It • Save to del.icio.us • Stumble It! • Kick It • DZone