Pex - test case 1

Hello,

I just started investigating Pex . In order to learn the tool I'm using several (academic) test cases to see how Pex can help there.

This case case is called Commercial Cut percentage for an order. In our Order system we have implemented a rule that will calculate the commercial cut based on the total OrderAmount. The following table depicts the rules for applying the commercial cut.

Total (rounded) order-amount (€)	<=1500	[1501-5000]	[5001-10000]	[10001-25000]	>25000
Commercial cut	0	2	3	4	5

I have implemented this rule in a c# method. For the sake of this exercise I will not argue about whether or not this should be a private method (Which by the way can perfectly be unit tested in the VS Test framework).

       public uint DetermineCommercialCut(uint aTotalOrderAmount)
       {

           if (aTotalOrderAmount <= 1500)
           {
               return 0;
           }
           else if (aTotalOrderAmount > 1500 & aTotalOrderAmount <= 5000)
           {
               return 2;
           }
           else if (aTotalOrderAmount > 5000 & aTotalOrderAmount <= 10000)
           {
               return 3;
           }
           else if (aTotalOrderAmount > 10000 & aTotalOrderAmount <= 25000)
           {
               return 4;
           }
           else
           {
               return 5;
           }
       }
   

Of course we will unit test this class method. This method should always return some positive integer value back depending on total order amount that is supplied via unsigned integer parameter. So I give a positive integer (or zero) to the method and I receive a positive integer (or zero) back. Of course the purpose of the method is determine the commercial cut percentage based on the order amount. Our table showed on the rules we need to apply and our code is (hopefully) constructed in the same way. So we will construct a unit test that will call this method.

   [TestMethod()]
   public void DetermineCommercialCut_ZeroOrderAmount()
      {
            Order target = new Order();
            uint aTotalOrderAmount =0;
            uint expected = 0;
            uint actual;
            actual = target.DetermineCommercialCut(aTotalOrderAmount);
            Assert.IsTrue(expected == actual);
   }

We could continue with testing all positive integers up to 4.294.967.295. Of course this is not worthwhile. We can pick certain values for this entire collection in such way that any value within certain ranges of this big collection , will (should) behave in the same way. This is referred to as equivalence classes in testing literature. So whether I pick 0 or 1500 as total (rounded) order amount or any value in between, it should always resolve to a return value of zero. So based on this principle I can detect several equivalence classes.

• Must be positive integer


• Integers between 0 and 1500 included


• Integers between 1501 and 5000 included


• Integers between 5001 and 10000 included


• Integers between 10001 and 25000 included


• Integers higher then 25000

I’m spotting 5 valid classes. Because the.NET type system helps me out here I don’t need explicit checks on the invalid class namely negative integers.

now I need to pick a value I will actually use in my unit tests. I will at least need to pick 5 values, one from each equivalence class. In my test I will verify if the reduction cut is appropriate for that value.
So instead of making 4.294.967.295 unit tests I only need 5 to “cover” all possible test situation ( different commercial cut percentages).

Another technique that is usually combined with the former is border value analysis. In the example I can with a minimal additional effort also verify if the comparison signs are as they were intended to be. Instead of picking any value in the equivalence class , I will look up the values that are used in the conditions for example. Instead of 2531 , I will pick 5000 just at the border. So 5000 should give me a commercial cut of 2. Just like 2531 but I have an extra check on the comparison sign with no extra effort. With a little more effort I could also include the values just under (or above) the border.

So basic techniques told me I need 5 test case to fully test my rule (i.e. Method). So this means 5 Visual Studio test methods. Actually , Visual Studio test Framework can, in this case , help us even further with reducing the number of unit test to …..one. That is one data-driven unit test. What we do, is externalize the test data values into for example an Excel file. This data-source will then “feed” our data-driven unit test with actual values. So although we have only one written unit test, we would like this test code to be executed 5 times. We now use this table (with some modification) as data-source for our data-driven unit test in Visual Studio 2008. For more info see MSDN.

        [DataSource("Microsoft.VisualStudio.TestTools.DataSource.XML",
                     "DataDirectory\\testData.xml",
                     "CommercialCut", DataAccessMethod.Sequential),
        DeploymentItem("TestProject1\\testData.xml")]
        [TestMethod()]
        public void DetermineCommercialCut_DataDriven()
        {
            Order target = new Order();
            uint aTotalOrderAmount = System.Convert.ToUInt32(this.TestContext.DataRow["input"]);
            uint expected = System.Convert.ToUInt32(this.TestContext.DataRow["expected"]);
            uint actual;
            actual = target.DetermineCommercialCut(aTotalOrderAmount);
            Assert.IsTrue(expected== actual);
        }

We determined on basis of our analysis that 5 test-cases should cover the testing of the rule. Here Visual Studio Test framework also can help us by instrumenting the code before executing the unit tests (Code coverage). A nice feature of the Visual Studio Test framework is the colorization of the code under test to show which parts have been “touched” by the unit tests and which parts didn’t.

When looking at the Visual Studio Code coverage results of one test executed :





You can see the statements in different coloring; touched versus not touched. This should tell you that you must add some test cases to see code paths in your method also work as you intended and consequently coded.

If you would run the data-driven test with the 5 test cases you would get the following result :




Everything is highlighted in blue, meaning that every “block” in our code has been executed during the run of our unit test. That should come as no surprise because we rigorously followed our table for our rule.

So far our “manual” effort . Lets see how Pex can help you. So the documentation states that Pex can create a test suite with a high coverage by exploring your code triggered from a parameterized test you will have to write.

I highly recommend reading the tutorial for more background information on Pex. Meanwhile I will continue with the example to show you how Pex can create a test suite for us.

First we must create a “parameterized” unit test to give Pex an entry point into our code under test. “Parameterized Unit Tests (PUTs) are... tests that take parameters. Unlike traditional unit tests, which are usually closed methods, PUTs take any set of parameters. Is it that simple? Yes! From there, Pex will try to generate the (minimal) set of inputs that fully coverthe code reachable from the test. “

So in our case this could look something like:

[PexMethod()]
public void CaluclateReductionPercentageTest(uint aSales)
{
 Order target = new Order();
 uint actual;
 actual = target.DetermineCommercialCut(aSales);
 PexValue.AddForValidation("result", actual);
}

You can launch Pex from within Visual Studio 2008 C# by right clicking the mouse
(or pressing CTRL+F8). You will see a menu option ” Run Pex Exploration”.
This will launch Pex in doing it’s job, running your code (several times) while generating the
right combination of input values for each possible code path and then generating the code
for all the test cases.

Pex generated 5 tests for us. It clearly also detected (in another way) that there are 5 test cases needed to cover the code. If we take our 5 equivalence classes we can see that this is correct. If our “conduct-of-test” specifies the use of border-value analysis where appropriate , then these values are not so well chosen in my opinion.

Lets say that we want to add an extra level of commercial cut for sales between 1000 and 1500 Euros. And we want to let our client enjoy a commercial cut of 1 percent.

In the manual test analysis we would come up with an additional equivalence class. Pex can also detect this.

But first lets re-run our initial set of unit tests generated by pex.

Our Code Coverage run should warn us that we missed out on some parts of the code. We need another unit test. To do so , we will re-run Pex on our pexMethod. The will force pex to examine our code again and it should find an extra path in our code.

Doing so resulted in pex finding an additional test (nr 3 - see screenshot). If we run the generated test methods and also look at the coverage findings we find a positive result.

So in a nutshell. In this (although academic) situation, Pex found the same test cases if we would use a manual technique. Border value Analysis would be advantageous in this test case but Pex doesn't propose these values while exploring this method based on the PexMethod we proposed.

So the most important lesson I think is , as also stated in the Pex FAQ : Use Pex as one of your tools not as the (only) tool.

Maybe you also have tried out Pex? Don't hesitate to share....

Thanks for reading.

Best regards,

Alexander

ps: I'm still experimenting with SyntaxHighLighter. If you have any tips how to improve showing code snippets in Blogger...let me know.