Pex - Test case 3

Hello,

This post is a continuation of previous posts. Feel free to check them out.

• Introduction


• Case study 1


• Case study 2

In this example I base my example on the following rule:
In our Order system we need to implement a validation rule to see whether we can accept an order or not. The criteria for acceptance where given to us (luckily) in the form of a decision table. There are three criteria ; an internal customer qualification scheme, the fact if the previous payments of the customer were usually on time and correct and the fact if the customer has sufficient credit possibilities.

Cust-code

a

a

a

a

b

b

b

b

c

c

c

c

u

u

u

u

paymentsOK

y

y

n

n

y

y

n

n

y

y

n

n

y

y

n

n

CredietLimitOK

y

n

y

n

y

n

y

n

y

n

y

n

y

n

y

n

AcceptOrder

y

y

n

n

y

y

y

n

y

y

y

y

n

n

n

n

    public enum CustomerType 
        { 
            U_Unknown = 0, 
            A_Status = 1, 
            B_Status = 2, 
            C_Status = 3 
        } 

    public class OrderManager
    {
        public bool IsOrderToBeAccepted(CustomerType aClientType,  bool aGoodpaymentHistory,  bool aCreditSufficient) 
            { 
                bool orderAccept = false; 
                switch (aClientType) { 
                    case CustomerType.A_Status: 
                        if (aGoodpaymentHistory == true)  { 
                            orderAccept = true; 
                        }
                       else
                        {
                            orderAccept = false;
                        }
                        break; 
                    case CustomerType.B_Status: 
                        if (aGoodpaymentHistory || aCreditSufficient) { 
                            orderAccept = true; 
                        } 
                        else { 
                            orderAccept = false; 
                        } 
                        break; 
                    case CustomerType.C_Status: 
                        orderAccept = true; 
                        break; 
                    default: 
                        orderAccept = false;  
                        break; 
                } 
            return orderAccept; 
            } 
    }

With the help to the decsion table (thanks to the analyst :-), we can easily determine the test cases.
There 2 test sitaution (different behaviours) we must account for : the order is accepted or is not accepted. So we need test cases that will result in a non-accepted order and we will need a test case that will result in an accepted order. Those this mean two test cases (i.e. unit tests) are enough. There is always this "test completeness criterium". If this were a black-box test (we don't "see" the code) we could use a tool like Pict33 to propsose the combination of input values.

In this "lab" test we both have design information (desicion table) and our code. The table in this case can help us enough. Each column will represent the values we will supply to the method under test. This means we’ll end up with 16 unit tests to achieve full coverage of our table (and hopefully also of our code). The unit test are there to verify if our “intentions” (i.e. implementing this method with the above rules) were correctly transformed into code. Both test situations (order accepted / order not accepted) should normally be taken into account with these 16 test cases.

We can write out our units tests :

  [TestMethod()]
        public void isOrderToBeAcceptedCase1()
        {
            OrderManager target = new OrderManager();
            CustomerType ClientType = CustomerType.A_Status;
            bool GoodpaymentHistory = false; 
            bool CreditSufficient = false; 
            bool expected = false; 
            bool actual;
            actual = target.isOrderToBeAccepted(ClientType, GoodpaymentHistory, CreditSufficient);
            Assert.AreEqual(expected, actual);
           
        }

       [TestMethod()]
       public void isOrderToBeAcceptedCase2()
       {
           OrderManager target = new OrderManager();
           CustomerType ClientType = CustomerType.A_Status;
           bool GoodpaymentHistory = true;
           bool CreditSufficient = false;
           bool expected = false;
           bool actual;
           actual = target.isOrderToBeAccepted(ClientType, GoodpaymentHistory, CreditSufficient);
           Assert.AreEqual(expected, actual);

       }

        //and so one 

Or we make a data-driven unit test.

      [DeploymentItem("TestProject1\\OrderAcceptanceCombinations.csv")]
        [DataSource("Microsoft.VisualStudio.TestTools.DataSource.CSV", 
            "|DataDirectory|\\OrderAcceptanceCombinations.csv",
            "OrderAcceptanceCombinations#csv", DataAccessMethod.Sequential)]
        [TestMethod]
        public void DataDrivenIsOrderAccepted()
        {
            OrderManager target = new OrderManager();
            CustomerType ClientType =  (CustomerType)(System.Convert.ToInt32(this.TestContext.DataRow["CustomerType"])) ;
            bool GoodpaymentHistory = System.Convert.ToBoolean(this.TestContext.DataRow["PaymentsOK"]);
            bool CreditSufficient= System.Convert.ToBoolean(this.TestContext.DataRow["CreditOK"]) ;
            bool actual;
            bool expected = System.Convert.ToBoolean(this.TestContext.DataRow["AcceptOrder"]) ;
            actual = target.isOrderToBeAccepted(ClientType, GoodpaymentHistory, CreditSufficient);
            Assert.IsTrue(actual == expected);
        }

Bottom line is we achieve acceptable code coverage. We can verify this with the Visual Studio Code Coverage facility of the test framework.

In that respect we can actually reduce the number of test cases and still achieve full coverage. A technique we can use, is "collapsing" the desicion table. If we closely look at our rules, we can detect that several combinations should result in the same result. This means that some conditions are not taken into account (or don’t make sense) in some test cases. For example whether or not a customer with customer-type UnKnown has good creditRecords or even goodpayments records, it doesn’t matter. We don’t trust “Unknown” customers. We don’t allow an order for them! In order words we can reduce the number of test cases for 4 to 1

If we do this exercise for the entire table , we should come up with the following decision table.

Cust-code	a	a	b	b	b	c	u
paymentsOK	y	n	y	n	n	-	-
CredietLimitOK	-	-	-	y	n	-	-
AcceptOrder	y	n	y	y	n	y	n

This would mean we only need to 7 test cases to fully test our rule. Collapsing tables is powerful in terms of test case reduction but can also be “dangerous” while making an error in our reasoning. We could miss out on something. Better some more than too little, I think.

How does Pex help me with testing this class method?

Our pex trigger could look like this :

     [PexMethod()]
        public void isOrderToBeAcceptedTest( CustomerType ClientType,
                                            bool GoodpaymentHistory,
                                           bool CreditSufficient)
        {
            OrderManager target = new OrderManager(); 
            bool actual;
            actual = target.isOrderToBeAccepted(ClientType, 
                                                GoodpaymentHistory, 
                                                CreditSufficient);
            PexValue.AddForValidation("actual",actual);
        }

This parameterized test should reach all our code if certain combination of input values is used. So Pex should be able to work with this test method.

You can launch Pex from within Visual Studio 2008 C# by rightclicking the mouse (or pressing CTRL+F8). You will see a menu option ” Run Pex Exploration”. This will launch Pex in doing it’s job, running your code while generating the right combination of input values for each possible code path and then generating the code for all the test cases.



It is the same number of test cases as with our "manual" approach. So this looks pretty good.

  public partial class OrderManagerTest
    {
        [TestMethod]
        [PexGeneratedBy(typeof(OrderManagerTest))]
        public void isOrderToBeAcceptedTestCustomerTypeBooleanBoolean_20080725_143741_000()
        {
            PexValue.Generated.Clear();
            this.isOrderToBeAcceptedTest(CustomerType.U_Unknown, false, true);
            PexValue.Generated.Validate("actual", "False");
        }

        [TestMethod]
        [PexGeneratedBy(typeof(OrderManagerTest))]
        public void isOrderToBeAcceptedTestCustomerTypeBooleanBoolean_20080725_143742_001()
        {
            PexValue.Generated.Clear();
            this.isOrderToBeAcceptedTest(CustomerType.A_Status, false, true);
            PexValue.Generated.Validate("actual", "False");
        }

        [TestMethod]
        [PexGeneratedBy(typeof(OrderManagerTest))]
        public void isOrderToBeAcceptedTestCustomerTypeBooleanBoolean_20080725_143742_002()
        {
            PexValue.Generated.Clear();
            this.isOrderToBeAcceptedTest(CustomerType.A_Status, true, true);
            PexValue.Generated.Validate("actual", "True");
        }

        [TestMethod]
        [PexGeneratedBy(typeof(OrderManagerTest))]
        public void isOrderToBeAcceptedTestCustomerTypeBooleanBoolean_20080725_143742_003()
        {
            PexValue.Generated.Clear();
            this.isOrderToBeAcceptedTest(CustomerType.B_Status, false, true);
            PexValue.Generated.Validate("actual", "True");
        }

        [TestMethod]
        [PexGeneratedBy(typeof(OrderManagerTest))]
        public void isOrderToBeAcceptedTestCustomerTypeBooleanBoolean_20080725_143742_004()
        {
            PexValue.Generated.Clear();
            this.isOrderToBeAcceptedTest(CustomerType.B_Status, true, true);
            PexValue.Generated.Validate("actual", "True");
        }

        [TestMethod]
        [PexGeneratedBy(typeof(OrderManagerTest))]
        public void isOrderToBeAcceptedTestCustomerTypeBooleanBoolean_20080725_143743_005()
        {
            PexValue.Generated.Clear();
            this.isOrderToBeAcceptedTest(CustomerType.B_Status, false, false);
            PexValue.Generated.Validate("actual", "False");
        }

        [TestMethod]
        [PexGeneratedBy(typeof(OrderManagerTest))]
        public void isOrderToBeAcceptedTestCustomerTypeBooleanBoolean_20080725_143743_006()
        {
            PexValue.Generated.Clear();
            this.isOrderToBeAcceptedTest(CustomerType.C_Status, false, true);
            PexValue.Generated.Validate("actual", "True");
        }

    }

We can check the coverage criterium within Visual Studio.



So Pex did an excellent work here. Of course the examples are somewhat academic and only use simple (primitive) types for arguments and/or return values. What about strings or complex types?

My next case study will check out how Pex works on a method accepting a string type. Complex types ....?

Thanks for reading.

Best regards,

Alexander