Pex - test Case 5

Hello,

This post is a continuation in a series of post about Pex, a test case generation tool from Microsoft research. This test scenario is a variation of test case 4. (sorry in advance for the table formatting , I still can not get it right :-(

In the Pex tutorial on p37 you'll see an example of testing a Regular Expression. I didn't find one in the samples though. Too simple maybe?

I'm no hero in regular Expressions. Thankfully there's a library loaded with regular expressions for every kind of application. The following regular expressing is for ensuring strong passwords.

Expression	^(?=.*\d)(?=.*[a-z])(?=.*[A-Z]).{4,8}$
Description	Password matching expression. Password must be at least 4 characters, no more than 8 characters, and must include at least one upper case letter, one lower case letter, and one numeric digit.
Matches	asD1 | asDF1234 | ASPgo123
Non-Matches	asdf | 1234 | ASDF12345

The code example where the regular expression is implemented in

    public class UserAccount
    {

        private string mPassword;

        public string Password
        {
            get { return mPassword; }
            set {
                if (value == null)
                    throw new System.ArgumentNullException("Please specify a valid password. (non-null)");
                else
                   {
                    bool valid = RegularExpressions.Regex.IsMatch(value, @"^(?=.*\d)(?=.*[a-z])(?=.*[A-Z]).{4,8}$");
                    if (valid)
                        mPassword = value; 
                    else
                        throw new System.ArgumentException("Please specify a Strong Password. ");
                    }          
                }
        }
    }

The pex method (keeping in mind the remarks in the comments from the previous test case: the invariant remark)

        [PexMethod()]
        public void PasswordTest( string aPassword)
        {
            UserAccount target = new UserAccount();
            target.Password = aPassword;

            //no problem calling target.password several times, stays the same
            Assert.IsTrue(target.Password != null);
            Assert.IsTrue(target.Password.Length != 0);
            Assert.IsTrue(target.Password.Length >= 4);
            Assert.IsTrue(target.Password.Length <= 8);
            Assert.IsTrue(RegularExpressions.Regex.IsMatch(target.Password, @"[A-Z]"));
            Assert.IsTrue(RegularExpressions.Regex.IsMatch(target.Password, @"[a-z]"));
            Assert.IsTrue(RegularExpressions.Regex.IsMatch(target.Password, @"[0-9]"));
         

            Assert.AreEqual(aPassword, target.Password);
        }

We could do this without Pex of course. We have have to come up with several input values to trigger an exception (at least that is what we intended to program and now are verifying) and some test values for some "positive" tests. The equivalence class technique can surely help us out.

Valid equivalence classes

• bigger or equal than 4 characters


• smaller or equal than 8 characters


• include at least one upper case letter


• include at least one lower case letter


• include at least one numeric digit

Invalid equivalence classes

• not null, non-existing


• smaller than 4 characters


• bigger than 8 characters


• no upper case letter


• no lower case letter


• no numeric digit

You normally start with searching a physical test case (password value) that covers all the valid equivalence classes and for each invalid equivalence class you should also make a test case (i.e. unit test as well). Try not to overlap any invalid equivalance classes so you can be sure that the forced invalidity throwed the exception.

• asDF1234 (covers all valid classes)


• null (non existing)


• Az1 (smaller than 4 chars)


• ASDf12345 (etc)


• abcd1


• ABCD1


• aBcDefGh

So 7 test cases should be enough ?

What would Pex do? It should be as good as this (if not better , at least the tutorial almost implies so).
If you would let the PexMethod work on the password setter method, you would get following results.

Run	aPassword	Summary/Exception	Error Messag
1	null	ArgumentNullException	Value cannot be null. Parameter name: Please specify a valid password. (non-null)
2	""""	ArgumentException	Please specify a Strong Password.
3	\0	ArgumentException	Please specify a Strong Password.
4	\u8000	ArgumentException	Please specify a Strong Password.
6	\0\0	ArgumentException	Please specify a Strong Password.
7	8	ArgumentException	Please specify a Strong Password.
8	\0\n	ArgumentException	Please specify a Strong Password.
10	\0\0\0	ArgumentException	Please specify a Strong Password.
11	\n	ArgumentException	Please specify a Strong Password.
12	\n\0	ArgumentException	Please specify a Strong Password.
16	\0\u8000\u8000	ArgumentException	Please specify a Strong Password.
21	\n\n	ArgumentException	Please specify a Strong Password.
25	\u00062\u00f1\u0016\n\n	ArgumentException	Please specify a Strong Password.
31	\09\u00ef\u0088f\r	ArgumentException	Please specify a Strong Password.
75	\u8000\0\0\0\n\0	ArgumentException	Please specify a Strong Password.
83	0\u0088o]O<
112	\u00062Oo\n\n	ArgumentException	Please specify a Strong Password.
113	\u00062Oo\n
114	\u00062Oo\n\n\n\n\n\n\n\n\n\n	ArgumentException	Please specify a Strong Password.

Pex established 19 test cases. A little bit more if you would go for the manual approach. But are the test input values better? In terms of "readability" no. But Pex generated some input values that I wouldn't have thought of in the first place.

The empty string was added in the Pex generated approach. In our manual exercise with didn't explicitly took that test case because this was implied in the "smaller than 4 chars" equivalence class. But again testing a little bit too much is maybe better (or saver) than testing too little.

Also the Pex generated unit tests include non alfanumeric signs like * or /. Did I intended to do that with my expression? If not I have to re-write it. With my manual technique I "forget" them. So Pex went a little deeper than my test analysis.
What about Escape characters such as "\n" (new line). pex included them and some tests passed. Was this the way I intended it? Were these the specs from the customer?

The examples I used to get acquinted with Pex may seem academic but I believ show potential for a tool like Pex. How to use it in your day-to-day applications will take some time and study. And like the Pex FAQ already states : it is not Pex versus unit testing (or other forms of testing or QA assurance measures like code analysis for example). I hope the Pex team will (continue to) find sponsers within Microsoft to integrate it in the next releases of Visual studio. At least the pex team already took a patent!.

Thanks for reading.

Best regards,

Alexander

Pex - test case 4

Hello,

I've got one more test scenario for Pex. From comments on my previous post
Case study 3 I understood that the way I specified the PUT was not done correctly (though Pex generated the unit test methods I expected).

A PUT should specify what the intended behaviour of the code under test should do. It should not do this like in a tradiotional unit test for a particular combination of test input but in a more general way.

Sounds reasonable but I find some difficulity in writing them. I'll try to come with a "better" PUT this time.

Our code under test is as follows (Sorry in advance for some bad formatting HTML tables later in the post.):

      private string mCode;

        public string Code
        {
            get { return mCode; }
            set {
                if (value == null ) 
                    {
                    throw new System.ArgumentNullException("Please specify a valid code. (non-null)");    
                    }
               else if (value.Length < 4)
                {
                    throw new System.ArgumentException("Please specify a valid code. (bigger than 3 characters.");
                }
                else if (value.Length > 10)
                {
                    throw new System.ArgumentException("Please specify a valid code. (max 10 characters. including XYZ prefix) ");  
                }
                else if (value.StartsWith("XYZ") == false)
                {
                    throw new System.ArgumentException("Please specify a valid code (starting with XYZ - uppercase !) ");
                }
                else
                {
                    this.mCode = value;
                }
              
              }

With the comments for my PUT in mind , I created the following PUT:

       [PexMethod()]
         public void CodeTest(string aCode)
         {
             measurement target = new measurement();
             string actual;
             target.Code = aCode;
             actual = target.Code;
             PexValue.AddForValidation("actual", actual);
             if (( aCode != null) && 
                 (aCode.Length > 3) &&  
                 (aCode.Length <= 10) && 
                 (aCode.StartsWith("XYZ")) ) 
                 PexAssert.IsTrue(actual == aCode);
            
         }

Running Pex on this PUT will result in several generated test cases.
(Pex reporting option : Enable via Options/Pex/Reports)

Parameter Values
Run	aCode	actual	Summary/Exception
1	null		ArgumentNullException: Value cannot be null. Parameter name: Please specify a valid code. (non-null) (testingPEX)
2	""		ArgumentException: Please specify a valid code. (bigger than 3 characters. (testingPEX)
3	\0\0\0\0\0\0		ArgumentException: Please specify a valid code (starting with XYZ - uppercase !) (testingPEX)
4	new string('\0', 14)		ArgumentException: Please specify a valid code. (max 10 characters. including XYZ prefix) (testingPEX)
5	XYZZZZZ	XYZZZZZ

Of course we first need to specify the expected exceptions (Pex will suggest these for you !) :

• Add attribute [PexAllowedExceptionFromAssembly(typeof(ArgumentNullException), "testingPEX")]
  


• Add attribute [PexAllowedExceptionFromAssembly(typeof(ArgumentException), "testingPEX")]
  

The generated unit tests look like this :

  public partial class measurementTest
    {

        [TestMethod]
        [PexGeneratedBy(typeof(measurementTest))]
        public void CodeTestString_20080728_141020_005()
        {
            PexValue.Generated.Clear();
            this.CodeTest("XYZZZZZ");
            PexValue.Generated.Validate("actual", "XYZZZZZ");
        }

        [TestMethod]
        [ExpectedException(typeof(ArgumentNullException))]
        [PexGeneratedBy(typeof(measurementTest))]
        public void CodeTestString_20080728_143541_000()
        {
            this.CodeTest((string)null);
        }

        [TestMethod]
        [ExpectedException(typeof(ArgumentException))]
        [PexGeneratedBy(typeof(measurementTest))]
        public void CodeTestString_20080728_143620_001()
        {
            this.CodeTest("");
        }

        [TestMethod]
        [ExpectedException(typeof(ArgumentException))]
        [PexGeneratedBy(typeof(measurementTest))]
        public void CodeTestString_20080728_143620_002()
        {
            this.CodeTest("\0\0\0\0\0\0");
        }

        [TestMethod]
        [ExpectedException(typeof(ArgumentException))]
        [PexGeneratedBy(typeof(measurementTest))]
        public void CodeTestString_20080728_143620_003()
        {
            this.CodeTest(new string('\0', 14));
        }

    }

If you would verify the code coverage results in Visual Studio or via the Pex reporting option, you would see 100% Block coverage.

Pex generated for us the strings to serve as input values. I don't know about you but I think this is pretty clever.

Still I'm a little confused about the PUT I created. If I would delete all the pexAssert statements the exercise would still be the same for the generated unit tests.

  [PexMethod()]
         public void CodeTest(string aCode)
         {
             measurement target = new measurement();
             string actual;
             target.Code = aCode;
             actual = target.Code;
             PexValue.AddForValidation("actual", actual);  
         }

Thanks for reading.

Best regards,


Alexander

Pex - Test case 3

Hello,

This post is a continuation of previous posts. Feel free to check them out.

• Introduction


• Case study 1


• Case study 2

In this example I base my example on the following rule:
In our Order system we need to implement a validation rule to see whether we can accept an order or not. The criteria for acceptance where given to us (luckily) in the form of a decision table. There are three criteria ; an internal customer qualification scheme, the fact if the previous payments of the customer were usually on time and correct and the fact if the customer has sufficient credit possibilities.

Cust-code

a

a

a

a

b

b

b

b

c

c

c

c

u

u

u

u

paymentsOK

y

y

n

n

y

y

n

n

y

y

n

n

y

y

n

n

CredietLimitOK

y

n

y

n

y

n

y

n

y

n

y

n

y

n

y

n

AcceptOrder

y

y

n

n

y

y

y

n

y

y

y

y

n

n

n

n

    public enum CustomerType 
        { 
            U_Unknown = 0, 
            A_Status = 1, 
            B_Status = 2, 
            C_Status = 3 
        } 

    public class OrderManager
    {
        public bool IsOrderToBeAccepted(CustomerType aClientType,  bool aGoodpaymentHistory,  bool aCreditSufficient) 
            { 
                bool orderAccept = false; 
                switch (aClientType) { 
                    case CustomerType.A_Status: 
                        if (aGoodpaymentHistory == true)  { 
                            orderAccept = true; 
                        }
                       else
                        {
                            orderAccept = false;
                        }
                        break; 
                    case CustomerType.B_Status: 
                        if (aGoodpaymentHistory || aCreditSufficient) { 
                            orderAccept = true; 
                        } 
                        else { 
                            orderAccept = false; 
                        } 
                        break; 
                    case CustomerType.C_Status: 
                        orderAccept = true; 
                        break; 
                    default: 
                        orderAccept = false;  
                        break; 
                } 
            return orderAccept; 
            } 
    }

With the help to the decsion table (thanks to the analyst :-), we can easily determine the test cases.
There 2 test sitaution (different behaviours) we must account for : the order is accepted or is not accepted. So we need test cases that will result in a non-accepted order and we will need a test case that will result in an accepted order. Those this mean two test cases (i.e. unit tests) are enough. There is always this "test completeness criterium". If this were a black-box test (we don't "see" the code) we could use a tool like Pict33 to propsose the combination of input values.

In this "lab" test we both have design information (desicion table) and our code. The table in this case can help us enough. Each column will represent the values we will supply to the method under test. This means we’ll end up with 16 unit tests to achieve full coverage of our table (and hopefully also of our code). The unit test are there to verify if our “intentions” (i.e. implementing this method with the above rules) were correctly transformed into code. Both test situations (order accepted / order not accepted) should normally be taken into account with these 16 test cases.

We can write out our units tests :

  [TestMethod()]
        public void isOrderToBeAcceptedCase1()
        {
            OrderManager target = new OrderManager();
            CustomerType ClientType = CustomerType.A_Status;
            bool GoodpaymentHistory = false; 
            bool CreditSufficient = false; 
            bool expected = false; 
            bool actual;
            actual = target.isOrderToBeAccepted(ClientType, GoodpaymentHistory, CreditSufficient);
            Assert.AreEqual(expected, actual);
           
        }

       [TestMethod()]
       public void isOrderToBeAcceptedCase2()
       {
           OrderManager target = new OrderManager();
           CustomerType ClientType = CustomerType.A_Status;
           bool GoodpaymentHistory = true;
           bool CreditSufficient = false;
           bool expected = false;
           bool actual;
           actual = target.isOrderToBeAccepted(ClientType, GoodpaymentHistory, CreditSufficient);
           Assert.AreEqual(expected, actual);

       }

        //and so one 

Or we make a data-driven unit test.

      [DeploymentItem("TestProject1\\OrderAcceptanceCombinations.csv")]
        [DataSource("Microsoft.VisualStudio.TestTools.DataSource.CSV", 
            "|DataDirectory|\\OrderAcceptanceCombinations.csv",
            "OrderAcceptanceCombinations#csv", DataAccessMethod.Sequential)]
        [TestMethod]
        public void DataDrivenIsOrderAccepted()
        {
            OrderManager target = new OrderManager();
            CustomerType ClientType =  (CustomerType)(System.Convert.ToInt32(this.TestContext.DataRow["CustomerType"])) ;
            bool GoodpaymentHistory = System.Convert.ToBoolean(this.TestContext.DataRow["PaymentsOK"]);
            bool CreditSufficient= System.Convert.ToBoolean(this.TestContext.DataRow["CreditOK"]) ;
            bool actual;
            bool expected = System.Convert.ToBoolean(this.TestContext.DataRow["AcceptOrder"]) ;
            actual = target.isOrderToBeAccepted(ClientType, GoodpaymentHistory, CreditSufficient);
            Assert.IsTrue(actual == expected);
        }

Bottom line is we achieve acceptable code coverage. We can verify this with the Visual Studio Code Coverage facility of the test framework.

In that respect we can actually reduce the number of test cases and still achieve full coverage. A technique we can use, is "collapsing" the desicion table. If we closely look at our rules, we can detect that several combinations should result in the same result. This means that some conditions are not taken into account (or don’t make sense) in some test cases. For example whether or not a customer with customer-type UnKnown has good creditRecords or even goodpayments records, it doesn’t matter. We don’t trust “Unknown” customers. We don’t allow an order for them! In order words we can reduce the number of test cases for 4 to 1

If we do this exercise for the entire table , we should come up with the following decision table.

Cust-code	a	a	b	b	b	c	u
paymentsOK	y	n	y	n	n	-	-
CredietLimitOK	-	-	-	y	n	-	-
AcceptOrder	y	n	y	y	n	y	n

This would mean we only need to 7 test cases to fully test our rule. Collapsing tables is powerful in terms of test case reduction but can also be “dangerous” while making an error in our reasoning. We could miss out on something. Better some more than too little, I think.

How does Pex help me with testing this class method?

Our pex trigger could look like this :

     [PexMethod()]
        public void isOrderToBeAcceptedTest( CustomerType ClientType,
                                            bool GoodpaymentHistory,
                                           bool CreditSufficient)
        {
            OrderManager target = new OrderManager(); 
            bool actual;
            actual = target.isOrderToBeAccepted(ClientType, 
                                                GoodpaymentHistory, 
                                                CreditSufficient);
            PexValue.AddForValidation("actual",actual);
        }

This parameterized test should reach all our code if certain combination of input values is used. So Pex should be able to work with this test method.

You can launch Pex from within Visual Studio 2008 C# by rightclicking the mouse (or pressing CTRL+F8). You will see a menu option ” Run Pex Exploration”. This will launch Pex in doing it’s job, running your code while generating the right combination of input values for each possible code path and then generating the code for all the test cases.



It is the same number of test cases as with our "manual" approach. So this looks pretty good.

  public partial class OrderManagerTest
    {
        [TestMethod]
        [PexGeneratedBy(typeof(OrderManagerTest))]
        public void isOrderToBeAcceptedTestCustomerTypeBooleanBoolean_20080725_143741_000()
        {
            PexValue.Generated.Clear();
            this.isOrderToBeAcceptedTest(CustomerType.U_Unknown, false, true);
            PexValue.Generated.Validate("actual", "False");
        }

        [TestMethod]
        [PexGeneratedBy(typeof(OrderManagerTest))]
        public void isOrderToBeAcceptedTestCustomerTypeBooleanBoolean_20080725_143742_001()
        {
            PexValue.Generated.Clear();
            this.isOrderToBeAcceptedTest(CustomerType.A_Status, false, true);
            PexValue.Generated.Validate("actual", "False");
        }

        [TestMethod]
        [PexGeneratedBy(typeof(OrderManagerTest))]
        public void isOrderToBeAcceptedTestCustomerTypeBooleanBoolean_20080725_143742_002()
        {
            PexValue.Generated.Clear();
            this.isOrderToBeAcceptedTest(CustomerType.A_Status, true, true);
            PexValue.Generated.Validate("actual", "True");
        }

        [TestMethod]
        [PexGeneratedBy(typeof(OrderManagerTest))]
        public void isOrderToBeAcceptedTestCustomerTypeBooleanBoolean_20080725_143742_003()
        {
            PexValue.Generated.Clear();
            this.isOrderToBeAcceptedTest(CustomerType.B_Status, false, true);
            PexValue.Generated.Validate("actual", "True");
        }

        [TestMethod]
        [PexGeneratedBy(typeof(OrderManagerTest))]
        public void isOrderToBeAcceptedTestCustomerTypeBooleanBoolean_20080725_143742_004()
        {
            PexValue.Generated.Clear();
            this.isOrderToBeAcceptedTest(CustomerType.B_Status, true, true);
            PexValue.Generated.Validate("actual", "True");
        }

        [TestMethod]
        [PexGeneratedBy(typeof(OrderManagerTest))]
        public void isOrderToBeAcceptedTestCustomerTypeBooleanBoolean_20080725_143743_005()
        {
            PexValue.Generated.Clear();
            this.isOrderToBeAcceptedTest(CustomerType.B_Status, false, false);
            PexValue.Generated.Validate("actual", "False");
        }

        [TestMethod]
        [PexGeneratedBy(typeof(OrderManagerTest))]
        public void isOrderToBeAcceptedTestCustomerTypeBooleanBoolean_20080725_143743_006()
        {
            PexValue.Generated.Clear();
            this.isOrderToBeAcceptedTest(CustomerType.C_Status, false, true);
            PexValue.Generated.Validate("actual", "True");
        }

    }

We can check the coverage criterium within Visual Studio.



So Pex did an excellent work here. Of course the examples are somewhat academic and only use simple (primitive) types for arguments and/or return values. What about strings or complex types?

My next case study will check out how Pex works on a method accepting a string type. Complex types ....?

Thanks for reading.

Best regards,

Alexander

Pex - test case 2

Hello,

I'm currently trying out Pex , an automated unit test generator. I'm trying out several scenarios. If you like you can read about a previous test case I tried out. In the same spirit , I'll start of with what you could do without Pex.

The code snippet is as follows. There is a "measurement class" that implements the following method (property).

     private int mTemperature;

        public int Temperature
        {
            get { return mTemperature; }
            set { 
                if (value > -274 & value < 1000)
                {
                     mTemperature = value; 
                }
                else
                {
                    throw new System.ArgumentOutOfRangeException("Please .....");    
                }
               
                }
        }

There are two test situations (different behaviours) in this unit to be tested.

• an integer value is assigned to a field


• an exception is thrown

If you would use the Equivalence class technique to establish a good test suite, you could come up with following findings.
Valid classes

• integer value


• integer value between -274 and 1000

Invalid classes

• NOT an integer value


• integer value smaller or equal than -274


• integer value bigger or equal than 1000

Because the .NET type system gives us a hand (int) , our method only can accept integers. So we only have 1 valid class and two invalid classes or one if you would combine then in one sentence (Not between -274 and 1000). Both approaches should cover our two test situations (assignation and exception).

Now we have to come up with actual values for these classes which shouldn't be too difficult. This would result in two physical test cases (i.e. unit tests). We could pick -30 and 1500 . Or -650 and 0. And so one... All these values should give equal test results while assuring complete coverage.

In case of two invalid classes you will have three test cases :

• valid : 25


• invalid 1 : - 300


• invalid 2 : 1250

A common place for errors are the comparison signs. The Border Value Analysis "technique" stipulates you should pick values around the "borders". In our example these are -274 and 1000. In addition you could also pick values aboove (and/or under) these border values. Any mistake (current or future) should be detected by our set of test cases (i.e. unit tests).

• valid 1: 273


• valid 2: 999


• invalid 1 : - 274


• invalid 2 : 1000

So which unit tests will Pex generate for us?

         [PexMethod()]
        public void TemperatureTest(int aTemperature)
        {
            measurement target = new measurement();
            int actual;
            target.Temperature = aTemperature;
            actual = target.Temperature;
            PexValue.AddForValidation("actual",actual);
        }

Pex, using the specified PexMethod, explored our code. It took him two runs to come up with the necessary values for temperature. On the last value (int.MinValue : the smalles possible 32-bit integer value in .NET ) it discovered an exception. Just like in "traditional" unit testing you can specify an "expected exception" . meaning when this exception is thrown, the test is succesfull because that is what you ....expected.



If you re-run the PexMethod again you will see two classic unit test.

   public partial class measurementTest
    {
        [TestMethod]
        [PexGeneratedBy(typeof(measurementTest))]
        public void TemperatureTestInt32_20080725_080720_000()
        {
            PexValue.Generated.Clear();
            this.TemperatureTest(0);
            PexValue.Generated.Validate("actual", "0");
        }

        [TestMethod]
        [ExpectedException(typeof(ArgumentOutOfRangeException))]
        [PexGeneratedBy(typeof(measurementTest))]
        public void TemperatureTestInt32_20080725_082344_001()
        {
            this.TemperatureTest(int.MinValue);
        }

    }

If you run these two unit tests and enable Code Coverage for the target assembly in VS Test framework, you will notice full coverage as understood by VS test framework.

If you compare the Pex analysis with our "manual" analysis you can see that Pex took the minimal set of test cases to achieve full coverage (as understood by Visual Studio). For example someone would change the comparision operator in the method, the unit tests generated by Pex will not detect it in this particular example.


But if you would slightly change the pexMethod (tip from Peli de Halleux) you would see one extra unit test. You need to re-write the assert to verify the border values.

      [PexMethod()]
        public void TemperatureTest(int aTemperature)
        {
            measurement target = new measurement(); 
            int actual;
            target.Temperature = aTemperature;
            actual = target.Temperature;
            PexAssert.IsTrue((actual > -274 & actual < 1000));
        }

The unit tests generated by Pex would now include the 1000 border value. I don't understand yet why -274 wasn't picked up. I guess it has to do with the fact 0 was the value chosen in the first run. Maybe there are better ways to write this PexMethod? I'll will need to look into the "assumptions" part of Pex (PexAssume class).

    public partial class measurementTest
    {
        [TestMethod]
        [PexGeneratedBy(typeof(measurementTest))]
        public void TemperatureTestInt32_20080725_111609_000()
        {
            this.TemperatureTest(0);
        }

        [TestMethod]
        [ExpectedException(typeof(ArgumentOutOfRangeException))]
        [PexGeneratedBy(typeof(measurementTest))]
        public void TemperatureTestInt32_20080725_111609_001()
        {
            this.TemperatureTest(int.MinValue);
        }

        [TestMethod]
        [ExpectedException(typeof(PexAssertionViolationException))]
        [PexGeneratedBy(typeof(measurementTest))]
        public void TemperatureTestInt32_20080725_111953_002()
        {
            this.TemperatureTest(1000);
        }

    }

Is it worthwile to look for test cases but beyond absolute minimal? Well, it dependends. One of the advantages of of having (automated) unit tests is the ability to use them over and over again. In the context of regression testing these test suitses can be of great help to assure that all code that is using your unit (i.e. method) will still function after you made modifications. or least you will know in which situation something will break so you can take appropriate measures (defaults, overloads, release notes ,...).

         private int mTemperature;

        public int Temperature
        {
            get { return mTemperature; }
            set { 
                if (value > -274 & value <= 1000)
                {
                     mTemperature = value; 
                }
                else
                {
                    throw new System.ArgumentOutOfRangeException("Please ...");    
                }
               
                }
        }                             
    

So if regression testing is of high priority , it think some methods need those extra test cases. They can happily live beside the ones generated by Pex. It is sometimes (value of software; money, safety) better to have "too much" than "too little" test cases I think (even if it means extra (test) code to maintain.

As shown in the example, the way you setup your Pexmethod will influence how Pex "explores" your code. Although the the call to the (setter) method was the same. The Assertion part of the Pexmethod made Pex do an extra effort.

If you have other thoughts or remarks about pex or devloper testing in general, don't hesitate to drop a comment.

Thanks for reading.


Best regards,


Alexander

Pex - test case 1

Hello,

I just started investigating Pex . In order to learn the tool I'm using several (academic) test cases to see how Pex can help there.

This case case is called Commercial Cut percentage for an order. In our Order system we have implemented a rule that will calculate the commercial cut based on the total OrderAmount. The following table depicts the rules for applying the commercial cut.

Total (rounded) order-amount (€)	<=1500	[1501-5000]	[5001-10000]	[10001-25000]	>25000
Commercial cut	0	2	3	4	5

I have implemented this rule in a c# method. For the sake of this exercise I will not argue about whether or not this should be a private method (Which by the way can perfectly be unit tested in the VS Test framework).

       public uint DetermineCommercialCut(uint aTotalOrderAmount)
       {

           if (aTotalOrderAmount <= 1500)
           {
               return 0;
           }
           else if (aTotalOrderAmount > 1500 & aTotalOrderAmount <= 5000)
           {
               return 2;
           }
           else if (aTotalOrderAmount > 5000 & aTotalOrderAmount <= 10000)
           {
               return 3;
           }
           else if (aTotalOrderAmount > 10000 & aTotalOrderAmount <= 25000)
           {
               return 4;
           }
           else
           {
               return 5;
           }
       }
   

Of course we will unit test this class method. This method should always return some positive integer value back depending on total order amount that is supplied via unsigned integer parameter. So I give a positive integer (or zero) to the method and I receive a positive integer (or zero) back. Of course the purpose of the method is determine the commercial cut percentage based on the order amount. Our table showed on the rules we need to apply and our code is (hopefully) constructed in the same way. So we will construct a unit test that will call this method.

   [TestMethod()]
   public void DetermineCommercialCut_ZeroOrderAmount()
      {
            Order target = new Order();
            uint aTotalOrderAmount =0;
            uint expected = 0;
            uint actual;
            actual = target.DetermineCommercialCut(aTotalOrderAmount);
            Assert.IsTrue(expected == actual);
   }

We could continue with testing all positive integers up to 4.294.967.295. Of course this is not worthwhile. We can pick certain values for this entire collection in such way that any value within certain ranges of this big collection , will (should) behave in the same way. This is referred to as equivalence classes in testing literature. So whether I pick 0 or 1500 as total (rounded) order amount or any value in between, it should always resolve to a return value of zero. So based on this principle I can detect several equivalence classes.

• Must be positive integer


• Integers between 0 and 1500 included


• Integers between 1501 and 5000 included


• Integers between 5001 and 10000 included


• Integers between 10001 and 25000 included


• Integers higher then 25000

I’m spotting 5 valid classes. Because the.NET type system helps me out here I don’t need explicit checks on the invalid class namely negative integers.

now I need to pick a value I will actually use in my unit tests. I will at least need to pick 5 values, one from each equivalence class. In my test I will verify if the reduction cut is appropriate for that value.
So instead of making 4.294.967.295 unit tests I only need 5 to “cover” all possible test situation ( different commercial cut percentages).

Another technique that is usually combined with the former is border value analysis. In the example I can with a minimal additional effort also verify if the comparison signs are as they were intended to be. Instead of picking any value in the equivalence class , I will look up the values that are used in the conditions for example. Instead of 2531 , I will pick 5000 just at the border. So 5000 should give me a commercial cut of 2. Just like 2531 but I have an extra check on the comparison sign with no extra effort. With a little more effort I could also include the values just under (or above) the border.

So basic techniques told me I need 5 test case to fully test my rule (i.e. Method). So this means 5 Visual Studio test methods. Actually , Visual Studio test Framework can, in this case , help us even further with reducing the number of unit test to …..one. That is one data-driven unit test. What we do, is externalize the test data values into for example an Excel file. This data-source will then “feed” our data-driven unit test with actual values. So although we have only one written unit test, we would like this test code to be executed 5 times. We now use this table (with some modification) as data-source for our data-driven unit test in Visual Studio 2008. For more info see MSDN.

        [DataSource("Microsoft.VisualStudio.TestTools.DataSource.XML",
                     "DataDirectory\\testData.xml",
                     "CommercialCut", DataAccessMethod.Sequential),
        DeploymentItem("TestProject1\\testData.xml")]
        [TestMethod()]
        public void DetermineCommercialCut_DataDriven()
        {
            Order target = new Order();
            uint aTotalOrderAmount = System.Convert.ToUInt32(this.TestContext.DataRow["input"]);
            uint expected = System.Convert.ToUInt32(this.TestContext.DataRow["expected"]);
            uint actual;
            actual = target.DetermineCommercialCut(aTotalOrderAmount);
            Assert.IsTrue(expected== actual);
        }

We determined on basis of our analysis that 5 test-cases should cover the testing of the rule. Here Visual Studio Test framework also can help us by instrumenting the code before executing the unit tests (Code coverage). A nice feature of the Visual Studio Test framework is the colorization of the code under test to show which parts have been “touched” by the unit tests and which parts didn’t.

When looking at the Visual Studio Code coverage results of one test executed :





You can see the statements in different coloring; touched versus not touched. This should tell you that you must add some test cases to see code paths in your method also work as you intended and consequently coded.

If you would run the data-driven test with the 5 test cases you would get the following result :




Everything is highlighted in blue, meaning that every “block” in our code has been executed during the run of our unit test. That should come as no surprise because we rigorously followed our table for our rule.

So far our “manual” effort . Lets see how Pex can help you. So the documentation states that Pex can create a test suite with a high coverage by exploring your code triggered from a parameterized test you will have to write.

I highly recommend reading the tutorial for more background information on Pex. Meanwhile I will continue with the example to show you how Pex can create a test suite for us.

First we must create a “parameterized” unit test to give Pex an entry point into our code under test. “Parameterized Unit Tests (PUTs) are... tests that take parameters. Unlike traditional unit tests, which are usually closed methods, PUTs take any set of parameters. Is it that simple? Yes! From there, Pex will try to generate the (minimal) set of inputs that fully coverthe code reachable from the test. “

So in our case this could look something like:

[PexMethod()]
public void CaluclateReductionPercentageTest(uint aSales)
{
 Order target = new Order();
 uint actual;
 actual = target.DetermineCommercialCut(aSales);
 PexValue.AddForValidation("result", actual);
}

You can launch Pex from within Visual Studio 2008 C# by right clicking the mouse
(or pressing CTRL+F8). You will see a menu option ” Run Pex Exploration”.
This will launch Pex in doing it’s job, running your code (several times) while generating the
right combination of input values for each possible code path and then generating the code
for all the test cases.

Pex generated 5 tests for us. It clearly also detected (in another way) that there are 5 test cases needed to cover the code. If we take our 5 equivalence classes we can see that this is correct. If our “conduct-of-test” specifies the use of border-value analysis where appropriate , then these values are not so well chosen in my opinion.

Lets say that we want to add an extra level of commercial cut for sales between 1000 and 1500 Euros. And we want to let our client enjoy a commercial cut of 1 percent.

In the manual test analysis we would come up with an additional equivalence class. Pex can also detect this.

But first lets re-run our initial set of unit tests generated by pex.

Our Code Coverage run should warn us that we missed out on some parts of the code. We need another unit test. To do so , we will re-run Pex on our pexMethod. The will force pex to examine our code again and it should find an extra path in our code.

Doing so resulted in pex finding an additional test (nr 3 - see screenshot). If we run the generated test methods and also look at the coverage findings we find a positive result.

So in a nutshell. In this (although academic) situation, Pex found the same test cases if we would use a manual technique. Border value Analysis would be advantageous in this test case but Pex doesn't propose these values while exploring this method based on the PexMethod we proposed.

So the most important lesson I think is , as also stated in the Pex FAQ : Use Pex as one of your tools not as the (only) tool.

Maybe you also have tried out Pex? Don't hesitate to share....

Thanks for reading.

Best regards,

Alexander

ps: I'm still experimenting with SyntaxHighLighter. If you have any tips how to improve showing code snippets in Blogger...let me know.

Starting with Pex (Program Exploration)

Hello,

I recently downloaded Pex from Microsoft Research. Pex is a tool for finding test cases for you on basis of “looking” and “executing” your code. It is fully integrated in Visual Studio 2008 (c#). Pex explores your code by executing special Pex test method you have written. It basically comes down by making a “parameterized” version of a unit test you would normally make. This Pex test method will then call you unit under test (i.e. class method). Pex supplies values for the parameterized unit test . Meanwhile it will instruments your code do see which conditions it needs to check and which code paths it took during the exploration phase. After completing it adventure, it will generated unit test code. This unit test can then be executed by the Visual Studio Test Framework. For more information..... Before taking Pex for a spin , let's look at why a Pex was created.


Automated Unit testing is considered a good practice in software development. Nevertheless it is usually perceived as a burden by the developer. Besides the writing the actual code the programmer has to create and maintain this extra test code and of course exercising these unit tests. Depending on the code the amount of test cases , hence the unit test, can quickly grow. Also if you exercise the practice of “good” code coverage by your unit tests, you’ll discover that writing unit tests is hard work.

Sometimes the amount of test code supersedes the code under test. Much effort goes into creating the right amount of test cases so they cover every “corner” of you code. This code coverage criterion should result in a higher probability to find errors in your code. There several techniques that can help you establish these test cases. But these still leave writing the unit tests yourself.

Here's where pex steps in. Pex from Microsoft research can help achieve this coverage criterion by dynamically introspecting your code and based on that analysis can generate different test cases in the form “traditional” unit tests for the Visual Studio test framework.

I currently trying out some scenario's. Keep an eye for future posting...

Thanks for reading.

Best regards,


Alexander

Design for Testability - Dependencies

Hello,


This post is a continuation on previous posts about design-for-testability (here and here).

Control and observation are important characteristics in test-automation. If our code under test makes an effort to facilitate this , we improve the testability of our code. Improving the testability should lead to find any errors in our code more easily. The more we errors we find early in the development cycle, the cheaper it is to correct them. We early testing and defect correction, we get a good feel on the quality of the software. We can make educated decisions on when to promote the code to QA and/or production. When our customer can use our software to do their job , they create value for the company they work for ….(I will stop rambling here) . But you see , testability (and in a broader sense quality assurance) should be baked into the software. The entire organization can benefit from it.

Let’s start on the “control” aspect of our code under test (CUT) and specifically on the controlling the dependencies of our CUT with other parts of the program or the environment it runs in.

It is in the nature of an object-oriented language to build up a program from lot of different “units” where some “units” or better classes work closely together to make something happen. Object-oriented principles like single-responsibility, information-hiding , inheritance , etc are common in this world. So it is quite natural that classes have dependencies on other classes. The term coupling is often used to state the existence of relationships between class (inheritance, composition , etc). coupling is evitable. If there were no coupling we would end up with monolithically part of codes, actually they would be separate programs. The large classes would be too complex and the cohesion of the operations and member data would very low.

In the realm of testability we sometimes want to be able to test a class in a controlled way, maybe even in total isolation. In order to do that we must be able to control the dependencies. The fewer the dependencies the easier it will be. Hence “managed” coupling can increase testability.

Programs run on machines and make also use of their environment. For example the file system, Message Queue system, database systems , web services, etc. These resource dependencies are expressed in the programs usually via string tokens. For example a folder name, database connection string, queue name, web service URL etc. A good practice of course is to externalize the values for these resources for example in a configuration file. These practice is also beneficial for testability. Avoiding “hard-coded” resource strings in your code is of course not a good programming practice. Because many times the resource string are depended on where the program actually will run. For example development , production or …testing. So controlling the resource tokens outside the program can give us flexibility will setting up tests.
.NET has excellent support to easily consume these externalized resource identifiers in your programs. In the Visual Studio test framework you can easily let the test framework pick up these configuration files and let your code under test use the configured element just like if the code under test was running in their proper host (windows app, web service, etc). In other words , you will copy the configuration info that you need for the code under test from the base app.config or web.config in your test project and modify it as needed.

Let’s focus again on class dependencies. So dependencies are inevitable but we can make them more manageable for testing purposes ( or other reasons). In order to continue our discussion lets introduce some terms. If a class A depends on another class B to fulfil its functionality A is called a client class and B is called a server class. During unit test we want to test a class in isolation. Therefore we want to substitute its server classes by stubs or mock objects . Hard-wired dependencies to server classes hinder this. A dependency on a server class is hard-wired if the server class can not be substituted by some other class (e.g. a subclass of the server class) without changing the source code of the client class. Similar problems arise during integration testing if a server class should be stubbed because it is not test-ready. So how can we manage class dependencies that will benefit testability.
One of the areas lies in the construction of the dependencies. We don’t control them. So our solution strategy should revolve around getting control about the creation of the dependencies. Several techniques can help us to come up with a solution:

• Interface-based programming : Separating the interface from the implementation is core principle. Our client class uses an abstraction of a server class (interface) , not a particular implementation of it (object). So during testing we could create another implementation of the server class that implements the correct interface. This “test-double” (Gerard Meszaros , xUnit patterns) version of our server class can for example always give a fixed set of values back or just ignores the fact that we ask it to do something. Interfaces are first-class citizen in .NET so you should have no problems with this. But this technique is just a first step toward the solution. We still need to hook up an implementation of the server class into the client class at test time.
• Service locator: We would delegate the creation of objects to a specialized object that for example would fabricate objects based on a string identifier.
• Provider model : Variation of Interface-based programming :. Abstract class , concrete implementation hooked up at runtime (configuration file). You can a test-specific implementation of the provider in order to control the tests
• Manual Dependency injection : We will make another class responsible to establish the link between the client and the server. We will add a parameter of the server type to a method of the client class (a constructor, or a method that requires access to a server instance, or a dedicated setup method) which allows other objects to set the link. During testing the parameter(s) can be used to establish a link to a “test-double” instead.
• Automatic dependency injection : a DI container framework takes care of the details of which objects are interconnected, so you can build each one independently. No need for passing the dependencies along with the constructor or methods or assigning properties with dependent objects

Still there some concerns (or trade-offs) you should take into account;

• Information hiding :Why can’t leave it up to A to know its dependencies : Exposing the dependencies in a constructor function can be viewed a violation of encapsulation because now a client of Class A would have to create instances of Class B first before calling the constructor function of class A
• The ability to substitute a server class is not always important for the “production” implementation. So is testability a good enough reason for implementing this possibility?
  We could make a special constructor that will accept the dependencies. The default constructor would use that hard-wired implementations.
  If there are dependencies on many server classes, this approach would result in too many parameters.
• DI Framework to the rescue ...but Someone has to know the dependencies
  Assembler , DI Framework via declarations or via config files
  

There are still other things besides managed coupling you can work on to improve testability , hence the quality of our software : cohesion, encapsulation, ... But that's for another post.

Maybe you have others techiques to improve testability? If you would like to share them, don't hesitate to drop a commont.

Thanks in advance.

Best regards,

Alexander

Winforms (VS2005) DataGridViewComboBoxColumn, CTRL+0 and Cannot set Column 'myColumnName' to be null. Please use DBNull instead.

Hello,

I recently had some trouble with setting a dataGridViewComboboxColumn to a dbNull value with the CTRL+0 (zero) keystroke. As you may know, the datagrid supports the CTRL+0 keycode to force a dbNull value into cell and subsequentialy into the bound datasource (a datatable in my case). In other words you can use that to clear out a combo box's value and set it back to "nothing". At least when the bound datatable column allows DbNull to be true.

This is where I experienced some strange behaviour. I used a datagridView to display some customer data. One of the fields specifies an optional characteristic , namely the qualification of a customer. At database level this is enforced with a referential integrity constraint in the form of foreign key specification with a look-up table. The primary key of this look-up table is an integer. Of course in the datagrid , on the comboxbox showing the customer qualification , you will see a description (cfr. valuemember versus DisplayMember).

This is where it goes wrong. When I try to clear an existing value in the combobox , I get the following error message. Cannot set Column '' to be null. Please use DBNull instead.

What is even more strange is that when you have a combobox in the datagridview where the valueMember's type is System.String, You will not have this exception. It just works. It is also only apparent with a datagridViewComboxBoxColumn. Because if I would use a
DataGridViewTextBoxColumn for the "integer" case, I would neither experince any problem with CTRL+0.

I looked for a solution on Google and I stumbled on this post . Somewhere on the middle you will see an entry by Mark Rideout , DataGridView Program Manager Microsoft (27 Jan 2006, 3:34 PM ) . ...And he talks about a bug. But more interssenting for me he also hints a possible work-around.

He talks about overriding the DataGridView.ProcessDataGridViewKey Method . This method calls the key-processing method appropriate to the key pressed. In this method you could check for the CTRL+0 keystroke (D0) . That is exactly what I immediately tried out. I made a custom DataGridView (i.e. I created a class inheriting from the System.Windows.Forms.DataGridView and overrided the ProcessDataGridViewKey(see MSDN) . In this method I check for the CTRL+0 keystroke and some other characteristics . And if everything is true, I programmatically set the value for the currentcell to system.DbNull.Value.

Public Class CustomDataGridView : Inherits System.Windows.Forms.DataGridView


    Protected Overrides Function ProcessDataGridViewKey(ByVal e As System.Windows.Forms.KeyEventArgs) As Boolean

        If (e.KeyCode = Keys.D0 or  e.KeyCode = keys.NumPad0) _
            AndAlso TypeOf MyBase.Columns(MyBase.CurrentCell.ColumnIndex) Is _
              System.Windows.Forms.DataGridViewComboBoxColumn _
            AndAlso MyBase.Columns(MyBase.CurrentCell.ColumnIndex).ValueType IsNot _
              GetType(System.String) Then
            MyBase.CurrentCell.Value = System.DBNull.Value
            Return True
        Else
            Return MyBase.ProcessDataGridViewKey(e)
        End If


    End Function
End Class

And guess what? It works (at least for me)! I also tried it in VS2008. There the same error is thrown ( I gues the bug report got lost somewhere ). But the same trick with ProcessDataGridViewKey works in VS2008.

I haven't tried it yet on other dataTypes like guid's and so. That why i check for datatypes in the code. But for my situation, this solution is OK .

Maybe there is another solution? If you know of one, don't hesitate to let me know...

Meanwhile if you like, you can download my test-program.

Thanks for reading.

Best regards,

Alexander

Design for testability - developer testing context

Hello,

Some more thoughts on design for testatbility in the developer testing context (see previous post ).

Test automation plays an important role in the developer testing context. The advent of the xUnit testing concept , nUnit and the integration of xUnit concept in Visual Studio, has put developer testing more in focus.
The test case is a description that tells us how we can see whether to system has done something as we expected it do that something.
Before executing a test , we must put the put the system in state so our test can actually operate. For example before testing the deletion of an order in our order system, the system must contain an order that we know of. For example a order record in a database.
The act of deleting an order necessitates that we can say to the system that we want to delete that explicit order . So not only to we need control of the initial situation, but we also need control of specifying the input and controlling the place where the order is physically kept (i.e. our test database with Order record with ID ORD1)
After executing the functionality, we must be able to verify if the system has actually done its piece of work. We must “see” the result of its processing. We need to compare this outcome with our initial expectations. A discrepancy can mean either the behaviour of the system was not correct or test case was not correct in terms of initial situation , the actions we took or the expected results.

An automated test (case) in xUnit terms can be viewed as follows (freely adapted on drawings found on http://xunitpatterns.com/ (Gerard Meszaros)).











Automated means that important steps in the execution of a test are preformed “automatically” . In other words exercising a test case on a unit (a method ) involves programmatically

• supplying the initial situation (setup)
• doing the actions (execute)
• comparing the actual results with expected values (assert)
• Cleaning up (teardown) so the next test case can proceed in a clean situation.

So how can we improve the testability in a developer context. On which aspects of the code can we influence the ease of performing developer test and facilitate the localisation of defects.

• Control : In order make automation possible we need to be control of the initial situation as of the all the input that a method needs, so we can steer the processing of the unit in such a way it will produce the outcome we expect.
• Observation : In order to compare actual and expected results , we must be able to observe what the outcome of the processing of a unit is.
• Complexity: Network integration, database integration, Message queuing, Security , Registry, Windows services, COM+ , …. Makes it difficult to set up a test environment.
  Large methods with complex conditional flow necessitates many test case in order to get desired test coverage.
  Inheritance : abstract base class can not be instantiated.
  Large inheritance tree : may test cases to see if combination of base and specialized code works as expected.
• Isolation: Ability to isolate certain dependencies and replacing them with test-doubles improves testability because we control what the test-doubles will do. That way we can concentrate on the logic in the CUT . The DUT will replaced by a test-double. Any calls from the CUT to the DUT (replaced by the test-double) will result in “controlled answers or behaviour”. Hence our test case for the CUT will be easier to setup
• Separation of concerns : The class with a clear responsibility should improve testability in terms of knowing what to test in the first place.
  Smaller things to test
• Heterogenity : Single language or language framework improves testability

If you have remarks or other thoughts, don't hesitate to drop a comment.

Best regards,

Alexander