Pex - test case 2

Hello,

I'm currently trying out Pex , an automated unit test generator. I'm trying out several scenarios. If you like you can read about a previous test case I tried out. In the same spirit , I'll start of with what you could do without Pex.

The code snippet is as follows. There is a "measurement class" that implements the following method (property).

     private int mTemperature;

        public int Temperature
        {
            get { return mTemperature; }
            set { 
                if (value > -274 & value < 1000)
                {
                     mTemperature = value; 
                }
                else
                {
                    throw new System.ArgumentOutOfRangeException("Please .....");    
                }
               
                }
        }

There are two test situations (different behaviours) in this unit to be tested.

• an integer value is assigned to a field


• an exception is thrown

If you would use the Equivalence class technique to establish a good test suite, you could come up with following findings.
Valid classes

• integer value


• integer value between -274 and 1000

Invalid classes

• NOT an integer value


• integer value smaller or equal than -274


• integer value bigger or equal than 1000

Because the .NET type system gives us a hand (int) , our method only can accept integers. So we only have 1 valid class and two invalid classes or one if you would combine then in one sentence (Not between -274 and 1000). Both approaches should cover our two test situations (assignation and exception).

Now we have to come up with actual values for these classes which shouldn't be too difficult. This would result in two physical test cases (i.e. unit tests). We could pick -30 and 1500 . Or -650 and 0. And so one... All these values should give equal test results while assuring complete coverage.

In case of two invalid classes you will have three test cases :

• valid : 25


• invalid 1 : - 300


• invalid 2 : 1250

A common place for errors are the comparison signs. The Border Value Analysis "technique" stipulates you should pick values around the "borders". In our example these are -274 and 1000. In addition you could also pick values aboove (and/or under) these border values. Any mistake (current or future) should be detected by our set of test cases (i.e. unit tests).

• valid 1: 273


• valid 2: 999


• invalid 1 : - 274


• invalid 2 : 1000

So which unit tests will Pex generate for us?

         [PexMethod()]
        public void TemperatureTest(int aTemperature)
        {
            measurement target = new measurement();
            int actual;
            target.Temperature = aTemperature;
            actual = target.Temperature;
            PexValue.AddForValidation("actual",actual);
        }

Pex, using the specified PexMethod, explored our code. It took him two runs to come up with the necessary values for temperature. On the last value (int.MinValue : the smalles possible 32-bit integer value in .NET ) it discovered an exception. Just like in "traditional" unit testing you can specify an "expected exception" . meaning when this exception is thrown, the test is succesfull because that is what you ....expected.



If you re-run the PexMethod again you will see two classic unit test.

   public partial class measurementTest
    {
        [TestMethod]
        [PexGeneratedBy(typeof(measurementTest))]
        public void TemperatureTestInt32_20080725_080720_000()
        {
            PexValue.Generated.Clear();
            this.TemperatureTest(0);
            PexValue.Generated.Validate("actual", "0");
        }

        [TestMethod]
        [ExpectedException(typeof(ArgumentOutOfRangeException))]
        [PexGeneratedBy(typeof(measurementTest))]
        public void TemperatureTestInt32_20080725_082344_001()
        {
            this.TemperatureTest(int.MinValue);
        }

    }

If you run these two unit tests and enable Code Coverage for the target assembly in VS Test framework, you will notice full coverage as understood by VS test framework.

If you compare the Pex analysis with our "manual" analysis you can see that Pex took the minimal set of test cases to achieve full coverage (as understood by Visual Studio). For example someone would change the comparision operator in the method, the unit tests generated by Pex will not detect it in this particular example.


But if you would slightly change the pexMethod (tip from Peli de Halleux) you would see one extra unit test. You need to re-write the assert to verify the border values.

      [PexMethod()]
        public void TemperatureTest(int aTemperature)
        {
            measurement target = new measurement(); 
            int actual;
            target.Temperature = aTemperature;
            actual = target.Temperature;
            PexAssert.IsTrue((actual > -274 & actual < 1000));
        }

The unit tests generated by Pex would now include the 1000 border value. I don't understand yet why -274 wasn't picked up. I guess it has to do with the fact 0 was the value chosen in the first run. Maybe there are better ways to write this PexMethod? I'll will need to look into the "assumptions" part of Pex (PexAssume class).

    public partial class measurementTest
    {
        [TestMethod]
        [PexGeneratedBy(typeof(measurementTest))]
        public void TemperatureTestInt32_20080725_111609_000()
        {
            this.TemperatureTest(0);
        }

        [TestMethod]
        [ExpectedException(typeof(ArgumentOutOfRangeException))]
        [PexGeneratedBy(typeof(measurementTest))]
        public void TemperatureTestInt32_20080725_111609_001()
        {
            this.TemperatureTest(int.MinValue);
        }

        [TestMethod]
        [ExpectedException(typeof(PexAssertionViolationException))]
        [PexGeneratedBy(typeof(measurementTest))]
        public void TemperatureTestInt32_20080725_111953_002()
        {
            this.TemperatureTest(1000);
        }

    }

Is it worthwile to look for test cases but beyond absolute minimal? Well, it dependends. One of the advantages of of having (automated) unit tests is the ability to use them over and over again. In the context of regression testing these test suitses can be of great help to assure that all code that is using your unit (i.e. method) will still function after you made modifications. or least you will know in which situation something will break so you can take appropriate measures (defaults, overloads, release notes ,...).

         private int mTemperature;

        public int Temperature
        {
            get { return mTemperature; }
            set { 
                if (value > -274 & value <= 1000)
                {
                     mTemperature = value; 
                }
                else
                {
                    throw new System.ArgumentOutOfRangeException("Please ...");    
                }
               
                }
        }                             
    

So if regression testing is of high priority , it think some methods need those extra test cases. They can happily live beside the ones generated by Pex. It is sometimes (value of software; money, safety) better to have "too much" than "too little" test cases I think (even if it means extra (test) code to maintain.

As shown in the example, the way you setup your Pexmethod will influence how Pex "explores" your code. Although the the call to the (setter) method was the same. The Assertion part of the Pexmethod made Pex do an extra effort.

If you have other thoughts or remarks about pex or devloper testing in general, don't hesitate to drop a comment.

Thanks for reading.


Best regards,


Alexander