Pex - test Case 5

Hello,

This post is a continuation in a series of post about Pex, a test case generation tool from Microsoft research. This test scenario is a variation of test case 4. (sorry in advance for the table formatting , I still can not get it right :-(

In the Pex tutorial on p37 you'll see an example of testing a Regular Expression. I didn't find one in the samples though. Too simple maybe?

I'm no hero in regular Expressions. Thankfully there's a library loaded with regular expressions for every kind of application. The following regular expressing is for ensuring strong passwords.

Expression	^(?=.*\d)(?=.*[a-z])(?=.*[A-Z]).{4,8}$
Description	Password matching expression. Password must be at least 4 characters, no more than 8 characters, and must include at least one upper case letter, one lower case letter, and one numeric digit.
Matches	asD1 | asDF1234 | ASPgo123
Non-Matches	asdf | 1234 | ASDF12345

The code example where the regular expression is implemented in

    public class UserAccount
    {

        private string mPassword;

        public string Password
        {
            get { return mPassword; }
            set {
                if (value == null)
                    throw new System.ArgumentNullException("Please specify a valid password. (non-null)");
                else
                   {
                    bool valid = RegularExpressions.Regex.IsMatch(value, @"^(?=.*\d)(?=.*[a-z])(?=.*[A-Z]).{4,8}$");
                    if (valid)
                        mPassword = value; 
                    else
                        throw new System.ArgumentException("Please specify a Strong Password. ");
                    }          
                }
        }
    }

The pex method (keeping in mind the remarks in the comments from the previous test case: the invariant remark)

        [PexMethod()]
        public void PasswordTest( string aPassword)
        {
            UserAccount target = new UserAccount();
            target.Password = aPassword;

            //no problem calling target.password several times, stays the same
            Assert.IsTrue(target.Password != null);
            Assert.IsTrue(target.Password.Length != 0);
            Assert.IsTrue(target.Password.Length >= 4);
            Assert.IsTrue(target.Password.Length <= 8);
            Assert.IsTrue(RegularExpressions.Regex.IsMatch(target.Password, @"[A-Z]"));
            Assert.IsTrue(RegularExpressions.Regex.IsMatch(target.Password, @"[a-z]"));
            Assert.IsTrue(RegularExpressions.Regex.IsMatch(target.Password, @"[0-9]"));
         

            Assert.AreEqual(aPassword, target.Password);
        }

We could do this without Pex of course. We have have to come up with several input values to trigger an exception (at least that is what we intended to program and now are verifying) and some test values for some "positive" tests. The equivalence class technique can surely help us out.

Valid equivalence classes

• bigger or equal than 4 characters


• smaller or equal than 8 characters


• include at least one upper case letter


• include at least one lower case letter


• include at least one numeric digit

Invalid equivalence classes

• not null, non-existing


• smaller than 4 characters


• bigger than 8 characters


• no upper case letter


• no lower case letter


• no numeric digit

You normally start with searching a physical test case (password value) that covers all the valid equivalence classes and for each invalid equivalence class you should also make a test case (i.e. unit test as well). Try not to overlap any invalid equivalance classes so you can be sure that the forced invalidity throwed the exception.

• asDF1234 (covers all valid classes)


• null (non existing)


• Az1 (smaller than 4 chars)


• ASDf12345 (etc)


• abcd1


• ABCD1


• aBcDefGh

So 7 test cases should be enough ?

What would Pex do? It should be as good as this (if not better , at least the tutorial almost implies so).
If you would let the PexMethod work on the password setter method, you would get following results.

Run	aPassword	Summary/Exception	Error Messag
1	null	ArgumentNullException	Value cannot be null. Parameter name: Please specify a valid password. (non-null)
2	""""	ArgumentException	Please specify a Strong Password.
3	\0	ArgumentException	Please specify a Strong Password.
4	\u8000	ArgumentException	Please specify a Strong Password.
6	\0\0	ArgumentException	Please specify a Strong Password.
7	8	ArgumentException	Please specify a Strong Password.
8	\0\n	ArgumentException	Please specify a Strong Password.
10	\0\0\0	ArgumentException	Please specify a Strong Password.
11	\n	ArgumentException	Please specify a Strong Password.
12	\n\0	ArgumentException	Please specify a Strong Password.
16	\0\u8000\u8000	ArgumentException	Please specify a Strong Password.
21	\n\n	ArgumentException	Please specify a Strong Password.
25	\u00062\u00f1\u0016\n\n	ArgumentException	Please specify a Strong Password.
31	\09\u00ef\u0088f\r	ArgumentException	Please specify a Strong Password.
75	\u8000\0\0\0\n\0	ArgumentException	Please specify a Strong Password.
83	0\u0088o]O<
112	\u00062Oo\n\n	ArgumentException	Please specify a Strong Password.
113	\u00062Oo\n
114	\u00062Oo\n\n\n\n\n\n\n\n\n\n	ArgumentException	Please specify a Strong Password.

Pex established 19 test cases. A little bit more if you would go for the manual approach. But are the test input values better? In terms of "readability" no. But Pex generated some input values that I wouldn't have thought of in the first place.

The empty string was added in the Pex generated approach. In our manual exercise with didn't explicitly took that test case because this was implied in the "smaller than 4 chars" equivalence class. But again testing a little bit too much is maybe better (or saver) than testing too little.

Also the Pex generated unit tests include non alfanumeric signs like * or /. Did I intended to do that with my expression? If not I have to re-write it. With my manual technique I "forget" them. So Pex went a little deeper than my test analysis.
What about Escape characters such as "\n" (new line). pex included them and some tests passed. Was this the way I intended it? Were these the specs from the customer?

The examples I used to get acquinted with Pex may seem academic but I believ show potential for a tool like Pex. How to use it in your day-to-day applications will take some time and study. And like the Pex FAQ already states : it is not Pex versus unit testing (or other forms of testing or QA assurance measures like code analysis for example). I hope the Pex team will (continue to) find sponsers within Microsoft to integrate it in the next releases of Visual studio. At least the pex team already took a patent!.

Thanks for reading.

Best regards,

Alexander