Thursday, 31 July 2008

Pex - test Case 5


This post is a continuation in a series of post about Pex, a test case generation tool from Microsoft research. This test scenario is a variation of test case 4. (sorry in advance for the table formatting , I still can not get it right :-(

In the Pex tutorial on p37 you'll see an example of testing a Regular Expression. I didn't find one in the samples though. Too simple maybe?

I'm no hero in regular Expressions. Thankfully there's a library loaded with regular expressions for every kind of application. The following regular expressing is for ensuring strong passwords.


Password matching expression. Password must be at least 4 characters, no more than 8 characters, and must include at least one upper case letter, one lower case letter, and one numeric digit.

asD1 | asDF1234 | ASPgo123

asdf | 1234 | ASDF12345

The code example where the regular expression is implemented in

public class UserAccount

private string mPassword;

public string Password
get { return mPassword; }
set {
if (value == null)
throw new System.ArgumentNullException("Please specify a valid password. (non-null)");
bool valid = RegularExpressions.Regex.IsMatch(value, @"^(?=.*\d)(?=.*[a-z])(?=.*[A-Z]).{4,8}$");
if (valid)
mPassword = value;
throw new System.ArgumentException("Please specify a Strong Password. ");

The pex method (keeping in mind the remarks in the comments from the previous test case: the invariant remark)

public void PasswordTest( string aPassword)
UserAccount target = new UserAccount();
target.Password = aPassword;

//no problem calling target.password several times, stays the same
Assert.IsTrue(target.Password != null);
Assert.IsTrue(target.Password.Length != 0);
Assert.IsTrue(target.Password.Length >= 4);
Assert.IsTrue(target.Password.Length <= 8);
Assert.IsTrue(RegularExpressions.Regex.IsMatch(target.Password, @"[A-Z]"));
Assert.IsTrue(RegularExpressions.Regex.IsMatch(target.Password, @"[a-z]"));
Assert.IsTrue(RegularExpressions.Regex.IsMatch(target.Password, @"[0-9]"));

Assert.AreEqual(aPassword, target.Password);

We could do this without Pex of course. We have have to come up with several input values to trigger an exception (at least that is what we intended to program and now are verifying) and some test values for some "positive" tests. The equivalence class technique can surely help us out.

Valid equivalence classes

  • bigger or equal than 4 characters

  • smaller or equal than 8 characters

  • include at least one upper case letter

  • include at least one lower case letter

  • include at least one numeric digit

Invalid equivalence classes

  • not null, non-existing

  • smaller than 4 characters

  • bigger than 8 characters

  • no upper case letter

  • no lower case letter

  • no numeric digit

You normally start with searching a physical test case (password value) that covers all the valid equivalence classes and for each invalid equivalence class you should also make a test case (i.e. unit test as well). Try not to overlap any invalid equivalance classes so you can be sure that the forced invalidity throwed the exception.

  • asDF1234 (covers all valid classes)

  • null (non existing)

  • Az1 (smaller than 4 chars)

  • ASDf12345 (etc)

  • abcd1

  • ABCD1

  • aBcDefGh

So 7 test cases should be enough ?

What would Pex do? It should be as good as this (if not better , at least the tutorial almost implies so).
If you would let the PexMethod work on the password setter method, you would get following results.

RunaPasswordSummary/ExceptionError Messag
1nullArgumentNullExceptionValue cannot be null.
Parameter name: Please specify a valid password. (non-null)
2""""ArgumentExceptionPlease specify a Strong Password.
3\0ArgumentExceptionPlease specify a Strong Password.
4\u8000ArgumentExceptionPlease specify a Strong Password.
6\0\0ArgumentExceptionPlease specify a Strong Password.
78ArgumentExceptionPlease specify a Strong Password.
8\0\nArgumentExceptionPlease specify a Strong Password.
10\0\0\0ArgumentExceptionPlease specify a Strong Password.
11\nArgumentExceptionPlease specify a Strong Password.
12\n\0ArgumentExceptionPlease specify a Strong Password.
16\0\u8000\u8000ArgumentExceptionPlease specify a Strong Password.
21\n\nArgumentExceptionPlease specify a Strong Password.
25\u00062\u00f1\u0016\n\nArgumentExceptionPlease specify a Strong Password.
31\09\u00ef\u0088f\rArgumentExceptionPlease specify a Strong Password.
75\u8000\0\0\0\n\0ArgumentExceptionPlease specify a Strong Password.
112\u00062Oo\n\nArgumentExceptionPlease specify a Strong Password.
114\u00062Oo\n\n\n\n\n\n\n\n\n\nArgumentExceptionPlease specify a Strong Password.

Pex established 19 test cases. A little bit more if you would go for the manual approach. But are the test input values better? In terms of "readability" no. But Pex generated some input values that I wouldn't have thought of in the first place.

The empty string was added in the Pex generated approach. In our manual exercise with didn't explicitly took that test case because this was implied in the "smaller than 4 chars" equivalence class. But again testing a little bit too much is maybe better (or saver) than testing too little.

Also the Pex generated unit tests include non alfanumeric signs like * or /. Did I intended to do that with my expression? If not I have to re-write it. With my manual technique I "forget" them. So Pex went a little deeper than my test analysis.
What about Escape characters such as "\n" (new line). pex included them and some tests passed. Was this the way I intended it? Were these the specs from the customer?

The examples I used to get acquinted with Pex may seem academic but I believ show potential for a tool like Pex. How to use it in your day-to-day applications will take some time and study. And like the Pex FAQ already states : it is not Pex versus unit testing (or other forms of testing or QA assurance measures like code analysis for example). I hope the Pex team will (continue to) find sponsers within Microsoft to integrate it in the next releases of Visual studio. At least the pex team already took a patent!.

Thanks for reading.

Best regards,



Peli said...

Regular expressions are really inefficient for simple string manipulation. This makes the task of Pex harder since we need to explore the regex automaton. Always keep in mind that the more complicated your code is, the harder it is to reason about it.

For example,

Assert.IsTrue(RegularExpressions.Regex.IsMatch(target.Password, @"[A-Z]"));

Could be rewritten as follows:

c => c >= A & c<= Z

In the next episode, you can also try to pass objects (or user types) as parameters.

Garrick Co Ida said...

Hi, Great.. Tutorial is just awesome..It is really helpful for a newbie like me.. I am a regular follower of your blog. Really very informative post you shared here. Kindly keep blogging. If anyone wants to become a .Net developer learn from Dot Net Training in Chennai. or learn thru ASP.NET Essential Training Online . Nowadays Dot Net has tons of job opportunities on various vertical industry.
or Javascript Training in Chennai. Nowadays JavaScript has tons of job opportunities on various vertical industry.