Monday, 11 August 2008

Pex - test case 6 (revisited)

Hello,

In the previous case I tried to come up with a scenario where a method takes a "complex" type instead of primitive types like integers of booleans. The exercise didn't go as expected. By the way , the latest downloadable version of Pex is 0.6.30728.0

To see whether pex still works on a method with primitive types, I re-wrote the method as follows.


public PensionContribution DeterminePensionContribution2(
uint aSalary,
uint aAge,
uint aServiceYears,
Contract aTypeContract)
{
//no pre-codition checks ....

PensionContribution contribution = new PensionContribution();

if (aSalary <= 3000)
{
if (aAge < 45)
{
if (aTypeContract == Contract.FullTime)
{
contribution.BaseContribution = 80;
contribution.SupplementaryContribution = 15;
}
else
{
contribution.BaseContribution = 80;
contribution.SupplementaryContribution = 0;
}
}
else
{
if (aServiceYears < 5)
{
contribution.BaseContribution = 80;
contribution.SupplementaryContribution = 15;
}
else
{
contribution.BaseContribution = 80;
contribution.SupplementaryContribution = 25;
}
}
}
else
{
if (aAge < 45)
{
if (aServiceYears < 5)
{
contribution.BaseContribution = 135;
contribution.SupplementaryContribution = 30;
}
else
{
if (aTypeContract == Contract.FullTime)
{
contribution.BaseContribution = 135;
contribution.SupplementaryContribution = 25;
}
else
{
contribution.BaseContribution = 135;
contribution.SupplementaryContribution = 20;
}
}
}
else
{
if (aServiceYears < 5)
{
contribution.BaseContribution = 135;
contribution.SupplementaryContribution = 30;
}
else
{
contribution.BaseContribution = 135;
contribution.SupplementaryContribution = 20;

}
}

}
return contribution;
}



The PUT method looks like this.



[PexMethod]
public void PUTDeterminePensionContribution(
[PexAssumeUnderTest]EmployeeServices target,
uint aSalary,
uint aAge,
uint aServiceYears,
Contract aTypeContract
)
{
PexAssume.IsTrue(aAge > 18 && aAge < 75);
PexAssume.IsTrue(aServiceYears > 0 && aServiceYears < 50);
PexAssume.IsTrue(aSalary > 1400 && aSalary < 10000);
PexAssume.IsTrue(aTypeContract == Contract.FullTime || aTypeContract == Contract.PartTime);

PensionContribution result = target.DeterminePensionContribution
(aSalary, aAge, aServiceYears, aTypeContract);
PexValue.AddForValidation("BaseContribution", result.BaseContribution);
PexValue.AddForValidation("SupplementaryContribution", result.SupplementaryContribution);

if (aSalary <= 3000)
Assert.IsTrue(result.BaseContribution == 80);
if (aSalary > 3000)
Assert.IsTrue(result.BaseContribution == 135);
if (aSalary <= 3000 && aAge >= 45 && aServiceYears < 5)
Assert.IsTrue(result.SupplementaryContribution == 15);
if (aSalary <= 3000 && aAge < 45 && aTypeContract == Contract.FullTime)
Assert.IsTrue(result.SupplementaryContribution == 15);
if (aSalary <= 3000 && aAge >= 45 && aServiceYears >= 5)
Assert.IsTrue(result.SupplementaryContribution == 25);
if (aSalary > 3000 && aAge < 45 && aServiceYears >= 5 && aTypeContract == Contract.FullTime)
Assert.IsTrue(result.SupplementaryContribution == 25);
if (aSalary > 3000 && aAge < 45 && aServiceYears >= 5 && aTypeContract == Contract.PartTime)
Assert.IsTrue(result.SupplementaryContribution == 20);
if (aSalary > 3000 && aAge >= 45 && aServiceYears >= 5)
Assert.IsTrue(result.SupplementaryContribution == 20);
if (aSalary > 3000 && aServiceYears < 5)
Assert.IsTrue(result.SupplementaryContribution == 30);
}


In no time Pex finds the test cases for full block coverage











targetaSalaryaAgeaServiceYearsaTypeContractBaseContributionSupplementaryContributionSummary/ExceptionError Message
1...8192642FullTime13530
2...81926430FullTime13520
3...8192312FullTime13530
4...81923130FullTime13525
5...1532642FullTime8015
6...15326430FullTime8025
7...1532312FullTime8015
8...8192642PartTime13530
9...1532312PartTime800
10...81923130PartTime13520



So there is nothing "wrong" with the method under test. The problem when you investigate the output window, lies maybe in the calculation of the "age" and "serviceYears" properties.


00:00:00.0> starting execution
00:00:00.0> reflecting tests
00:00:00.3> TestProject1
00:00:00.3> EmployeeTest
00:00:00.3> DeterminePensionContributionIntern(Employee)
maxconstraintsolvertime - 1s (constraint solver time out after 1 seconds)
[execution] 4 runs
[execution] 15 runs
[test] (run 20) DeterminePensionContribution
[test] (run 21) DeterminePensionContribution
[execution] 21 runs, 49/81 blocks covered
[test] (run 25) DeterminePensionContribution
[execution] 29 runs, 49/81 blocks covered
[execution] 41 runs, 49/81 blocks covered
[modelsearch] could not flip 2 branches in a row
[execution] 50 runs, 49/81 blocks covered
[execution] imprecision at testingPEX.Employee.differenceInYears, offset 0x26
[modelsearch] could not flip 4 branches in a row
[execution] 77 runs (97,40% unique paths), 49/81 blocks covered
timeout - 120s
Timeout, 1 times
MaxConstraintSolverTime, 60 times
[coverage] 49/81 block (60,49%)

00:02:01.4> [finished] 3 generated tests (0 failures), 00:02:01.3757768.
-- 0 critical errors, 0 errors, 0 warnings.

[reports] skipping report generation

EXPLORATION SUCCESS


monitored process exited with Success (0)
finished



Pex needs to find dateTimes in order to test the conditions specified in the method. I used a helper method to calculate the years to date for the age and serviceyears:

private uint differenceInYears(System.DateTime aDate1, System.DateTime aDate2)
{
//missing checks ....
//approximative
uint years = (uint)(aDate1.Year - aDate2.Year);
if (aDate1.Month == aDate2.Month)
{
if (aDate2.Day < aDate1.Day) years = years - 1;
}
return years;
}



Could it be the unsigned integer type declarations? Or are the assumptions wrongly formulated? Or both? What does "could not flip 2 branches in a row" mean?

I'll try to do some more test around this. Feel free to comment!

Thanks for reading.

Best regards,

Alexander

8 comments:

Gerri said...

Hi! Thanks so much for these PEX articles! Our team has just started studying Pex yesterday. These are very helpful articles. Thank you.

I have a newbie question though. We'd like to implement Pex in our Data access layer (strongly typed datasets). What test could i possibly generate from a simple typed dataset that contains a tableadapter with a simple select query based on a single parameter?
"select id, name, score from tblScores where score > 50"

THANK YOU.

anowak said...

Hello gerri,

Thanks for your interest!

Regarding your question, I would to note down some "concerns".
1) Pex can not dive into the SQL engine. For pex, the integer parameter that would hold 50, is just an integer. Pex isn't able to see what the conditions would be (symbolic analysis of the code). Maybe Pex Assumptions would help?
2) In theory this would not be unit testing anymore. The dependency of the code-under-test (i.e. database) makes the setup, assertion and cleanup of a unit test harder. You need a table with different scores. This must stay fixed during the test run. You must know many records there are for each score for the asserts after the method execution. if you have update/delete/insert logic to test, setting the databse table to "known" state becomes even harder as well as the assertions (you will have to go to the database again to see whether the Update/Delete/Insert was like expected.
3) Is it worthwile to test generated code?
4) To test pure SQL statements/stored procedures, you could take a look at Visual Studio Database Edition which has something called "database unit test".

Hope this helps (a little).

Best regards,

Alexander

Gerri said...

Hi thanks so much! Ok I understand PEx a little more now because of you. Thank you.

I'm doing a little test though and everytime I "Run Pex Exploration",
these attributes are decorated on every unit test generated:

[TestMethod]
[Ignore]
[Description("the test state was: path bounds exceeded")]

so i can't proceed with any test. I just ran my program on vista. I'm not sure if im making any sense though.

Thank you.

Anonymous said...

This is the code that I have (just for testing)

[PexMethod]
public void InsertAnyName(string name, int status)
{
InsertNameTableAdapter in = new InsertNameTableAdapter();
int expected = 1;
int actual;
actual = in.InsertAnyName(name, status);
PexAssert.AreEqual(expected, actual);
}

I'm simply referencing a dll that contains .xsd connecting to my DB, to my Test project. But when I "run pex exploration", attributes:

[Ignore]
[Description("the test state was: path bounds exceeded")]

are declared on top of generated unit test methods, thus returning "path bounds exceeded"
=(

Thank you. Please tell me where I'm wrong.

Anonymous said...

Oh, I forgot to include in InsertAnyName():

PexAssume.IsTrue(status == 1)

at the very start. Thus, if 1 is used as the argument, the error message is returned. Also, I executed the very same code inside a [TestMethod()] method, and supplying "anonymous" and 1 as parameters, it was successful.

THANK YOU VERY MUCH FOR THESE ARTICLES.

Anonymous said...

hey,

how do i know how much maxbranches should i declare?

i added this with that simple line of code and it worked!
[PexMethod(MaxBranches = 100000)]

THanks!

anowak said...

Hello,

Pex makes "suggestions" to augment the MaxBranches. You can see these in the "suggestion window". In theory you could set it to Int.MaxValue. (see tutorial chapter 7.10). This gives Pex some more time to explore. But bootom line is that this is more a "trail and error" thing.

But I think here in lies a danger. What is Pex going to explore? Pex will pretty soon end up in the deep caves of System.Data(.SQLClient).The adpater will be constructed. The adpater will make a connection, paramater object are filled in, ExecuteNonQuery, etc
...

The InsertAnyName method is pretty thin I suspect. You don't put much logic in it. This TableAdapter convience method is maybe not very well suited for Pex and you will be better off with traditional unit tests.

There are some attrubutes you might try out to Suppress Instrumentation of types in the System.Data.(SQLClient) realm.
http://research.microsoft.com/pex/wiki/book.html#PexSuppressUninstrumentedMethodFromAssemblyAttribute


Best regards,

Alexander

Nikolai Tillmann said...

> what "does "could not flip 2 branches in a row" mean?

That is just an internal log message.

It means that Pex tried to solve 2 constraint systems, but they didn't have a solution. (Or in other words, Pex was trying to figure out if particular execution paths in the program are feasible, but the answer was no, twice in a row.)

Since solving constraint systems can take a long time, the purpose of this kind of message is really just to give an indication that Pex is still alive and doing something.

I realize that the message's wording is obscure; I will change the wording.