Wednesday, 13 February 2008

Don’t bother System testers with programming errors.

Hello ,



Suppose you're a team lead on a project and you take a look in your bug tracking tool one fine morning. You see that the tester on the team has filled some bugs.







Here are some details of it:



Bug ID: 6
Short desc: Conversion from string "r" to type 'Double' is not valid.

Last changed by FuncTest
Reported By FuncTest
Reported On 2008-02-13 2:32
Project Petshop Redux
Organization Omega
Category bug
Priority high
Assigned developer
Status new



comment 6 posted by FuncTest on 2008-02-13 2:32

test case # 5487 Enter new Order



Entering the non-number quantity for a certain product resulted in an error after filling out the order form. The error message was : Conversion from string "r" to type 'Double' is not valid.



There is no check for numbers ?




Bug ID: 5
Short desc: Could not find a part of the path 'c:\PetShopRedux\orders.xml'

Last changed by FuncTest
Reported By FuncTest
Reported On 2008-01-31 11:09
Project Petshop Redux
Organization Omega
Category bug
Priority high
Assigned developer
Status new



comment 5 posted by FuncTest on 2008-01-31 11:09

test case reference (#4531) : Export Open orders to xml file.



I wanted to export the open oders. Unfortunately I received the follwing error message Could not find a part of the path 'c:\PetShopRedux\orders.xml'.



When I look at my c drive I can not find a folder named PetShopRedux.



Can you fix this?



Bug ID: 3
Short desc: Index was outside the bounds of the array.

Last changed by FuncTest
Reported By FuncTest
Reported On 2008-01-31 10:38
Project Petshop Redux
Organization Omega
Category bug
Priority high
Assigned developer
Status new



comment 2 posted by FuncTest on 2008-01-31 10:38

test case reference (#5781) : Show overview Orders to be saved.



When i press the nenu to show me the orders that i have entered until a certain point , I get a message saying Index was outside the bounds of the array.



Can you fix this? I need to test this feauture before Wednesday.



Bug ID: 2
Short desc: Object reference not set to an instance of an object.

Last changed by FuncTest
Reported By FuncTest
Reported On 2008-01-31 10:25
Project Petshop Redux
Organization Omega
Category bug
Priority high
Assigned developer
Status new



comment 1 posted by FuncTest on 2008-01-31 10:25

Test case reference (#4578) ; Create Order with no products.



I get this strange message saying "Object reference not set to an instance of an object." But I have not made a reference to an order yet ??



Can you check this? Urgent, is blocking me.


Ok , you say :

"The tester is doing a great job ! I'll quickly instruct the developers to fix
these errors. "

Or .... you could take a step back and analyze the type of bug the tester has found. You shoudl may be take a look at the test cases result of the tester . You might find out that the tester wasn't able to do a lot of testing after all because he was blocked by the bugs...

The Bug list (artifical) shows a high digree of programming errors . There are no real findings of missing functionality, wrongly interpreted specifiaction. In other words the function tester has been able to do his/her thing. The tester got hold up by types of errors that are more technical in nature. Now the functional tester is somehow blocked until the corrected version is released again in QA. Your project manager will not be happy to see his/her people being idle.

It's time to revise your development process , educate your developers and give them the means to conduct testing themselves : enter development testing and test automation.

Maybe your developers will answer :

  • What are you talking about, we already do testing! : Maybe that is true . For example making a Form with 23 buttons and text boxes that test some piece of code they have written. The trouble with this approach is that it a custom implementation. Each developer can do what he likes. It can not be (easily) automated : you must click on the buttons and verify the result in a file or through the textboxes.
  • We don't have time to do testing! : Yes testing is hard and does take time. But finding a bug early on is "cheaper" than when some poor tester finds it after a candidate release . Or even worse when the customer/end-user finds it. It will avoid long cycles of develop-test-correct-retest. When a developer finds a bug , he/she knows the context and can more easily hunt down the bug and eliminate it.
  • It is not our job , we have testers on the team. Why do it twice?: The developer testing is is somewhat different from functional testing (or non-functional testing like performance). In developer testing you primarly want to eliminate the any programming errors. Sure you have some clue how and end-user will use the application, but you're master of the code and therefore have more insight then, anyone where problems lure. So your entry point is the code itself . A functional tester looks at application from the outside (white-box vs. black-box). He doesn't see the code in motion. He only sees the interface of the system. The functional tester will concentrate more end-to-end scenario's that are possible (or that should not be possible) . Also testing throughout the process avoid big bang testing . When you test the smallest parts and continue testing by putting the tested parts together, the chance of (programming) errors coming up later in the process are reduced.

Introducing developer testing and preferably also test automation (for example Visual Stuid Test framwork or NUnit) is not soley a technical matter. You should get the developers to test that their code meets the "their" intentions . There will always be a loss infomation before a letter of code is typed. So system testing remains necessary . Developers should think about what can go wrong with what they're coding, as they're coding it or even before they start coding (TDD). Early detection of errors cost less in the construction phase. System / acceptance test never can reach same coverage of testing. Developers should control quality before shipping next phase (minor for example at check-in or major : release to QA) .

Of course testing alone won't cut it. Testing does not put quality into the code. It just gives insight in the quality of the code (application). You will still need tons of other measures to assure quality code...But that's for another post.

Best regards,

Alexander

No comments: