Showing posts with label Testing. Show all posts
Showing posts with label Testing. Show all posts

Thursday, 17 July 2008

Design for Testability - Dependencies

Hello,


This post is a continuation on previous posts about design-for-testability (here and here).

Control and observation are important characteristics in test-automation. If our code under test makes an effort to facilitate this , we improve the testability of our code. Improving the testability should lead to find any errors in our code more easily. The more we errors we find early in the development cycle, the cheaper it is to correct them. We early testing and defect correction, we get a good feel on the quality of the software. We can make educated decisions on when to promote the code to QA and/or production. When our customer can use our software to do their job , they create value for the company they work for ….(I will stop rambling here) . But you see , testability (and in a broader sense quality assurance) should be baked into the software. The entire organization can benefit from it.

Let’s start on the “control” aspect of our code under test (CUT) and specifically on the controlling the dependencies of our CUT with other parts of the program or the environment it runs in.

It is in the nature of an object-oriented language to build up a program from lot of different “units” where some “units” or better classes work closely together to make something happen. Object-oriented principles like single-responsibility, information-hiding , inheritance , etc are common in this world. So it is quite natural that classes have dependencies on other classes. The term coupling is often used to state the existence of relationships between class (inheritance, composition , etc). coupling is evitable. If there were no coupling we would end up with monolithically part of codes, actually they would be separate programs. The large classes would be too complex and the cohesion of the operations and member data would very low.

In the realm of testability we sometimes want to be able to test a class in a controlled way, maybe even in total isolation. In order to do that we must be able to control the dependencies. The fewer the dependencies the easier it will be. Hence “managed” coupling can increase testability.

Programs run on machines and make also use of their environment. For example the file system, Message Queue system, database systems , web services, etc. These resource dependencies are expressed in the programs usually via string tokens. For example a folder name, database connection string, queue name, web service URL etc. A good practice of course is to externalize the values for these resources for example in a configuration file. These practice is also beneficial for testability. Avoiding “hard-coded” resource strings in your code is of course not a good programming practice. Because many times the resource string are depended on where the program actually will run. For example development , production or …testing. So controlling the resource tokens outside the program can give us flexibility will setting up tests.
.NET has excellent support to easily consume these externalized resource identifiers in your programs. In the Visual Studio test framework you can easily let the test framework pick up these configuration files and let your code under test use the configured element just like if the code under test was running in their proper host (windows app, web service, etc). In other words , you will copy the configuration info that you need for the code under test from the base app.config or web.config in your test project and modify it as needed.

Let’s focus again on class dependencies. So dependencies are inevitable but we can make them more manageable for testing purposes ( or other reasons). In order to continue our discussion lets introduce some terms. If a class A depends on another class B to fulfil its functionality A is called a client class and B is called a server class. During unit test we want to test a class in isolation. Therefore we want to substitute its server classes by stubs or mock objects . Hard-wired dependencies to server classes hinder this. A dependency on a server class is hard-wired if the server class can not be substituted by some other class (e.g. a subclass of the server class) without changing the source code of the client class. Similar problems arise during integration testing if a server class should be stubbed because it is not test-ready. So how can we manage class dependencies that will benefit testability.
One of the areas lies in the construction of the dependencies. We don’t control them. So our solution strategy should revolve around getting control about the creation of the dependencies. Several techniques can help us to come up with a solution:

  • Interface-based programming : Separating the interface from the implementation is core principle. Our client class uses an abstraction of a server class (interface) , not a particular implementation of it (object). So during testing we could create another implementation of the server class that implements the correct interface. This “test-double” (Gerard Meszaros , xUnit patterns) version of our server class can for example always give a fixed set of values back or just ignores the fact that we ask it to do something. Interfaces are first-class citizen in .NET so you should have no problems with this. But this technique is just a first step toward the solution. We still need to hook up an implementation of the server class into the client class at test time.
  • Service locator: We would delegate the creation of objects to a specialized object that for example would fabricate objects based on a string identifier.
  • Provider model : Variation of Interface-based programming :. Abstract class , concrete implementation hooked up at runtime (configuration file). You can a test-specific implementation of the provider in order to control the tests
  • Manual Dependency injection : We will make another class responsible to establish the link between the client and the server. We will add a parameter of the server type to a method of the client class (a constructor, or a method that requires access to a server instance, or a dedicated setup method) which allows other objects to set the link. During testing the parameter(s) can be used to establish a link to a “test-double” instead.
  • Automatic dependency injection : a DI container framework takes care of the details of which objects are interconnected, so you can build each one independently. No need for passing the dependencies along with the constructor or methods or assigning properties with dependent objects

Still there some concerns (or trade-offs) you should take into account;

  • Information hiding :Why can’t leave it up to A to know its dependencies : Exposing the dependencies in a constructor function can be viewed a violation of encapsulation because now a client of Class A would have to create instances of Class B first before calling the constructor function of class A
  • The ability to substitute a server class is not always important for the “production” implementation. So is testability a good enough reason for implementing this possibility?
    We could make a special constructor that will accept the dependencies. The default constructor would use that hard-wired implementations.
    If there are dependencies on many server classes, this approach would result in too many parameters.
  • DI Framework to the rescue ...but Someone has to know the dependencies
    Assembler , DI Framework via declarations or via config files

There are still other things besides managed coupling you can work on to improve testability , hence the quality of our software : cohesion, encapsulation, ... But that's for another post.

Maybe you have others techiques to improve testability? If you would like to share them, don't hesitate to drop a commont.

Thanks in advance.

Best regards,

Alexander

Wednesday, 9 July 2008

Design for testability - developer testing context

Hello,

Some more thoughts on design for testatbility in the developer testing context (see previous post ).

Test automation plays an important role in the developer testing context. The advent of the xUnit testing concept , nUnit and the integration of xUnit concept in Visual Studio, has put developer testing more in focus.
The test case is a description that tells us how we can see whether to system has done something as we expected it do that something.
Before executing a test , we must put the put the system in state so our test can actually operate. For example before testing the deletion of an order in our order system, the system must contain an order that we know of. For example a order record in a database.
The act of deleting an order necessitates that we can say to the system that we want to delete that explicit order . So not only to we need control of the initial situation, but we also need control of specifying the input and controlling the place where the order is physically kept (i.e. our test database with Order record with ID ORD1)
After executing the functionality, we must be able to verify if the system has actually done its piece of work. We must “see” the result of its processing. We need to compare this outcome with our initial expectations. A discrepancy can mean either the behaviour of the system was not correct or test case was not correct in terms of initial situation , the actions we took or the expected results.

An automated test (case) in xUnit terms can be viewed as follows (freely adapted on drawings found on http://xunitpatterns.com/ (Gerard Meszaros)).











Automated means that important steps in the execution of a test are preformed “automatically” . In other words exercising a test case on a unit (a method ) involves programmatically

  • supplying the initial situation (setup)
  • doing the actions (execute)
  • comparing the actual results with expected values (assert)
  • Cleaning up (teardown) so the next test case can proceed in a clean situation.

So how can we improve the testability in a developer context. On which aspects of the code can we influence the ease of performing developer test and facilitate the localisation of defects.

  • Control : In order make automation possible we need to be control of the initial situation as of the all the input that a method needs, so we can steer the processing of the unit in such a way it will produce the outcome we expect.
  • Observation : In order to compare actual and expected results , we must be able to observe what the outcome of the processing of a unit is.
  • Complexity: Network integration, database integration, Message queuing, Security , Registry, Windows services, COM+ , …. Makes it difficult to set up a test environment.
    Large methods with complex conditional flow necessitates many test case in order to get desired test coverage.
    Inheritance : abstract base class can not be instantiated.
    Large inheritance tree : may test cases to see if combination of base and specialized code works as expected.
  • Isolation: Ability to isolate certain dependencies and replacing them with test-doubles improves testability because we control what the test-doubles will do. That way we can concentrate on the logic in the CUT . The DUT will replaced by a test-double. Any calls from the CUT to the DUT (replaced by the test-double) will result in “controlled answers or behaviour”. Hence our test case for the CUT will be easier to setup
  • Separation of concerns : The class with a clear responsibility should improve testability in terms of knowing what to test in the first place.
    Smaller things to test
  • Heterogenity : Single language or language framework improves testability

If you have remarks or other thoughts, don't hesitate to drop a comment.

Best regards,

Alexander

Friday, 23 May 2008

Design for Testability

Hello,

I have been reading up on "design for testability" because some of the code I deal with is not always easy to test. Some thoughts ....

Context


“Business” wants software application to help them do their job. They use them because they bring “value” to them. In order to make that happen software application must do what they are required to do (fit for purpose) and must be “reliable”.

Through a mixture of process guidance, technology and human skills, to use the previous to elements efficiently and effectively, IT tries to deliver those application on time while adhering to the intrinsic quality attributes like for example reliability. Quality processes are put in place to ensure achievement. In order to assess the if the quality attributes are indeed incorporated into the software product, we need to check it. This is where testing comes in. Testing should give us insight into the quality of a system. This insight should help us decide to release the software for usage by the business because it can bring value to the business. Or it can tell us that there are still some discrepancies and we need to improve quality before handing it over to the business.

Testing is some something we must do although it does not immediately raises the quality of the system. It reports only. Many times “business” can not understand why we can’t deliver something right the first time. Testing costs money! Software creation is still an error-prone endeavour for many reasons despite all advancement in processes , technology and skills of people. In other words we start off with the assumption (certainty!) that software contains errors. Both functional and programming wise. Testing is about finding those. So it is important that we make the best possible test cases to reveal them. Use tooling to support our testing effort etc. Bust beside the test process it self , we can do more.

During the inception and construction of software we can introduce measures to make this testing effort even more efficient and effective. In other words we must make our system as “testable” as possible in order to and to facilitate performing the test and finding the errors.


Testability



Testability is about the ease of performing test against your system and ability of the tests to reveal any defects in a certain context. The context of the test , system testing (functional and non-functional) or developer testing, puts different meanings to testability. Also is it important that other software artefacts beside code, affect testability ; for example the requirement specification documents.

These are some aspects that influence the testability :

  • Understanding: The better you know what the thing is supposed to do , the better you can test for it.
  • Controlling ; You must be able to manipulate the input , in order to verify the correctness of the output.
  • Operational : We can only test it when it works !
  • Visibility : You can only verify the result , if you can see the output.
  • Simplicity : The less there is to test , the more quickly we can test it
  • Stability : The fewer the changes to less test if need to re-do and/or re-write

So manipulating the influencers, can change the testatbility !

It is important to acknowledge that testability does not come on its own. We must take testing into account from the beginning of our software development endeavour. Because we know our software will contain errors (this will only increase with the size and complexity of today’s systems), we must prepare for it to easily test our system and find them before we ship the code.

In other words we must design for testability. It is important because testing costs money. Also faulty software systems can cost the business money (in the best case or worse in case of threat to safety or life) . Therefore we must make the testing as effective and efficient as possible. But Business will benefit from this as well in the long run because we should be able to find defects thus raise the quality of the system, thus raise the value to the business.

Also future changes to the software to incorporate new requirements will benefit from this testability feature because we should be more confident to change working code with regression testing. Also external factors such as SOX compliance make it important for the business to assemble test evidence. Testable systems certainly can aid .

Design For testability



To increase both the ease and value of testing we can tack different areas of the software development process and the software product itself. For example

  • The software development methodology (“testable” requirement concept, etc)
  • The software technology (Procedural vs. Object-Oriented, Mainframe, etc )
  • The software architecture & design (Layered vs. monolithic, Coupling , Encapsulation , etc)
  • The code (Assertions , mocks/stubs , etc )
  • Tool support (Automation , Xunit framework , etx)
  • The developer (Training and education , code reviews , etc)

This broad spectrum calls for an holistic approach of course. The test context will put certain focus in that approach. For example

  • For functional testing you need good test reference in form requirements documents to establish your test cases
  • For System testing , you must be able to let system work under operational condition together with data and sometimes external systems.
  • For developer testing , you must sometimes be able test your unit in isolation and sometimes you would like to integrate a set of unit together.

Currently I'm interested in the developer testing context . So if you have experiences in raising the testability of your code and wish to share them , please do so. I would to hear from you how dependency injection help you ( or not) , how unit testing helped you find defects easily , etc.

Best regards,

Alexander

Friday, 5 October 2007

Testing types classification

There are different types of test you can conduct. These test types have different objectives and characteristics. There are several classification criteria to organize these types of test.




  • Who uses the information of the test results? (development team, end-user, operations , Audit firms, etc)

  • Phase in development cycles (Requirements gathering, Analysis, Design , Construction , etc)

  • Phase in life cycle of a application (New versus maintenance)

  • Who establishes and/or executes the test? (Development team, Test organization, end-user)

  • Dynamic tests versus static tests

  • Functional test versus non-functional test (performance, load, security ,etc)

Sometimes the classification name for type means one thing for one person and another for person. I tried to compile a list of test types from several Internet resources and books. I don't give any formal definition but try to specify the test type through its characteristics.



  • Objective

  • How are the tests executed?

  • Who executes the tests?

  • Who is the main interesseted party in the test-results?

  • Where are the tests executed?

  • When are the tests executed?



Static testing



  • To find errors before they become bugs early in the software development cycle.
  • Code is not being executed.
  • Usually take the form of code reviews (self-review, peer–review, walk-troughs, inspections, audits) BUT can also be conducted on level of requirements, analysis and design.
  • Usage of check-list for verification.
  • Code analysis preferably executed through tools.
  • Executor : Individual developer , Development team, Business analyst / Customer in case of requirements testing ,Business analyst in case of analysis reviews, Architect/Senior designer in case of design review
  • Primary beneficiary is the project team self
  • Can be executed on the private development environment
  • Can be executed in separate environment ( test environment , build environment in Continuous Integration scenario)
  • Code review usually in construction phase; Audits tend to be later in the cycle.
  • Reviews on requirements, analysis and design are executed early in the project


Unit testing



  • A.k.a. Program testing , Developer testing
  • The main objective is to eliminate coding errors
  • Testing the smallest unit in programming environment (class , function , module)
  • Typically the work of one programmer
  • In OO environment usage of Mock Libraries/Stubs to isoloated units
  • Executed by the developer
  • Feedback immediately used by the developer.
  • Test results are not necessarly logged somewhere.
  • Executed on the private development environment. So Tested in isolation of others developers.
  • But can be executed in separate environment ( build environment in Continuous Integration scenario)
  • Executed during construction phase


Unit Integration testing



  • A.k.a. Component testing, Module testing
  • To check if units can work together
  • It is possible for units to function perfectly in isolation but to fail when integrated
  • Typically the work of one programmer
  • Executed by the Developer
  • Test results immediatly used by Developer
  • Test results are not necessarly logged somewhere.
  • Executed on the private development environment. So Tested in isolation of others developers.
  • Can be executed in separate environment ( build environment in Continuous Integration scenario)
  • Exectued during construction phase


System testing



  • Compares the system or program to the original objectives
  • Verification with a written set of measurable objectives
  • Focuses on defects that arise at this highest level of integration.
  • The execution of the software in its final configuration, including integration with other software and hardware systems
  • Includes many types of testing: usually strong distinction between functional and non-functional requirements like usability, security, localization, availability, capacity, performance, backup and recovery, portability
  • Who's testing depends on sub type but usually a separate tester role within the project team
  • Test results are used by the Project team or Operations (performance , volume , -> capacity)
  • Executed on the separate test environment
  • Generally executed towards the end of construction when main functionality is working.


Functional testing



  • Finding discrepancies between the program and its external specification
  • Test result used primarily by the project team
  • focuses on validating the features of an entire function (use case)
  • Main question: Is it performing as our customers would expect?
  • White box approach
  • Single-user test. It does not attempt to emulate simultaneous users
  • Look at the software through the eyes of targeted end-user.
  • Preferable separate tester (non-biased)
  • Separate test environment
  • Generally more towards the end of construction



Beta testing


  • Check how the software affects the business when a real customers use the software
  • "Test results" used by customer and project team
  • Usually not a completely finished product
  • Sometimes used in parallel with previous application on the end-user environment.
  • Is usually not structured testing. No real test cases established. Application is used in everyday scenarios. Although some end-users use implicit error guessing techniques to discover defects.
  • Executed by selected end-users
  • In separate test environment or real production environment
  • It can be conducted after some construction (iterations) but normally before formal acceptance tests.


Acceptance Testing



  • Executed by the Customer or some appointed by the customer (Not the delivering party)
  • Determine if software meets customer requirements and to whether the user accepts (read pays) for the application.
  • Who defines the depth of the acceptance testing?
  • Who creates the test cases?
  • Who actually executes the tests?
  • What are the pass/fail criteria for the acceptance test?
  • When and how is payment arranged?
  • Test results primarily used by customer but are handed over to project team . Could be that application is accepted under some conditions . A.k. known bugs.
  • Separate test environment or Production environment
  • Executed after complete testing by delivering party



Performance testing



  • Searches for bottlenecks that limit the software’s response time and throughput
  • Results primarily used by development team and Operations
  • To identify the degradation in software that is caused by functional bottlenecks and to measure the speed at which the software can execute
  • Kind of system test
  • Mimic real processing patterns , mimic production-like situations
  • To identify performance strengths and weaknesses by gathering precise measurements
  • Don’t intentionally look for functional defects
  • Executed by separate test role
  • Executed on the separate test environment
  • Generally toward the end of construction unless performance-critical components are identified beforehand (calculations, external connectivity through WAN/Internet, etc)


Usability testing



  • Check if human Interface complies with the standards at hand and UI is easy to work with
  • Kind of system test
  • Test results serve the development team and end user
  • Components generally checked include screen layout, screen colours, output formats, input fields, program flow, spellings, Ergonomics , Navigation , and so on
  • Preferable separate tester (non-biased) conducts these test (speciality)
  • Executed on the separate test environment
  • Generally toward the end of construction


System Integration testing



  • To check if several sub-system can work together as specified
  • Kind of system test
  • Development team
  • Larger-scale integration than unit integration
  • Generally combines the work of several programmers.
  • Separate test role with the project team (delivering party)
  • Test result used by project team
  • Executed on the separate test environment
  • During construction when workable sub-systems are ready to be tested (i.e. basic functionality works in happy path)



Security testing



  • Tries to compromise the security mechanisms of an application or system.
  • Kind of system test
  • Need for separate test environment to mimic production environment
  • Separate test role
  • Executed at end of construction unless is primordial in the application



Volume testing



  • Other terms or specialisations Scalability testing, Load testing, Capacity testing
  • To determine whether the application can handle the volume of data specified in its objectives (current + future). So what happens to the system when a certain volume is processed over a longer period of time (resource leaks, etc)
  • Test results important for project team and Operations
  • Volume testing is not the same as stress testing (tries to break the application)
  • Separate test role
  • Executed on the separate test environment
  • Generally towards end of construction unless load/scalability is a critical success factor


Stress testing



  • To find the limits of the software regarding peak data volumes or maximum number of concurrent users
  • Interested parties are : Project team and Operations
  • Create chaos is the key premises.
  • Go beyond potential processing patterns
  • Try to demonstrate that an application does not meet certain criteria such as response time and throughput rates, under certain workloads or configurations
  • Kind of system test
  • Don’t look for functional defects (intentionally)
  • Need for separate test environment to mimic production environment
  • Special test software used to simulate load /concurrent users
  • Separate test role
  • End of construction

Availability testing



  • To assure that our application can continue functioning when certain type of faults occur.
  • Test in function of fail-over technology.
  • Don’t look for functional defects (intentionally)
  • Need for separate test environment to mimic production environment
  • Special test software used to simulate certain fault(network outage, etc)
  • Separate test role
  • End of construction



Deployment testing



  • A.k.a Installation testing
  • To verify installation procedures
  • Important for Release manager and Operations (start up/ business continuity)
  • checking automated installation programs
  • Configuration meta data
  • Installation manual review
  • Conducted by separate test role
  • Separate environment by default (clean machine principle)
  • Construction during stage promotion for example


Regression testing



  • To avoid that software goes into regression in light of changes or bug fixes
  • Regression bugs occur whenever software functionality that previously worked as desired stops working or no longer works in the same way that was previously planned. Typically regression bugs occur as an unintended consequence of program changes
  • To make sure a fix/change correctly resolves the original problem reported by the customer.
  • To ensure that the fix/change does not break something else.
  • On different levels Unit, integration , system, functional
  • Executor depends on level of test
  • Primary interested party depends on level of test
  • Executed during Construction or Maintenance

Several Other Specialised tests


  • Testing for operational monitoring: Do faults we simulated find their way in the monitoring environment of our application?
  • Testing for portability: Does our product work on the targeted Operation Systems?
  • Testing for interoperability;For example does our software work with different targeted database systems?
  • Testing the localized versions.
  • Testing for recoverability after certain system failures.
  • etc.