The support of business by IT and applications is preferentially described by business process models as part of a solution blueprint. Typical requirements are the demand for more efficiency, lower production costs and compliance with different policies, standards and regulations. Hence IT applications are increasingly interwoven with business complexity. “How to ensure the level of correctness needed for customized applications to avoid impact on the business due to insufficient quality assurance?” is one of the major questions on this matter.In agile projects the key issue in terms of testing methodologies is indeed the setup of a valid and plausible metrics scheme that permits to measure and compare the quality of a solution continuously over a period of time.
Business process management (BPM) as a methodology is helpful to build a foundation for such metrics. BPM methodology and tools are used to model and describe the solution scope of an application by a top-down decomposition of the business architecture to the service and implementation level. The 360-degree view is achieved by a composition of business procedures to end-to-end processes like procure-to-pay, order-to-cash or make-to-stock in order to design the support of a business process by the application landscape. A bottom-up approach starting at the application level can’t cover possible use cases/business scenarios. Therefore a metric just counting the successfully passed test cases/steps is not appropriate to measure the quality of a software implementation.
The logic of defining a metrics scheme can be taken from software engineering. There the program code describes an application’s logic. In BPM, however, the logic is described by process models. In program code we have textual statements like “if”, “else”, “and”, “case” which are similar to XOR, OR and AND gateway/decision symbols in a process model. In program code we refer to data files and tables for input/output, in process models we have business documents instead. Over many years large organizations like DoD and NASA have developed proven quality assurance (QA) methods and metrics to control the huge amount of mission critical software with respect to their requirements and ensure high quality deliverables with low operation defect rates.
The so-called “statement coverage” concept (see http://en.wikipedia.org/wiki/Statement_coverage) can be borrowed from software engineering and be transferred to business and application test engineering. A statement coverage (expressed as Cx – C0, C1, etc.) defines the test coverage of a process model and therefore describes a valid metric for QA:
- C0 just covers all activities in a model
- C1 covers all branches (transitions between activities)
- C2 covers all logical gateways (covers all paths)
- C3 covers all logical decisions of gateways (e.g. for an AND: first take the left branch, then the right one or vice versa)
A higher level of Cx means a better quality and in return higher efforts for the test plan execution. Running a process model through this algorithm generates test plans (end-to-end process paths with related test cases) as well as the test coverage matrix depending on the Cx level. By directly using the process models created with a BPM tool the metrics can be automatically computed. For a process model of medium complexity with e.g. ten XOR gateways a human simply can’t describe the possible number of test scenarios required for a C2 coverage: ten binary XORs amount 2^10 = 1024 combinations. In model-driven software development this approach is also known as “model-based” testing meaning that a model, its constituents and their behavior are simulated automatically. Because of the model complexity all alternative manual approaches will fail sooner or later.
Also metrics can be defined as the degree of coverage or fulfilment of project-relevant compliance, technical or exceptional issues like
- SOX or risk financial controls
- Organizational and technical interfaces
- Error conditions or exceptions in the process flow
- Business requirements to be covered by user acceptance testing (UAT)
Test coverage metrics calculations based upon process models provide trustable answers on costs, time and quality for your project-, application- and testing-related questions.