The Gartner Group's annual "Hype study" showed once again that the much discussed model-based testing (MBT) is still far from a practice-relevant use.
In this article we want to examine the result of this study more differentiatedly since as so often much is only a matter of definition.
If you would like to know which options RapidRep offers you in this area, please click here: model-based testing with RapidRep.
First we want to clarify what we mean by model-based testing. The topic model-based testing has many facets and is still struggling for a universally accepted definition.
Wikipedia and other sources of definitions focus on the early stages in the test:
"Model-based testing (MBT) is a generic term for the use of models for the
- automation of test activities
- generation of test artefacts in the test process.
This includes especially the generation of test cases from models (e.g. by using the UML) that describe the target behavior of the system to be tested."
Source: German Wikipedia (transl.)
From our point of view, the generation of test artefacts is only a partial aspect of MBT. The mechanical creation of test cases and/or test data from a business model contributes to a more efficient automation in the test preparation.
The greatest benefit of MBT is the option to be able to highly automate the always repetitive test execution and test evaluation.
The following graphic is very well suited to represent and explain the embedding of MBT in the development process of software. It shows a form, modified for technical systems, of the so called scientific knowledge process, which has proven itself in the empiric sciences for centuries. The scientific knowledge process presupposes a real system, however (e.g. planetary orbits, atomic nuclei etc.). Hence, the illustration describes an analogue approach for systems that have still to be developed and built. Such systems can be for example an elevator in a high-rise or a software program.
Requirements
All features that the technical system shall have in the end have to be provided as precisely as possible in the beginning. The Requirements Engineering collects and specifies all functional and non-functional requirements.
Construction plan
The construction plan has to be so complete that the technical system can be precisely built from this information.
Technical system
The system built gets usually integrated into a surrounding system. As soon as the built system is executable within a system environment, experiments can be performed at this system. The system gets stimulated via impulses and reacts accordingly.
System data
By experiments at the technial systems, input/output correlations can be gained. Subsequently, the system data of a technical system are used to compare the system behaviour with the requirements posed.
If the system behaviour determined by means of the system data does not meet the requirements, adjustments to the construction plan or to constructing the technical system have to be made.
These iterative changes to the technical system and the repeated testing of the system are very cost-intensive and delay the release.
In contrast, model-based testing is a clearly more advantageous approach!
The MBT is highlighted green in the diagram.
Abstract model
Beginning with the construction plan, the approach is based on an abstract model. Like any other model, this model is created by abstraction and idealisation. The abstract model is a simplified construction plan.
Simulation model (Reference implementation)
The abstract model serves as a template for the creation of a simulation model. The term simulation model is derived from the scientific knowledge process and should rather be called reference implementation in the context of MBT. The verification ensures whether the simulation model has been formed defectless from the abstract model.
Model data
The reference implementation is an executable simulation model. The model's behaviour is derived from response behavior gained by experiments.
The validation of a model can only be done by comparing system data and model data. As far as the abstract model consists of mathematical formulas or equations, deduction enables a direct analytical determination of model data (rather rarely in practice).
Fields of application
Scenario A: the technical system has not been built yet
The technical system is in planning but not yet existent. As a consequence, there are no system data available yet either.
A specialist must compare the model data to the requirements and make them plausible. If the model data shows that not all requirements are met, changes to the construction plan or the abstract model have to be made.
As soon as a specialst confirms the model's correctness, creating the technical system can begin.
As soon as the technical system is operational, scenario B for the test automation gets used in addition.
Scenario B: the technical system already exists
The technical system exists and is available for experiments to determine system data.
The model data from the simulation model present the target behaviour of the technical system. The same experiments (= same stimulation) can be executed parallel on the technical system and the simulation model. For each executed experiment (=test) system data and model data must match, unless the technical system is wrong or incomplete.
Since the abstract model is a simplification through idealisation and abstraction, tolerable deviations between system and model data within a certain intervall have to be considered.
Conclusions
The modelling of technical systems represented in the diagram is very well applicable to the IT sector.
The principal of a software provides the specific, functional requirements in form of a performance description or a functional specification document. From this, software engineers deduce a DV-concept and give it to the developers for its implementation. The principal accepts the program once the test meets the specified requirements.
The software development process in its current form describes the process to develop software from the requirements up to acceptance testing. No matter wheter agile software methods or V-models are used: errors cost money, delay the employment of programs and the software test is difficult to automate without MBT and costly.
Benefits through MBT
- MBT detects mistakes in the construction plan before the technical implementation has yet begun (scenario A).
- The reference solution can be adjusted much easier and communication with the business department is easier done with models than with showing lines of program in a programming language.
- The reference solution can be very efficiently validated by a specialist. The test data lead fast and directly to prototypical results. Test data can be generated from the model or alternatively be used from the systematic test case determination.
- The implementation of the software on base of a validated construction plan leads inevitably to less bugs in the software to be created.
- Once the software is in use, it can be tested highly automatically. Thereby the reference implementation acts as absolutely reliable test oracle in test execution and test evaluation.
- A high software quality is permanently ensured with minimal effort.
Bottom Line
Model-based testing will substantially influence the software development process because the resulting benefits are immense.
FINARIS has very successfully employed MBT in two projects with the help of the RapidRep Test Suite. Hence, from our point of view MBT has long left the "hype state".