What is the pyramid of tests?
Testing embedded products can be challenging, but it is essential to ensure the reliability and functionality of these complex systems. To comprehend the hierarchy of tests, we turn to the well-known Pyramid of Tests, which is a framework for organizing testing efforts in an IoT/embedded project. This framework helps organize tests by granularity and level, ensuring a systematic approach to testing.
The pyramid of tests: from unit testing to system testing
1. Unit testing
At the base of the pyramid lies unit testing. This involves the automated validation of specific components, algorithms, or functions within the code. Unit tests evaluate individual parts of the software with various parameters to ensure they function as expected.
In brief:
- New units are tested separately by developers.
- Isolated source code is tested to validate expected behavior.
- These tests are typically done by the developer of the component.
2. Component testing
Moving up the pyramid, we encounter component testing. This level involves testing larger components or subsystems, such as an embedded system driver. Just like unit testing, components are tested separately by developers, usually the ones who developed them.
3. Integration testing
Integration tests examine how different components behave when assembled. This level of testing focuses on the interaction between various layers and services in an embedded solution, such as OTA updates, which require coordination between cloud platforms, firmware uploads, and data reporting.
In brief:
- Component integration tests are made by the developers (or by an integrator tester).
- They help verify that the component works and communicates with those on which it depends or which depend on it.
- If the other components do not exist, it is possible to simulate them.
4. System testing
At the peak of the pyramid is system testing. This level employs end-to-end testing, simulating user interactions with the complete IoT system. System tests validate all business features, providing a holistic view of the system’s functionality.
Made by the testers, the purpose of these tests is to verify that the new functionality as well as the software works as a whole and meets the requirements set (acceptance criteria, user stories, etc).
These tests help detect bugs, reduce risk, validate features, validate patches, and gain confidence in product quality.
In brief:
- These tests ensure that the various systems interface and communicate and that the various software / application elements work together as a global solution.
- They also ensure that the various software and apps integrate correctly with the hardware.
And then what...?
Product validation with user & operational acceptance tests
Device manufacturers need to validate that the technical and functional aspects of the product are fulfilled. Finding numerous bugs in this phase is usually a bad sign for the health of the project. That’s why they usually use user acceptance tests and operational acceptance tests before launching their product on the market.
- User acceptance tests: Validates the functional requirement in an environment as close as possible to reality.
- Operational acceptance tests: These tests validate the technical capacities of the platform (limit, load, backup, restore, maintenance, migration…) compliance with certain legislation/regulations specific to the device manufacturer’s industry.
Maintenance tests
Finally, you’ll probably need to test your product during its whole lifetime to make sure it stays robust.
Maintenance tests are conducted on each update deployed to products already in the field, alongside non-regression tests. While maintenance tests allow to validate the new features and fixes introduced by the update, the non-regression tests verify that all critical elements of the product will continue to function correctly.
Maintenance tests encompass a range of methods, from static testing to dynamic testing, including both white box and black box testing. These tests are also performed before migrating the product to a new environment. This involves testing the backup of data and its loading into the software in the new environment.
Preventive tests can also be conducted, such as following the refactoring of an element to prevent potential failures in the production environment. A comprehensive pass of non-regression tests is carried out, with a particular focus on the elements targeted by the refactoring.
Maintaining old software can be challenging for several reasons: some skills may no longer be available within the company, documentation and tools may be outdated or poorly maintained, and maintainability may not have been considered in the original software architecture.
How to do these tests?
After exploring all the testing options available to ensure the robustness of your product throughout its lifecycle, the next step is to determine how to implement these tests. Check out our article on manual vs automated testing to get a clear understanding of the options available to you.
