Why Automated Testing
Generally, there are numerous well-known benefits that Automated Testing brings to product development. The benefits may be more significant if the product being developed is an embedded system.
From our experience, the most important benefits are:
High reliability and safety
Embedded systems are expected to operate reliably, for the main reasons:
- Embedded systems could be located at places where human intervention is limited or impossible, consequently, it expected the system to operate without failure over a long period.
- Failure may cause expensive downtimes.
- Expensive recalls and/or warranty claims if a product fails to meet quality standards.
- Reputational damage
- Functional safety - in safety-critical domains like medical, industrial, and automotive control systems, reliability can be a matter of life and death.
Reduced development costs
Setting up an automated testing environment requires time and resources, thus initial investment is high, but after that the amount of time needed for testing rapidly decreases, fewer test engineers are needed for the test execution thus eventually resulting in lower costs. For embedded devices, this could be additionally beneficial for the End of Line Testing process. With automated testing, the EOL is much shorter and more reliable.
Shorter Time-to-Market
Constant execution of test cases, Continuous Integration, makes it possible to detect defects in early phase and minimize product integration time.
Modular product design open for customizations
It is common that the market needs a product that can be easily customized and has a lot of varieties. This can be achieved by having a product that is designed to be modular. Combining different modules requires integration testing to make sure that the product as a whole is error-free. Without an automated testing system, it is practically not possible to conduct reliable and efficient integration testing.
What Does Enlight Offer as a Service in This Field
Automated Testing is a field in which we have the know-how needed to successfully set up a project, manage the project, define requirements and test cases, set up technical infrastructure, and implement all parts of an Automated Testing System. Essentially, we can say that the known-how we possess is based on two mutually correlated aspects:
- Procedural
- Technical
The most important part of the Procedural aspect is Requirement Management which is described in detail here.
The technical aspect requires that the Requirements are defined and clarified to the extent needed to be able to set up technical parts of the project. The most important technical parts of the project are:
1. Defining the architecture of the test system.
The typical architecture of the automated system is shown in the figure below. The definition of architecture assumes that we have defined all elements shown in the figure: Testing Framework, CI Server, Test Management System, Issue Tracking System, Testing Equipment, ECU communication protocol, etc.
2. Based on the previous step, setting up entire technical infrastructure.
3. Writing and adding into the Test Management System acceptance criteria and test cases for each requirement.
4. Implementing Hardware Abstraction Layer.
This layer has an important role to decouple test logic from the hardware, e.g. it could be required that ECU supports different industrial communication protocols, regardless of the protocol in use, the test scenarios are the same (the business logic is the same).
5. Implement all Test Cases defined in step 3
6. The important technical aspect is the testing pyramid.
The system shown in the figure is designed to cover all three levels of the pyramid: unit testing, integration testing, and end-to-end testing.
Note: Since, embedded systems are comprised of software and hardware, where hardware may include mechanical parts, setting up an automated testing system for embedded development is regularly more complex in comparison to the pure software systems.
Tools, Frameworks and Protocols
Typically, we use well-known tools such as:
- Jira and Bugzilla - issue management
- TestLink and RTM Jira plugin - test management and requirements management
- Jenkins, TeamCity, Bitbucket, Bamboo, GitLab, Azure DevOps - continuous integration service
- Python unittest, Pytest and Robot Framework – Testing Framework
- CANopen, J1939, Modbus, UART, ECU communication
- CompactRIO, LabJack, PLC and Raspberry Pi etc. – testing equipment
