What is DevSecOps?
DevSecOps is an evolution of DevOps, a practice that merges software development (Dev) and operations (Ops).
The goal of DevOps is to shorten the systems development life cycle while continuously delivering high-quality software. It focuses on collaboration, continuous integration, continuous delivery, and agility throughout the design, development, and deployment stages.
DevSecOps builds on this by integrating security practices into the DevOps “way or working”. The “Sec” in DevSecOps stands for security, adding essential security measures to the continuous practices of DevOps. This integration ensures that security is a shared responsibility throughout the entire lifecycle of application development and deployment.
The rise of DevSecOps for continuous compliance
Today, complying with cybersecurity regulations like the Cyber Resilience Act (CRA) is synonymous with implementing DevSecOps best practices and tools for developing, deploying, and continuously updating products until the end of their life.
Continuous compliance indeed involves constant monitoring to verify, audit, and respond to incidents, thereby reducing risks associated with non-compliance. It also involves risk management, starting from the inception of a new project, for maintaining operations and protecting assets.
Risk analysis: the cornerstone of DevSecOps practices
Risk analysis involves identifying potential attack vectors in embedded systems. This security assessment is typically conducted at the outset and throughout the product lifecycle to ensure the device remains secure over its lifespan.
Security risk analysis is a fundamental activity of the DevSecOps methodology, as it enables:
- Asset identification and classification
- Impact rating (from negligible risk to severe)
- Threat modeling
- Attack path and feasibility assessment
- Risk mitigation decisions
Thanks to this activity device manufacturers can continuously mitigate security risks and protect their reputation in the event of vulnerabilities.
Note that it’s one of the requirements of the Cyber Resilience Act. Read our article on security risks assessment to know more.
Now, let’s delve into some of the key steps of device development and operational lifecycle management to understand how to integrate DevSecOps practices, starting with the build phase of the embedded system.
How to integrate DevSecOps best practices in product development & maintenance activities?
1 – Build management
In DevOps, continuous integration and build management are essential practices. This involves continuously compiling your product in a highly automated manner, ensuring that you can systematically reproduce any version of the software that will be deployed. Quality checks are also an essential part of the process, using tools to validate that the build is clean and reliable.
DevSecOps methodology adds an extra layer of caution to this process, particularly in managing dependencies. When integrating external dependencies into the embedded system, it is important to ensure they are reliable, secure, and well-maintained. Tools can help verify the quality and maintenance status of these dependencies, ensuring they meet the necessary standards.
2 – Secret management
Cybersecurity is closely tied to system risk analysis. If there is a risk that your equipment could be compromised physically, locally, or remotely, you need to implement remediation measures. These could include different security systems, certificate-based authentication systems, physical locks, or auto-locking mechanisms to protect against physical tampering.
Managing configuration and secrets is a critical aspect of the DevSecOps approach. This involves technical implementation through software code relying on the appropriate hardware component. Both hardware and software must be integrated to work conjointly to ensure a high level of cybersecurity.
Differentiating secrets
Managing secrets is a complex task. We can differentiate between secrets outside the device and those within it.
Secrets outside the device can vary in nature, such as server certificates for production, private keys for binary signatures in manufacturing, or certificate authorities for device provisioning. Each type of secret has a different level of criticality. In both cases, you will need to manage:
- a development environment, which is less critical but should be easy for the team to use,
- and a production environment, which is extremely critical and must be well protected, even from the internal team.
Secrets within the device also vary in nature, ranging from provisioning keys to user passwords for your application.
Some methods allow you to manage both internal and external device secrets. The next paragraph will explain how to store them.
Secret storage
The storage of secrets depends on their type and sensitivity. They need to be stored in zones with different access controls. For instance, a secret might be stored in a hardware-based security component, such as a Hardware Security Module (HSM) on the server side, or a Trusted Platform Module (TPM) or Secure Element (SE) on the embedded side.
Managing a secret is not only a matter of hardware but also a question of process. For example, a highly secure provisioning process should preferably be managed with a key pair generated in a secure element on the embedded system, used to generate a Certificate Signing Request (CSR) signed by your Public Key Infrastructure (PKI), rather than a basic symmetric key copied into the device.
In our Welma Linux distribution, we offer a Trust Zone-based implementation of secure storage, relying on the internal security component of the processor to ensure privacy. For more details, please refer to our documentation here.
Enrollment and commissioning
The enrollment process involves integrating a device into a family of other devices, assigning it a unique identifier and a unique “key” for secure communication with the infrastructure.
Commissioning entails generating credentials that enable a device to connect to a network, whether local or global.
The lifecycle of secrets varies depending on the product’s stage, and there should be no overlap between secrets used during development and those used in manufacturing.
Root of trust
The root of trust is essential for ensuring software authenticity. Preloaded secrets must be inaccessible and immutable, typically using One-Time Programmable (OTP) keys offered by the microprocessor itself and programmed into the device during the manufacturing process. This relies on cryptographic algorithms and secure storage solutions that isolate secrets from the rest of the system, along with a unique implementation in the MPU ROM Code.
The next steps in the chain of trust are implemented with dedicated software: the bootloader verifies the authenticity of the kernel, and the kernel authenticates the filesystem.
This implementation must be done carefully because a single weak point in the chain can compromise the entire system.
That’s why we offer a full out-of-the-box implementation of root of trust for embedded systems in our Linux distribution.
3 – Testing
Once your system is coded and compiled, it is crucial to test it thoroughly. DevOps practices request continuous testing, but what exactly does that entail? There are various levels of testing that can be activated and automated.
Levels of testing
The most basic level is static code analysis, which helps ensure your code is clean and adheres to development standards.
Next, there are unit tests, which verify the proper functioning of specific code segments and algorithms. These tests typically run in a continuous integration environment or a simulated environment.
Finally, there are on-board tests, which can be automated (including component, integration, or system tests) and provide daily or weekly reports on the results of your automated test suite.
Consult our comprehensive guide on embedded testing here.
Security testing
But what does DevSecOps add to this? It involves pushing your automated tests to consider additional cybersecurity aspects. There are two main types of security testing:
- Static Application Security Testing (SAST): SAST involves static code analysis with a focus on cybersecurity, to identify potential risks such as buffer overflows that could be exploited.
- Dynamic Application Security Testing (DAST): DAST, on the other hand, is less common and more challenging to implement. It is akin to automated on-board testing but focuses on security. This includes automated security test suites and fuzz testing to check how protocols behave when subjected to various data types.
4 – Manufacturing & update deployment
After testing, the next challenge is deployment. Whether it is deploying during manufacturing or through updates, the goal remains the same. DevOps practices emphasize continuous deployment of artifacts, enabling seamless manufacturing programming.
But it is also crucial to think about post-manufacturing updates and management. Implementing a robust update system and ensuring your compilation platform can generate update artifacts is a fundamental best practice.
DevSecOps adds critical security features to this process. Implementing a secure boot is a way to enhance security within your system as well as secure remote communication which guarantees that updates are deployed through secure channels. At a minimum, it ensures that update packages are signed and verified. These measures should be tailored to your specific risk analysis.
5 – Vulnerability identification & patching
One of the main challenges in system maintenance, particularly for Linux platforms, is identifying and deploying vulnerability patches. This involves addressing both software and hardware vulnerabilities.
Silicon vendors like STMicroelectronics have developed platforms that catalog vulnerabilities, providing accessible documents that describe each reported vulnerability. These platforms follow a specific process, including an embargo period that allows for assessing the vulnerabilities, understanding their impacts, and protecting clients during this time. This process ensures that clients are notified of potential vulnerabilities and have time to implement the necessary patches. It is a good idea to check with your specific vendor to see if they have implemented a similar system.
For all standard software, vulnerabilities can be identified with the appropriate tools that use reference vulnerability databases like the NVD from the NIST organization.
For instance, The Embedded Kit provides a vulnerability management tool called CVE Scan for embedded systems, which includes both detection and management of these vulnerabilities.
Implementing a routine for managing vulnerabilities
Establishing a maintenance routine is an effective way to manage vulnerabilities. Here is a basic routine to follow:
- Identify components: Start by identifying all software and hardware components of a product.
- Scan for vulnerabilities: Next, scan for potential vulnerabilities in these components, primarily focusing on software, using dedicated SCA tools.
- Manual analysis: Follow this with a manual analysis to evaluate the risk associated with each vulnerability.
- Apply patches: Apply patches and regularly update your applications. Staying up to date with the latest Long-Term Support (LTS) versions ensures that all corrective and vulnerability patches are applied.
- Continuous testing: The ultimate step is testing, which leads to delivery. While you may not deliver with every iteration, continuous testing is essential. This phase involves a lot of automation, allowing for a smooth and continuous flow. Adhering to DevOps and DevSecOps practices simplifies the process and provides guarantees without significant manual effort.
Watch this webinar on product long-term maintenance hold with STMicroelectronics.
6 – Device lifecycle management
Device lifecycle management is an essential part of the DevOps cycle. It involves understanding what happens to your products once they are deployed in the field. What state are your software and hardware in? Are there any issues? This continuous improvement mindset is crucial for maintaining the quality and functionality of your products.
DevSecOps enhancements: certificates
DevSecOps adds an extra layer to this process by focusing on the management and renewal of certificates. These certificates are often subject to critical deadlines and must be effectively managed using temporary keys. Integrating these steps into the lifecycle management process is vital. Key stages include provisioning and decommissioning.
Provisioning and decommissioning
Provisioning involves preparing a product for deployment, while decommissioning is about safely removing a product from its lifecycle. A critical aspect of decommissioning is ensuring that the product’s identity cannot be reused maliciously. This involves erasing credentials from devices that are no longer in use and removing access authorizations from platforms and connectivity systems. Neglecting these steps can lead to identity theft and other security issues.
DevSecOps tools
In DevSecOps, tools play a crucial role, integrating both software layers and hardware. Having the right tools for continuous integration, vulnerability management, and security is essential. These tools help you save time, reduce costs, and deliver a higher quality product more quickly.
At The Embedded Kit, we offer the tools necessary to implement these best practices on the software side of your device. Our offerings include:
- Linux distribution: A robust platform for development and deployment.
- Vulnerability management tool: Facilitates the detection and remediation of vulnerabilities in embedded software ecosystems. If you want to learn more about how it works, please visit our dedicated page.
- Device management platform: Helps manage the lifecycle of your devices, ensuring they remain secure and functional.
- Automated testing tool: Supports continuous testing, which is essential for maintaining high quality and security standards.
The concept of “continuous” is central to DevOps, where continuous improvement and ongoing work are fundamental principles. This approach encourages us to constantly reassess and enhance our ways of working.
With new cybersecurity regulations like the Cyber Resilience Act (CRA), the integration of “Sec” into these practices has paved the way for DevSecOps. This shift introduces several important concepts, emphasizing a change in mindset, increased transparency, and adherence to additional best practices focused on the security aspects of engineering teams’ development activities.
Ultimately, implementing the right routines and tools will simplify the continuous activities of DevSecOps, ensuring a more secure and efficient development process.
