In the early days of embedded development, it was common practice to ship the exact same software image used during development straight into production. This meant that tools, debug utilities, default passwords, and even development-only packages made their way into the final product.
Unsurprisingly, this approach often led to last-minute scrambles—removing tools, changing credentials, and revalidating everything—just to make the system production-ready. It was inefficient, error-prone, and risky.
Today, we know better. Modern embedded development, especially with Yocto on ARM-based Linux systems, embraces a more structured approach: dual image management. This strategy separates development and production environments from the very beginning, ensuring both flexibility for developers and security for end users.
Why dual images?
1. Developers need tools and privileges
During development, engineers require full access to the system. This includes:
- SSH access and login credentials
- Debugging and tracing tools like strace and tcpdump to investigate what is happening in the system
- Interactive programs like top or killall, as well as text editors
- Signing keys for testing authenticated updates
- The ability to simulate edge cases (e.g., killing processes, removing profile permissions, filling up filesystems…)
In graphical environments using Wayland/Weston, developers might also need to run the system with debug flags or capture screenshots—capabilities that should never be present in a production image.
This is where the development image (dev image) comes in. It’s tailored for experimentation, diagnostics, and iteration.
2. Production systems require lean, secure software
In contrast, production systems must be:
- Minimal and secure
- Free of unnecessary tools and debug utilities
- Compliant with licensing constraints (e.g., avoiding GPLv3 components that conflict with secure boot policies)
- Configured with controlled access and hardened settings
This is the role of the production image (prod image) – a clean, optimized, and secure version of the system ready for deployment in the field.
Check our article on development vs production images in embedded systems
The Yocto workflow for dual images
Here’s how this dual-image setup typically works in a Yocto-based project:
1. Two recipes, one foundation
The development team writes two image recipes:
- <image>.bb: the base production image
- <image>-dev.bb: which includes prod-image.bb and adds development-specific packages
This ensures minimal divergence between the two images, reducing maintenance overhead and avoiding inconsistencies.
2. Two output images
These recipes generate two .wic images – Yocto’s standard format for bootable media. Each .wic file includes:
- Partition tables
- Bootloader
- Kernel
- Root filesystem
These images can be flashed directly to USB drives, SD cards, or eMMC storage.
3. Development and testing flow
Developers primarily work with the dev image but also test the prod image to ensure it behaves as expected. Once validated, the prod image moves through QA and into the operational environment.
Bootloader configuration
The bootloader is the first piece of code executed on the device. In Yocto-based systems, generic bootloaders like U-Boot are commonly used. Developers often rely on the bootloader’s interactive command line to interrupt the boot process, run commands, and test new kernels.
However, in production, this interactivity poses a security risk.
To mitigate this:
- The interactive command line is disabled using CONFIG_BOOTDELAY=-2
- Unnecessary commands like tftp, go, etc are removed
- These settings are managed via the UBOOT_CONFIG variable in the machine configuration
conf/machine/msc_sm2s_imx8mp.conf
UBOOT_CONFIG = “sd sd-dev”
UBOOT_CONFIG[sd] = “msc_sm2s_imx8mp_defconfig”
UBOOT_CONFIG[sd-dev] = “msc_sm2s_imx8mp_defconfig_dev”
By defining multiple configurations (e.g., sd for production and sd-dev for development), both bootloader binaries can be built simultaneously and embedded into their respective images.
Linux kernel configuration
The Linux kernel follows the bootloader. While developers may want verbose console output, production systems often require silence for security or aesthetic reasons.
Instead of maintaining two separate Linux kernel builds, this behavior is controlled via kernel command-line parameters passed by the bootloader. This keeps the kernel consistent across environments while allowing different runtime behaviors.
Managing packages
To simplify their activities, developers will add extra packages in their dev image. Yocto provides image features to manage package inclusion:
- tools-debug: Adds debugging tools like gdb, strace, etc.
- ssh-server-openssh: Enables SSH access
- Custom features can be defined to group interactive and tracing tools into package groups (e.g., packagegroup-dev-tools)
These features are included only in the dev image, keeping the production image lean and secure.
<image>-dev.bb:
IMAGE_FEATURES += “tools-debug”
IMAGE_FEATURES += “ssh-server-openssh”
IMAGE_FEATURES += “development”
FEATURE_PACKAGES_development = “\
packagegroup-welma-interactive \
packagegroup-welma-tracing
Read-only file systems
Read-only root filesystems are common in embedded systems for consistency and integrity. However, developers need writable areas to test and deploy code.
To avoid discrepancies:
- Maintain the same read/write layout in both dev and prod images
- Use overlay or bind mounts to simulate writable areas
- Use update mechanisms or scripts to temporarily remount filesystems as read/write during development
Yocto’s read-only-rootfs image feature helps configure these behaviors, but additional customization may be needed for non-root partitions.
<image>.bb:
IMAGE_FEATURES += “read-only-rootfs”
Root password and access
To streamline development, Yocto offers the debug-tweaks feature:
<image>-dev.bb:
IMAGE_FEATURES += “allow-empty-password allow-root-login empty-root-password”
This configuration:
- Sets an empty root password
- Enables root login over SSH
- Modifies /etc/passwd, /etc/shadow, and SSH configuration
This is strictly for development and must be excluded from production images.
Interactive shells
Developers prefer user-friendly shells like bash for features like command history and colored prompts. However, dash is often preferred in production for its speed, size, and permissive licensing (not GPLv3).
To balance both:
- Set /bin/sh to dash for system scripts
- Set the root user’s login shell to bash in the dev image
- Include bash only in the dev image using post-processing commands in the image recipe
<image>-dev.bb:
ROOTFS_POSTPROCESS_COMMAND:append = “set_bash_login_shell; “
set_bash_login_shell () {
…
sed -i “s%\(^root:[^:]*:[^:]*:[^:]*:[^:]*:[^:]*:\).*%\1/bin/bash%” \
${IMAGE_ROOTFS}/etc/passwd
Common pitfalls and how to avoid them
While managing dual images in Yocto brings structure and clarity, it also introduces potential pitfalls that teams must proactively address.
1. Inconsistent tooling across images
A common issue arises when different implementations of the same tool exist in dev and prod images. For example, if the dd command in the dev image comes from coreutils and in the prod image from BusyBox, developers might unknowingly use command-line flags that are unsupported in production. This can lead to runtime failures that are hard to trace.
Best practice: Standardize toolsets where possible, or clearly document differences. Encourage developers to test scripts against the production image to catch incompatibilities early.
2. Shell script compatibility
If your dev image uses bash and your prod image uses dash, scripts written with bash-specific syntax (like arrays) may fail in production.
Best practice: Enforce dash-compatible scripting across the team, and validate scripts in both environments.
3. Read-only vs. read-write filesystems
As discussed earlier, discrepancies in filesystem permissions can cause unexpected behavior. A script that writes to a temporary file in a writable location during development may fail in production if that location is read-only.
Best practice: Maintain consistent filesystem layouts between dev and prod images. Use overlay or bind mounts to simulate writable areas in a controlled way.
4. Root password oversights
Never ship a production image with an empty or default root password. While debugging tools are useful for development, they must be excluded from production builds.
Monitoring and managing differences
To keep dev and prod images aligned:
- Use Yocto’s image manifests (found in tmp/deploy) to compare the list of installed packages.
- Review differences regularly to ensure they are intentional and justified.
However, these manifests won’t catch all issues – like shell incompatibility or missing debug flags. Consider developing custom tools or scripts to summarize and validate critical differences such as root password status, filesystem permissions, and shell configurations.
Testing
Here is a set a useful testing practices and tools to implement to make sure both images function as expected:
- CI/CD pipelines: Automate builds for both dev and prod images across all target boards.
- Vulnerability scanning: Run security scans on both images to identify potential risks.
- Offline testing: Analyze .wic files and their contents to validate structure and configuration.
- On-device testing: Run regression tests on real hardware using the dev image. When possible, also perform full functional testing on the production image to validate real-world behavior.
Cybersecurity considerations
Security is a key driver behind dual image management:
- Minimal attack surface: Remove unnecessary packages and services from the production image.
- Read-only filesystems: Prevent unauthorized modifications and improve system integrity.
- Secure boot: Ensure only authenticated software runs on the device. Read our article on how to enable secure boot.Note that in production, you will need to burn the “closed” eFuses to enforce signature verification, while during development, teams typically burn only the public key hash, leaving the “closed” eFuses unset to allow flexibility.
Important: Use the same signing key for both dev and prod images during development. Before release, re-sign the production image with a separate, trusted authority’s key.
Final takeaways
To wrap up, here are the key principles for managing dual images in Yocto:
- Keep dev and prod images as close as possible to reduce surprises.
- Monitor and document differences using manifests and custom checks.
- Test both images thoroughly, including on real hardware.
- Never ship development configurations (like empty root passwords or debug tools) in production.
By following these practices, you’ll build embedded systems that are not only robust and secure but also developer-friendly and maintainable.
