Why distinguish development vs production images?
In the old days, we sometimes ended up in the development phase with a Linux image running perfectly, meeting all functional requirements, and when it was time to send it to production and have it installed on products and shipped to end users, we suddenly realized that passwords should be changed and some tools used by developers should be removed (because they were not needed, cumbersome, and would possibly help attackers).
And it turned out that changing a password in a Linux image, be it an ext4 filesystem image or something more complex, is not always a piece of cake.
What’s more, removing unneeded tools can be a long and tedious task, as it is not always clear how programs or libraries depend on each other, and tools that look unneeded at first sight may finally turn out to be actually needed.
So, how do we deal with that? Developers want interactive and tracing tools, whereas end users do not want these.
Dual images: definitions
This is where the dual image comes into play. Let’s have developers work with a development image and deliver a production image.
Production image
A production image is the one intended for the end products:
- with application software
- without tools for software developers
Development image
A development image is the one intended for software developers:
- with the same software as in the production image
- and with tools for developers
The impact of the bootloader: the example of U-Boot
Moreover, the production and development images can also differ regarding the bootloader. A popular bootloader is U-Boot. By default, U-Boot offers an interactive command line that lets developers call various commands, such as format non-volatile storage, download files by TFTP, list files of the filesystem, modify CPU registers, and so on.
Example of a U-Boot command that downloads and starts a Linux kernel:
u-boot=> tftp 0x200000 /tftpboot/kernel
u-boot=> bootm 0x200000
Of course, in a production image, these should be deactivated.
Managing production & development images with Yocto
Image recipes definition
To effectively manage these dual images within the Yocto Project, two distinct image recipes are created:
- Production image: project-image.bb
- Development image: project-image-dev.bb
The development image recipe extends the production image, adding features such as:
- A trivial root password.
- An SSH server with root login enabled.
- Developer-centric packages including:
- Interactive tools like a text editor (vim), top…
- Tracing tools such as tcpdump, strace…
This approach equips software developers with necessary tools while ensuring the production image remains free from superfluous components.
Implementation in Yocto recipes
The organization of package groups in Yocto recipes enhances clarity and maintainability. Here’s an example of the configuration for the interactive and tracing tools.
File: packagegroup-interactive.bb
DESCRIPTION = “Package group bringing in programs useful for user interaction”
RDEPENDS:${PN} = “\
bash \
coreutils \
ethtool \
iproute2 \
killall \
less \
mmc-utils \
net-tools \
socat \
systemd-analyze \
time \
util-linux-hexdump \
vim \
which \
“
File: packagegroup-tracing.bb
DESCRIPTION = “Package group bringing in programs useful for tracing system activity”
RDEPENDS:${PN} = “\
ltrace \
strace \
tcpdump \
“
And in our development image recipe:
File: project-image-development.bb
require project-image-production.bb
IMAGE_INSTALL += “packagegroup-interactive packagegroup-tracing”
IMAGE_FEATURES += “debug-tweaks”
For in-depth explanations regarding these parameters, consult the Yocto Project documentation on IMAGE_INSTALL, IMAGE_FEATURES, and debug-tweaks.
Considerations and limitations
We still have to be careful though, especially if developers test their code with the development image, which is sure to happen somehow.
Example 1: rootfs
Think about this example: rootfs is read-only in the production image but read-write in the development image because developers need to be able to install or modify files. Then they forget about that and develop a new feature that needs some read-write storage. If they locate these files in rootfs, then the feature will work in the development image but not in the production image!
Example 2: shell interpreters
Or this more tricky example about shell interpreters.
What we call “shell” here is a text command-line user interface that provides various commands for file management, process management, etc. It is generally used by developers in an interactive mode to communicate with the operating system via a terminal. It is also widely used as a high-level programming
language.
Several shells can be used, two of them being “dash” and “bash”.
Dash (compared to bash):
- is many times faster
- is much smaller (takes less space and has a smaller attack surface)
- has a more permissive license
Bash:
- has some more advanced syntax (e.g., arrays)
- is more user-friendly in interactive mode, with colored prompts and command history and much more
- is licensed under GPLv3
So, let’s say that the production image has dash, and developers install bash in the development image, in order to benefit from a comfortable interactive environment.
Dash and bash have globally the same syntax, except for some bash-specific expressions that are invalid in dash. What happens if a developer writes a shell script using a bash-specific syntax that correctly runs on the development image, but not on the production image?… It will be detected later in the development process, or worse, in the field.
This is why we have to be careful and minimize differences between production and development images.
Dev vs prod images: best practices
Establishing a clear distinction between production and development images is an essential best practice in software engineering. As a standard in our embedded Linux environment, we advocate for:
A production image with:
- dash as default shell (/bin/sh -> /bin/dash)
- a rootfs that is read-only
A development image with:
- dash as default shell (same)
- read-only rootfs (same)
- bash for interactive sessions
- interactive and tracing packages
By adhering to this image segmentation, embedded software developers can cultivate a robust and secure embedded system development framework.
