I recently got to dive into the embedded Linux build systems world while working on a project where I needed to create a distribution for an embedded device with limited resources.

Amongst the different options, I had identified two major tools : Buildroot and Yocto (See this fantastic talk by Alexandre Belloni and Thomas Petazzoni from Bootlin : Buildroot vs. OpenEmbedded/Yocto Project).

I decided to go for Yocto even if the learning curve was supposedly a bit steeper. In this article I'll try to describe my experience and give some tips along the way.

In a nutshell, what I wanted to do

So, I want to build an image for an Intel-based board, the Minnowboard Turbot. My goal is to have an image that boots silently, that runs a X environment and that can launch a fullscreen browser on boot.Pretty much a kiosk.

The specs of the board are here : https://minnowboard.org/minnowboard-turbot/technical-specs

Getting started

Building embedded Linux images is not really tricky per se, but it needs a lot of resources, and a lot of time. I mean a lot because you may be reading this article on your quad core 8Go Macbook Pro and probably think "ahah". Don't you dare. You need at least 100Go of disk and a good 16Go of RAM to have correct build times for Yocto. And when they say 100Go, be sure that they will be filled quite fast, and that the RAM /CPU usage will be close to 100% all the time, so it's a pain to work on the same machine at the same time.

Of course, I won't copy and paste the whole Getting started section of the Yocto project website since it's not really useful, but let's comment on a few steps here.

Get the correct docs

Here is the up to date documentation first :
https://www.yoctoproject.org/docs/latest/yocto-project-qs/yocto-project-qs.html

Pro tip™ : if you search for "yocto getting started" on Google, you most likely will land on an outdated version of the same documentation, which is .. unsettling.

Decide which machine you will use for the build

→ refers to the resources in the docs

I had a medium range linux machine (16Go, quad core) that was not used, so I decided to wipe it clean and reinstall a brand new Debian 9 on it.

When they say you need 50Go at least, double that number to be sure. My current folder is actually 64Go, and I cleaned most of previous builds and images to free up space already :

root@builder:~$ du -h -s yocto
64G	  yocto

You can use docker containers (via CROPS) but I would not recommend it since you will likely need a lot of resources.

The full uname -a for the machine I used is :
Linux 4.9.0-5-amd64 #1 SMP Debian 4.9.65-3+deb9u2 (2018-01-04) x86_64 GNU/Linux

Having a lot of RAM on the build machine can certainly help but having more than 4 cores shouldn't change your build time that much, since the things that take time will usually block your builds (many other packages depend on it), so it needs to be built before.

Processor speed can give you shorter build times.

Packages

→ refers to the packages in the docs

You need a few packages to be able to build everything correctly. Apart from the standard packages that the doc tells you to install ... :

apt install gawk wget git diffstat unzip texinfo gcc-multilib \
 build-essential chrpath socat cpio python python3 python3-pip python3-pexpect \
 xz-utils debianutils iputils-ping libsdl1.2-dev xterm

(Note that I installed git and not git-core since the latter is obsolete)

... I would recommend these as well :

Tools, always handy to have to modify a local conf easily or get something from the web

vim htop wget curl screen tree
Packages I needed to install at some point to resolve an error in the build process (no clear explanation as why, but hey, the more packages the merrier)

libssl-dev libev-dev asciidoc bc

Building and running in qemu

Cloning the repo is straightforward, you can source the oe-init-build-env file
and start the build with bitbake core-image-sato from the build folder for example.

The getting started tutorial is quite well-written in that regard.

4 hours minimum later, and you can emulate your machine with :

runqemu qemux86

(that is, if you have not changed any file, or option, for now)

Pro tip™: if you have a headless machine : runqemu qemux86 nographic

Copying to a SD card for testing on the target hardware

When compiling for Intel targets, a .wic image is created and is pretty straight-forward to use : just dd it to your SD card :

sudo dd of=/dev/diskX if=yourimage.rootfs.wic

Note : by default meta-intel .wic images only have an EFI bootloader.

Pro tip™: when you want to copy, use bs=512k. It's the standard bootloader size, and I've been told numerous times that it's always better to use this size. I stick with it and I think it's a good idea. And moreover I've had cases where using a different size would make the resulting SD card unable to boot at all.

Customize everything

So far, it's an easy ride. But we don't want a standard Sato GUI. So let's start customizing things. That's where things get a little harder.

In a nutshell

There is a quite complex image that gives you a general overview of the architecture of the Yocto environment :

... but I find it too abstract, so I'll simplify the concept and try to give you a more basic view of how it works in practice :

To customize your build, what you want to do is create a layer to hold everything in it. This layer (in green above) will reside alongside the other layers of your build.

A layer (Named along the lines of meta-my-layer for example) contains recipes, which can be image recipes (a special kind of recipe). Technically, a layer is a folder with some conventional files in it.

You create your layer, and you create all your recipes in it, that you can organize in recipe folders (recipes-common, recipes-my-app, etc ... in grey on the schema). A 'recipe folder' is just a folder. A recipe is also just a folder that contains at least one mandatory recipe file (a priori, a .bb file

Your recipes will possibly be only appending stuff to an already-existing recipe (a conf file, some bash scripts, etc). In this case, the recipe file is a .bbappend file

Finally, your image recipes will possibly call the packages from the recipes you created in your own layer, and packages from other layers too.

The process for customizing

Basically, you want to start with the smallest image that suits your needs, and add the packages you need in it (let's say chromium, X, and a custom app of yours, for instance). Two different steps to take:

Clone the different layers that contain the recipes for the packages that you want to install
Create your own layer and recipes for your custom app, and your modifications to existing packages if needed
Finally, create an image layer that will hold the reference to all these packages : the ones that exist and your custom ones.

The files you must care about

Apart from your layer folder (we'll come to that in a bit), there are only two files that are important for the build :

build/conf/local.conf
build/conf/bblayers.conf

These will be modified quite often while you create your perfect image, so be sure to keep them at hand.

Adding existing layers to your build

Now, before you can install any package onto your build, you might want to add the layer in which this package is. This is the role of the build/conf/bblayers.conf file, that allows you to add layers in your build.

First, you have to download the layer in a folder somewhere. I use the root folder of poky so that all the layers are in the same place, but it's up to you.

If you are looking for layers or recipes, this is the place to go to :

You will then have to define the path of your layer in the build/conf/bblayers.conf file.

Suppose you want to use the meta-intel layer. First, clone it with :

git clone git://git.yoctoproject.org/meta-intel

Then, add its path in the file :

POKY_BBLAYERS_CONF_VERSION = "3"

BBPATH = "${TOPDIR}"
BBFILES ?= ""

BBLAYERS ?= " \
  ${TOPDIR}/../meta \
  ${TOPDIR}/../meta-poky \
  ${TOPDIR}/../meta-yocto-bsp \
  ${TOPDIR}/../meta-intel \         # <----- HERE
  "

BBLAYERS_NON_REMOVABLE ?= " \
  ${TOPDIR}/../meta \
  ${TOPDIR}/../meta-yocto \
  "

Once you rebuild your image or any package, this layer will be taken into account and the necessary recipes will be found.

Creating your layer

Now, let's create our own layer and recipes.

We'll name it meta-tchap. It will reside in the poky folder, where all the other meta-folders reside too.

We'll just create the basic structure here, with an images folder that will contain an image recipe.

root@builder:~/yocto/poky$ tree meta-tchap
meta-tchap
├── conf
│   └── layer.conf
└── recipes-core
    └── images
        └── core-image-iot.bb

Pro tip™: There is a tool for that if you don't want to dive into the files : yocto-layer, that you can access when you have sourced everything in your build folder. Something simple like yocto-layer create meta-tchap should do the trick.

The layer.conf file

It's a relatively standard file that is not likely to change, it justs imports all the recipes of the layer :

    # We have a conf and classes directory, add to BBPATH
    BBPATH .= ":${LAYERDIR}"

    # We have recipes-* directories, add to BBFILES
    BBFILES += "${LAYERDIR}/recipes-*/*/*.bb \
        ${LAYERDIR}/recipes-*/*/*.bbappend"

    BBFILE_COLLECTIONS += "meta-tchap"
    BBFILE_PATTERN_meta-tchap= "^${LAYERDIR}/"
    BBFILE_PRIORITY_meta-tchap = "100"

Since it's my own layer, I usually bump the priority up (Each layer has a priority, which is used by bitbake to decide which layer takes precedence if there are recipe files with the same name in multiple layers. A higher numeric value represents a higher priority.)

Creating the image recipe

An image recipe is a recipe. You just have to define what will be installed on it as a basis, with features and packages.

Let's create a simple image with X11, Chromium, Node and some other packages. We'll call it core-image-iot and it will reside in ...meta-tchap/recipes-core/images/core-image-iot.bb :

SUMMARY = "A full-featured + basic X11 / Chromium / NodeJS image."
LICENSE = "MIT"

IMAGE_FEATURES_append = " splash x11-base hwcodecs ssh-server-openssh post-install-logging"
IMAGE_FEATURES_remove = "allow-empty-password"
IMAGE_FEATURES_remove = "empty-root-password"

IMAGE_INSTALL = "\
    packagegroup-core-boot \
    packagegroup-core-full-cmdline \
    psplash \
    chromium-x11 sudo vim git curl htop wget unzip usbutils \
    nodejs nodejs-npm \
    ${CORE_IMAGE_EXTRA_INSTALL} \
    "

inherit core-image extrausers

# Here, we set the root password for the image
# password = test
EXTRA_USERS_PARAMS = "usermod -p m7or76bu6AEY6 root;"

This image inherits the core-image recipe, that has a lot already (see the file).

In this recipe, we use:

IMAGE_FEATURES : it allows us to add features to the images, such as using the post-install-logging or adding x11. Features are just wrappers around packages, in fact
IMAGE_INSTALL : it adds packages to the image, such as git, chromium-x11, etc ...
EXTRA_USERS_PARAMS : it's a module that allow to work with users. Here, we use it to set the root password easily. You have to inherit it first.

Once we have this file, we can build this image very easily with :

bitbake core-image-iot

You need to clone a few layers before being able to build this image, since it relies on packages that are not in the base layer : https://github.com/OSSystems/meta-browser and https://github.com/imyller/meta-nodejs

Adding existing packages to your build

There are some packages that we don't want to put into our image recipe, but in the configuration file first, so it's easier to switch on/off between different builds or for testing.

To simply add a package to your build, you can add it to your build/conf/local.conf file :

IMAGE_INSTALL_append = " htop"

Pro tip™ : note the " " (space) after the opening quote, it's important, the string will be concatenated.

In this example, the htop package (that exist here).

As we have seen before, you have to have the layer somewhere beforehand. In the htop case, it's the meta-oe layer. You thus have to clone the repository (see info here):

git clone https://layers.openembedded.org/layerindex/branch/master/layer/meta-oe/

The meta-oe layer itself is in a subfolder of that git repo (as indicated in the openembedded.org page), so you should had the relevant path to your build/conf/bblayers.conf file like this :

${TOPDIR}/../meta-openembedded/meta-oe \

which give us a bblayers.conf file like this :

BBLAYERS ?= " \
  ${TOPDIR}/../meta \
  ${TOPDIR}/../meta-poky \
  ${TOPDIR}/../meta-yocto-bsp \
  ${TOPDIR}/../meta-openembedded/meta-oe \ # <----- HERE
  "

Now, you're ready to rebuild your image and tada, the htop binary should be available once you boot it.

 root@iotboard:~/$ htop --version
    htop 2.0.2 - (C) 2004-2016 Hisham Muhammad
    Released under the GNU GPL.

Congrats ! You now have a functionning build with your own image, your own recipes and probably a few standard package factored in too, that's great.

Append an existing recipe

Now you might have a recipe that exists but that you wish to modify somehow. You could create a whole new recipe, copy and paste the content of the recipe you wish to amend and call it a day (if your layer has a higher priority of course). But then, it's not really the Yocto way.

If you want to append something to a recipe, you can easily create a bbappend file and make your changes there.

Let's say you want to change the Chromium X11 recipe to add some flags to the build. Create the recipe folder as if you would create your own recipe and create a chromium-x11_%.bbappend file in it, with the following content :

FILESEXTRAPATHS_prepend := "${THISDIR}/${PN}:"
PACKAGECONFIG = "use-egl kiosk-mode proprietary-codecs"

Here, I'm adding these flags to the build (second line). The first line is a boilerplate instruction telling bitbake where it should look for extra files if any; Just put it there in all your append recipes.

In my layer folder (meta-tchap), I now have a folder more with the following structure :

recipes-browser/
└── chromium
    └── chromium-x11_%.bbappend

And that's it ! (don't forget to add your custom layer in the conf/bblayers.conf if you haven't done so yet).

PS : the name recipe-browsers is completely up to you. chromium is not : it's the name of the recipe you are appending.

If you want to check that your append is taken into account correctly, you can use the bitbake-layers utility :

bitbake-layers show-appends

It will show you which recipe is appended and by whom. Very practical.

Quiet boot with standard kernels

It's as easy as adding :

APPEND += "quiet vt.global_cursor_default=0"

.. to your image recipe, or your local.conf file.

Quiet boot with meta-intel

Well, not that easy, and so far I did not manage to pass boot options when using the meta-intel layer.

See : https://stackoverflow.com/questions/49033507/amend-boot-cmdline-in-custom-image-build

It seems complicated. As far as I understand the kernel does not take the APPEND variables into account. This seems to work on non-Intel builds, but not with the meta-intel layer.

To circumvent this limitation, I created a postinst script (see below) with the following content :

pkg_postinst_${PN} () {
    #!/bin/sh -e
    if [ x"$D" = "x" ]; then
        echo "default boot" > /boot/loader/loader.conf
        sed -i 's/console=tty0/quiet splash vt.global_cursor_default=0 vt.cur_default=0/' /boot/loader/entries/boot.conf
    else
        exit 1
    fi
}

This will modify options directly in the boot loader entries under /boot. This works well.

Exploring psplash

psplash is the de facto standard on poky to display a splash image while the system it booting.

Disclaimer : it's tied to sysvinit by default, so if you want to use systemd as your init system of choice, psplash won't work. Patches exist, though (I have not tested them) — one I found is https://patchwork.ozlabs.org/patch/351283/.

Create a custom psplash image

Two options here. Either you create it yourself and add the image file (as a .h file) in your layer, or you let bitbake recompile the image file to a .h file each time you build.

In both cases, you need to install psplash :

IMAGE_INSTALL_append = " psplash"

And create the bbappend file (I suggest psplash_git.bbappend) along with the files directory that will hold the image :

vi ...meta-tchap/recipes-core/psplash/psplash_git.bbappend
mkdir ...meta-tchap/recipes-core/psplash/files/

(do not forget to adapt for your own path)

1. First option : Create the image yourself

Clone the repo somewhere :

git clone git://git.yoctoproject.org/psplash && cd psplash

Considering your image is at ./psplash-poky.png :

./make-image-header.sh ./psplash-poky POKY
mv psplash-poky-img.h ...meta-tchap/recipes-core/psplash/files/.

(change the paths to fit your config obviously here too)

Your bbappend file will look like this :

FILESEXTRAPATHS_prepend := "${THISDIR}/files:"
SPLASH_IMAGES = "file://psplash-poky-img.h;outsuffix=default"

If you want to change the image, you have to redo this process (except the git clone of course)

2. Second option : Let bitbake do it

Well, pretty straightforward. No need to clone anything, create the bbappend file directly and put your PNG image in ...meta-tchap/recipes-core/psplash/files/ directly :

    FILESEXTRAPATHS_prepend := "${THISDIR}/files:"
    DEPENDS += "gdk-pixbuf-native"
    SRC_URI += "file://psplash-poky.png"
    SPLASH_IMAGES = "file://psplash-poky-img.h;outsuffix=default"

    do_configure_append () {
        cd ${S}
        # will create psplash-poky-img.h for you :
        ./make-image-header.sh ./psplash-poky POKY
    }

This will compile the image file at configure time.

psplash rotation

Because sometimes you need it, add this to your psplash_git.bbappend (if you want 90°) :

do_install_append() {
    echo 90 > ${D}/etc/rotation
}

Chromium on a basic X11 image

Working on my kiosk image, adding nodejs and npm was a walk in the park (just add the layer and append the recipes to the image). But adding Chromium and making it start at boot required a little more work.

You need a working X environment to display Chromium. Fortunately the x11-base image feature (IMAGE_FEATURES_append = " x11-base") provides just that, and no more.

You can then go for a full-fledged window manager but I personnaly think this is overkill if you're working on a device with limited resources such as an embedded system.

The X11 base feature contains just the vital minimum for this use case :

A very simple and lightweight window manager (Matchbox)
the Mini-X-Session session manager

Matchbox is an relatively old piece of code (~2012), is mainly intended for embedded systems and differs from most other window managers in that it only shows one window at a time. That's exactly what we need, in fact.

Mini-X-Session is a very simple session manager for X, that provides just the right boilerplate for us to create our session and launch the browser (see http://cgit.openembedded.org/openembedded-core/tree/meta/recipes-graphics/mini-x-session/mini-x-session_0.1.bb)

The mini_x session file

So, how do we glue all this together ? Well, first I'd recommend you create a user to run your X session, let's call him myUnprivilegedUser. You don't really want to run X as root on a probably user-facing linux box.

I suppose I have an app running at http://localhost:3001 for the sake of it (The app you want to run in your browser).

You can then create a session file in /etc/mini_x/session.d/ (or, quicker, replace /etc/mini_x/session altogether if you want to lock things a bit) — I've commented the file below so you know what it does exactly :

#!/bin/sh

# This script will be called via mini X session on behalf of the file owner, after
# being installed in /etc/mini_x/session.d/.

xset s off  > /dev/null 2>&1        # don't activate screensaver
xset -dpms   > /dev/null 2>&1       # disable DPMS (Energy Star) features.
xset s noblank   > /dev/null 2>&1   # don't blank the video device

# Set a resolution
xrandr -s 1920x1080

# Takes care of rotating the screen based on the content of /etc/rotation
# I've only implemented a single case (90 CW) but it is trivial to amend.
if grep -Fxq "90" /etc/rotation
then
    xrandr -o left
fi

# Run the wm first, so that our Chromium window is "aware" of the screen, and can resize correctly (if you don't run a wm first, your Chromium window will likely be half the size of the screen)
matchbox-window-manager -use_titlebar no -use_cursor no &

# Now, run chromium 'as' your user
su -l -c "/usr/bin/chromium --user-data-dir=/tmp --disable-session-crashed-bubble --disable-infobars --noerrdialogs --disable-restore-background-contents --disable-translate --disable-new-tab-first-run http://localhost:3001" myUnprivilegedUser

All the flags

All the flags are carefully crafter here in an attempt to minimize UI junk like popups and infobars. Customise for your own need.

But you might notice that some flags are missing for a proper kiosk experience. This is because they have been incorporated in the chromium build itself with Yocto, as I'm using a recipe that allows for it : https://github.com/OSSystems/meta-browser/tree/master/recipes-browser/chromium

For instance, to build Chromium with the kiosk mode already in, it's a easy as appending the Chromium recipe (we have covered that previously, but I'll just redetail here quickly)

Create a chromium-x11_%.bbappend file in meta-tchap/recipes-browser/chromium with the following content :

FILESEXTRAPATHS_prepend := "${THISDIR}/${PN}:"
PACKAGECONFIG = "kiosk-mode"

Rebuild your image (don't forget that building Chromium is a loooong process), and the flag will already be present (i.e., when running /usr/bin/chromium, it will already be in kiosk mode by default).

Trace/Breakpoint trap ? Oh, I see, it can be due to Chromium not being able to write to the user/data directory. This is the reason why I added the --user-data-dir=/tmp option when launching the browser to be sure that it doesn't panic on start (took me a while to figure this out in fact)

How to run a script on first boot (and only on first boot)

As we have seen, there are a few things that seem complex to do on specific kernels / under specific conditions, or that need hardware to complete (and thus cannot be done before the image is booted on the actual hardware). For these, a good old script that launches at first boot and then erases itself is clearly the best option. It's fairly transparent, and chances are good that your hardware will boot at least once before being put in production.

Fortunately, this is easily done in Poky. Enter pkg_postinst.

pkg_postinst_${PN} is a function that is run just after the package in which it resides is installed in the image. If the execution of pkg_postinst exits with success (exit 0), the script is removed (since they have run already). If not, they are keept and run again at first boot. With this, you can defer the execution until the first boot easily.

Let's take an example. I will create a specific recipe in my meta-tchap layer, just for that. Let's call it "startup" (any name will do). I have put this recipe under recipes-common but again here, any recipes folder is fine.

root@builder:~/yocto/poky$ tree meta-tchap
meta-tchap
├── conf
│   └── layer.conf
└── recipes-common
    └── startup
        └── startup_1.0.bb

In this startup_1.0.bb file, I will just run a very simple script on first boot that creates the file /tmp/i_was_here. The file /tmp/i_was_installed will be installed before, when the package do_install is called :

SUMMARY = "Just a simple recipe to test postint"
LICENSE = "MIT"
PR = "r1"

S = "${WORKDIR}"

do_install () {
    # You have to do something here ! Else, the package will likely not be installed
    touch ${D}/tmp/i_was_installed
}

pkg_postinst_${PN} () {
    #!/bin/sh -e
    if [ x"$D" = "x" ]; then
        # This will run on first boot
        touch /tmp/i_was_here
    else
        # This will run after package installation
        echo "Skipping postinst script, will do on first boot"
        exit 1
    fi
}

Do not forget to add the package to your image installs in your local.conf :

IMAGE_INSTALL_append = " startup"

If you build the image and open a devshell (see next section) afterwards to check which files are here, you will notice that only the /tmp/i_was_installed file will have been created.

Once you boot this image on the actual hardware, the /tmp/i_was_here file will be created, too, and the script will be deleted from the system automatically.

Pro tip™ : the list of all poky standard target filesystem paths are in the source here

The devshell

A devshell is a shell that opens in the recipe's target directory so you can check what has really been done / installed, and what will be copied onto the image.

bitbake -c devshell <recipename>

Pro tip™ : you have to fully build your recipe first (bitbake <recipename>), to access all the folders that I go through below. Doing so can help you troubleshoot specific problems with your build. IF you don't, only the source folder will be available (the git folder, for instance)

This will get you in the working dir for your recipe and source a shell with bitbake’s environment set up. Several folders there :

git : where the git source are fetched and built (that is, in the case where you use git to fetch your sources)
package : is where the files are copied for creating the ipk (or rpm) package that will be installed at the end. This is technically a replica of the / folder of your image, with just the folder and files that your recipes own and will install
deploy-ipks (could be named deploy, or deploy-* depending on the package manager that you choose in your local.conf — I choose ipk). It's what will be installed via the package manager onto the image. Generally, you have three packages, one for production, one for development and one for debug.

You'll likely explore the package folder to see what you recipe created and check for potential problems.

Pro tip™ : exit to go back to your shell

Where does that variable value comes from ?

You will surely often wonder where does a specific configuration comes from, or what is its value.

For instance, you might want to know what is the preferred provider for the kernel, that is behind an obscure virtual/kernel variable.

In this case, bitbake -e comes in handy :

$ bitbake -e <your_image_name_here> | grep "^PREFERRED_PROVIDER_virtual/kernel"
   PREFERRED_PROVIDER_virtual/kernel="linux-intel"
   PREFERRED_PROVIDER_virtual/kernel_poky-tiny="linux-intel"

Great ! We now know the value !

Reconfigure the kernel you say ?

If you want to reconfigure the kernel, you can use the menuconfig utility. This page has relevant information on how to do it, and especially on how to save your modifications to a .config file and then find this bloody file in the Yocto tree.

bitbake -c menuconfig virtual/kernel

Pro tip™ : for an intel corei7 build the path you are looking for is something similar to :

    build
     └ tmp
       └ work
         └ corei7-64-intel-common-poky-linux
           └ linux-intel
             └ 4.9.81+gitAUTOINC+*
               └ linux-corei7-64-intel-common-standard-build
                  └ .config

For the Minnowboard, one configuration item that needs to be changed is CONFIG_IGB=y. It adds the Intel Gigabit Ethernet driver directly into the kernel instead of as a module, and helps recover the eth0 interface that is not created at boot otherwise.

∗ ∗ ∗

Alright. This was a bit long, but I hope it can help anyone fiddling with Yocto to not bang its head over an obscure bug / behaviour. If you spot a mistake or think I have missed something, do not hesitate to drop me a line or comment below.