I was trying to setup a full instance of Mattermost recently and decided to use an existing Docker host so that I can test it out and easily rebuild / tinker with it.

The official documentation states that there is a community-driven repository that offers just that, Mattermost on Docker:

mattermost/mattermost-docker

Dockerfile for mattermost in production. Contribute to mattermost/mattermost-docker development by creating an account on GitHub.

GitHubmattermost

... but the README greats us with a warning message:

The current state of this repository doesn't work out-of-the box since Mattermost server v5.31+ requires PostgreSQL versions of 10 or higher.

Mattermost being a Go application backed by a PostgreSQL server, I figured that this should not be overly difficult to achieve on its own, so I opened up a terminal and created a simple compose file to deploy it easily.

Structure

• We need a Dockerfile to build the actual server container
• We need to create a config.json file
• And a docker-compose.yml file to setup the database along the server, and bind them together

The Dockerfile is pretty straightforward. We start for a relatively stable debian Buster:

FROM debian:buster

ENV version=5.34.2

# Install the PHP extensions we need
RUN set -eux; \
	apt-get update; \
	apt-get install -y --no-install-recommends \
	ca-certificates \
	openssl \
	curl \
	gnupg

RUN apt-get purge -y --auto-remove -o APT::AutoRemove::RecommendsImportant=false; \
	rm -rf /var/lib/apt/lists/*

# Get release
ADD https://releases.mattermost.com/${version}/mattermost-${version}-linux-amd64.tar.gz /

RUN tar -xzf /mattermost-${version}-linux-amd64.tar.gz
RUN mv /mattermost /opt/.

RUN rm -rf /mattermost-${version}-linux-amd64.tar.gz

# Add correct configuration
COPY config.json /opt/mattermost/config/config.json

# Set up a system user and group called mattermost that will run this service, and set the ownership and permissions.
RUN useradd --system --user-group mattermost
RUN chown -R mattermost:mattermost /opt/mattermost
RUN chmod -R g+w /opt/mattermost

USER mattermost:mattermost

VOLUME /opt/mattermost/data

CMD ["/opt/mattermost/bin/mattermost"]

Key points here are:

• We need ca-certificates and openssl so that we can access https resources and APIs
• We want the container to run with a non-root account (hence the mattermost user)
• We want to expose the Mattermost data in a volume

You need a .env file to hold the env vars for the PostgreSQL database

POSTGRES_USER=mmuser
POSTGRES_PASSWORD=a_strong_password
POSTGRES_DB=mattermost

And a docker-compose.yml file to hold all this together:

version: '3.7'

services:
  mattermost-postgres:
    container_name: mattermost-postgres
    image: postgres:13.2-alpine
    environment:
      POSTGRES_USER: ${POSTGRES_USER}
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
      POSTGRES_DB: ${POSTGRES_DB}
      PGDATA: /mattermost/postgres
    volumes:
       - postgres:/mattermost/postgres
    networks:
      - mattermost
    restart: unless-stopped

  mattermost-server:
    container_name: mattermost-server
    build:
      context: ./
    image: mattermost-server:latest
    environment:
      POSTGRES_USER: ${POSTGRES_USER}
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
    volumes:
      - data:/opt/mattermost/data
    networks:
      - mattermost
    ports:
      - "127.0.0.1:8066:8065"
    restart: unless-stopped
    depends_on:
      - mattermost-postgres
    ulimits:
      nofile: 49152

networks:
  mattermost:

volumes:
  postgres:
    driver: local
    driver_opts:
      o: bind
      type: none
      device: /mattermost/postgres
  data:
    driver: local
    driver_opts:
      o: bind
      type: none
      device: /mattermost/data

On the host, you need to have the /mattermost/data and /mattermost/postgres directories available.

The config.json file that is referenced in the Dockerfile is a copy/paste of the standard Mattermost configuration that you can find here:

https://github.com/mattermost/mattermost-docker/blob/master/contrib/aws/app/mattermost/config/config.json

The main keys to update are:

• ServiceSettings > SiteURL: You want to put your domain name here, with the https:// part, ex: https://mattermost.my_domain.com
• SqlSettings > DriverName and DataSource: DriverName should be postgres and DataSource is your full connection string which should be along those lines: postgres://mmuser:[password]@[postgres_container]:5432/mattermost?sslmode=disable&connect_timeout=10
  Don't forget to change your password for the one that you setup in your .env file, and the postgres_container for the name that you gave to your container in the docker-compose.yml file (mattermost-postgres in my example)
• EmailSettings: If you want to have notifications, you need to put a working SMTP configuration here (in EnableSMTPAuth, SMTPUsername, SMTPPassword, SMTPServer and SMTPPort, and also set SendEmailNotifications to true, and fill in FeedbackName, FeedbackEmail and ReplyToAddress)

Protip for these keys:

• FileSettings > Directory
• PluginSettings > Directory and ClientDirectory

You want to put absolute directories in here, instead of relative ones, it should work better (ex /opt/mattermost/plugins instead of ./plugins)

Start up

Now that you're all set, you can:

Build the server container

With docker compose build mattermost-server

Start the app

With docker compose up -d

You should be able to visit 127.0.0.1:8066 on your Docker host to assess that everything runs fine.

Reverse proxy

So far, the Mattermost instance is only available locally on your Docker host (we specifically requested it to with "127.0.0.1:8066:8065" in the ports section of the docker-compose.yml). But we have mattermost.my_domain.com waiting for us, so let's reverse proxy it !

On the host

If you already have a web server on the host that serves some other containers to the public web, it's easy to use it to reverse proxy the Mattermost app.

I'm mainly using Caddy nowadays, and it's as easy as using this configuration:

mattermost.my_domain.com {
    reverse_proxy 127.0.0.1:8066
}

Via another container

We could also use Traefik (See the docker images here) to reverse-proxy the 8065 port of the mattermost-server container to the world.

This is quite easy to do and I'll leave that as an exercise for the reader.

Once you have the server and the proxy up and running, you should be able to access the instance, create the system account if not done before, and start using Mattermost:

🎉

More than a year ago, I decided to roll up my sleeves and create my own private cloud. I wrote extensively about it here.

Now, it has become my whole personal infrastructure and it works very well (to my standards, at least), so let's look at the changes that it underwent in that time and how I improved it to be more resilient, robust, and fit my needs.

New services & replacements

Over the course of the year, I got to know better the services I used and some of them I was not perfectly fond of.

I've updated the repository continuously, check out https://github.com/tchapi/own-private-cloud.

🗓 Spinning my own CalDAV backend

I had chosen Baikal in my first instance and it worked OK, but it proved quite cumbersome to configure and hard to maintain or evolve due to the legacy codebase.

I thus created my own backend, Davis, based on Symfony 5 and Bootstrap 4, that follows best practices, is easy to configure and dockerize thanks a pretty standard .env workflow.

I also added some nice features such as delegation and sharing (directly in the management section), IMAP authentification, etc ...

It's completely backward-compatible with Baïkal so you can use the exact same MySQL database you used before if you're migrating from it (that's what I did).

Grab the latest release here.

📝 Cryptpad

What was missing in the infrastructure was a document editor, in the like of Google Docs. I had tested OnlyOffice but it seems a bit too complete and not suited for a simpler, personal use.

I settled on Cryptpad: https://cryptpad.fr/what-is-cryptpad.html

The team behind it strongly advocates security and privacy, and although the interface has rough edges, it works quite well.

I have adapted a quite simple and robust Dockerfile for it (you can have more details on the repository itself), that does not allow new registrations and configures a few things:

👋 Wekan → Planka

Eventhough Wekan looked promising on the surface, I never got to enjoy using it. The UI is functional but not on par with what Trello, for instance, can offer.

I went on a quest to find another "task" / kanban software and found Planka, which is still in heavy development, but looks really promising:

I installed it and I was right away convinced by how snappy it felt. It still lacks a lot of things but I decided to try it anyway (and contribute if I can).

It's actually a work in progress that you can find in the services/planka branch on the repo here.

🔀 Reverse proxy

I started with an Nginx + Certbot combo, only to realize that it was quite cumbersome and hard to maintain (especially the certificates).

I chose to replace both of them with Traefik: https://doc.traefik.io/traefik/

It's an edge router that is quite used in the Docker world, and is easy to configure (via labels), while being quite opinionated.

It has its own certificate resolvers / endpoints compliant with ACME, so you can use Let's Encrypt and get certificates in a breeze.

Moreover, you can have a very nice dashboard listing all your routes and endpoints, quite practical for debugging purposes:

📬 Mails

I have had in mind to switch to a more privacy-respectful email provider but I never actually did it and kept using my {insert any mainstream provider here} address.

But then I read this: https://poolp.org/posts/2019-08-30/you-should-not-run-your-mail-server-because-mail-is-hard — a blog post from Gilles Chehade who is an OpenBSD contributor and creator of OpenSMTPD, and it convinced me right away that I had to try to set up my own mail server.

And so I did.

I grabbed a supplementary IPv4 address for my mail server that I attached to the same OVH cloud instance, and created two containers:

• one for OpenSMTPd (SMTP service)
• one for Dovecot (IMAP service)

That would provide me with the bare minimum for my mail server to work. I followed all the security best practices and made sure that I had a clean, standard installation, and just as described in the blog post that I linked above, it worked quite well from the beginning !

I spent quite a lot of time improving the two Dockerfiles but my mail server has been running flawlessly for about 8 months now, never losing a single mail — and generally being more responsive than the other services that I used.

Dovecot, moreover, is robust and standard software and I can access my mails from any Apple or Android phone, and any mac OS computer, which is all I need (I guess it works well on Linux and Windows, too).

Next steps

I'm updating the containers quite regularly, and I'm always looking out for new types of services to self-host.

To date, I have not had another need but I might install a private git server sometime soon to host all my code.

When installing a new VPS for a new project, I had to dive in my previous nginx boilerplate configuration files to create a new one, slightly different, to accomodate for my various needs on the project.

While doing so, I remembered how long and tedious these configuration files were — let's see for example a simple reverse proxy for a simple blog like Ghost, with redirects and SSL (this is not optimized - just a quick snippet):

server {
    listen 80;
    server_name www.myghost.blog myghost.blog;
    return 301 https://www.myghost.blog$request_uri;
}
server {
    server_name myghost.blog;

    ssl_certificate /etc/letsencrypt/live/myghost.blog/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/myghost.blog/privkey.pem;

    return 301 https://www.myghost.blog$request_uri;
}
server {
    server_name www.myghost.blog;

    client_max_body_size 10m;

    listen 443 ssl;

    ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
    ssl_prefer_server_ciphers on;
    ssl_dhparam /etc/ssl/certs/dhparam.pem;
    ssl_ciphers 'ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA';
    ssl_session_timeout 1d;
    ssl_session_cache shared:SSL:50m;
    ssl_stapling on;
    ssl_stapling_verify on;
    add_header Strict-Transport-Security max-age=15768000;

    ssl_certificate /etc/letsencrypt/live/myghost.blog/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/myghost.blog/privkey.pem;

    location / {
        proxy_set_header   X-Real-IP $remote_addr;
        proxy_set_header   Host      $http_host;
        proxy_set_header   X-Forwarded-Proto https;
        proxy_pass         http://127.0.0.1:4369;
    }
}

This is a lot (and some parts are missing here still), and it's mainly here to compensate the facts that the defaults are not suited.

I decided to look for an alternative web server, that would be production-grade, but that would be much more easy to configure while enforcing safe standards by default.

... and I came across Caddy.

The same configuration in Caddy looks like this :

myghost.blog, www.myghost.blog {
    encode zstd gzip
    reverse_proxy localhost:4369 {
        header_up X-Forwarded-Proto {scheme}
    }
    header {
        Strict-Transport-Security max-age=31536000;
        X-Content-Type-Options nosniff
        Referrer-Policy no-referrer-when-downgrade
    }
}

And that's it ! No supplementary configuration, no need to indicate the SSL certificates and the sensible ciphers or the stapling (Caddy provision and renew them automatically), and every default is well-chosen and does not need tweaking (in this case).

This gives an A+ in SSL labs with basically no effort :

Performance-wise, I find it on par with what nginx does, delivering roughly the same amount of requests per second. I haven't extensively tested it yet though — your mileage may vary.

If you read through the docs (https://caddyserver.com/docs/), you can see all the options that are available for all the directives. It's pretty modular and you can change pretty much every aspect of the server.

But in most cases, you don't need too.

The need for sensible defaults

In fact, what this boils down to is that the extensive configuration in nginx is very powerful, but it lacks consistent sensible default values for all of them.

I find that Caddy is quite opinionated in this regard, but this is quite a good thing in fact; when you read the Caddy configuration, you instantly get, in 10 lines, the full grasp of what it does.

You don't even need to scroll to see the full configuration file.

A lot of expert, production-grade software would clearly benefit from this approach. When using sensible (and probably opinionated) default values :

1. the configuration is easier, simpler, shorter,
2. it promotes good practices,
3. you can configure it right, even if you don't know all the configuration specifics.

It has cons, too. Specifically, you can lose touch with all the under-the-hood technicalities of the software you use, where all the defaults are hidden at first sight.

But I think that this is a false problem; The majority of nginx configuration files I encountered along my way were mere patchworks of copy/paste of StackOverflow answers. So a long, convoluted, very specific configuration does not necessarily mean that the person who uses it does understand all the ins and outs of it. There's no relation between your expertise and the use of default values, if they are well-chosen.

And "opinionated" means that different best practices exist. In the case of Caddy vs. Nginx, I think we can all agree that enforcing HTTPS, and setting everything so that your website gets an A+ in the ssllabs test is enough a consensus to be declared a universal best practice. But it may depend on the software.

Bonus: Some more advantages of Caddy

• Automatic certificates and renewal via LE
• Static binary, no dependencies
• Zero-downtime configuration reload

Give it a try !

Alright, this is going to be quite a long article, about a subject that I have grown particularely fond of lately : online privacy.

We're going to deal with self-hosting too, a lot 😇. And we'll get technical, don't worry.

In a nutshell, this article is my personal take on what I value when I say online privacy, and how I decided to try to mitigate the risks by using open-source tools to create my own public cloud ☁️.

Your mileage may vary on the definition of privacy, on the risks taken, and on the legitimity of the tools I will describe and implement, but I think this article can give you a good head start if you're looking into this kind of stuff at the moment.

⎈ Summary

    
  1. What is the problem, to begin with?
  2. A personal cloud?
  3. Technical implications
    a. Self-hosted to the extreme ?
    b. Docker / Compose
    c. Risks and mitigations
  4. Solutions
    a. Drive
    b. Bookmarks
    c. Passwords
    d. Tasks / Boards
    e. Notes
    f. Calendar / Contacts
    g. Raw syncing
  5. Go
    a. Get docker and compose up and running
    b. Retrieve the repo
    c. Create your cloud infrastructure
    d. Add or remove services
    e. Tweak the configuration
    f. Build the custom images
  6. Launch
  7. Final words
    a. A note on E2E encryption
    b. Backups
    c. Donate and contribute

1. 💁🏻 What is the problem, to begin with?

Privacy is becoming a mainstream subject right now, and that is good. The immense amount of data that one generates while using various internet services like social networks, online services or softwares is now more and more a personal stress factor; for techies of course but also more and more for the profanes.

Because of data breaches, first. Not one day goes by without its new security breach (Equifax, Option Way, CircleCI, CapitalOne, Movie Pass, LinkedIn, Twitter ...) exposing thousands of people's personal data, including, in the worst cases, passwords in plain text.

Passwords in plain text ? We're in 2019 for Christ's sake — this is just mind-boggling that platforms or saas providers that have thousands or millions accounts to manage do not have the slightest knowledge of what basic security / encryption / good practice is.

You can also check the 453+ pages of https://plaintextoffenders.com to check if a service you want to use has questionable practices.

But also and maybe more importantly (and more concerning to be honest) because of the marketing and targetting mayhem this data is subject to. What does Google do with my emails ? Does it read my Drive documents ? And these Facebook searches ? Why do I keep seeing ads that are directly related to my previous internet searches ? Why do I have to load more than 80ko of trackers on each and every website I browse ? etc...

This is not a problem per se, but it could well become one when the companies / countries that control these targetting and marketing efforts decide or are forced to hand over our data to malicious third parties.

I don't want to be too alarming, but I'm myself concerned about the data I create and how it may or may not be used by the providers I trust it with.

That's why I went (and am still) on a quest to mitigate this risk.

I have quite a bit of history with what I would call "incumbent" service providers, since I've been mainly active on the Internet at the time these providers were gaining a lot of traction: I have Gmail emails and associated Google accounts. I use it for contacts and calendar, too. I have a Dropbox account. I use Trello for todos and whatnots. I use various sync services for my browsers (Firefox, Safari) ... the list goes on.

Now I'm not saying that these providers do a bad job regarding privacy, but regarding all the recent heat a few of them (all of them?) have received, I no longer trust any "big-enough" actor with my personal data.

Now that means that I could find and use smaller, local providers, but how can I be sure that they are well-funded and can provide a full-featured service in the long-run (I'm happy to pay for a service of course, but even then)? And what if they get bought later on ? How can I assess their security measures and processes ? At least I'm sure Google has a team dedicated (even many teams) to secure and backup its servers and put them back online in a snap.

So a small provider is not a really satisfying answer.

Of course I could revoke all these accounts and do everything offline; but we're in the 21th century and as a lot of people out there, I'm not ready to trade the practicality of online tools such as mail, calendar, contacts, file repository, readily available on any device anywhere, for an extreme privacy posture.

TL;DR: Richard Stallman is right on many points, but his position is generally too absolute and radical, and not compatible with a fair and reasonable use of today's tools (I think).

2. 🌥 A personal cloud?

So a personal cloud could be a good solution, in this regard. A small provider, that happens to be yourself. Not perfect, but better than other options.

To me, here are some pros and cons of this solution:

• The data is on your servers / you own your data for real
• You control the applications you put on these servers
• You can modify them if needed, to suit your needs
• You can choose solutions that are open-source and have auditable security
• You can find feature-rich solutions that are the alsmost exact equivalent of Google Drive, Trello, etc ...

• You are the admin of your cloud apps. This means you are the technical admin of your cloud apps.
• In case of a problem, it's up to you to fix it
• It needs a bit of work to set up correctly
• It needs a bit of work to maintain and to update

So yes, you need to get a bit under the hood to be able to create your own cloud, that's the culprit. But the amount of work needed is not overwhelming, and we're going to use standard technologies here, with extensive documentation, communities, and easy-to-access help on all the different stacks.

You've got this.

What exactly does a personal cloud cover in terms of functionality ?

It depends. You might need a feed reader, a webmail, a calendar, and a file storage solution, or you might just need a todo list and a task manager, with a VPN, ... that's up to you to decide.

In this article, I will focus on some standard needs (that happen to be my needs 😁) : a drive, a calendar and contacts, a bookmarks manager, a password manager, a notes app, a kanban-board (à la Trello) and a sync/backup solution.

This should cover a wide range of needs for a personal user, and could very well work for small associations or companies too, that want to go the open-source / privacy-focused way (and that have a technical contact that can manage that of course).

3. 🛠 Technical implications

Ok, now let's dive into the how.

a. Self-hosted to the extreme ?

The first obvious technical issue we're facing is obviously that this cloud will need to be hosted somewhere on the Internet.

Two options are available : either you push the paradigm to the extreme and use a real, physical server that you physically own, and that will sit behind your home broadband connexion, or you have to rely on the "cloud", that is, a third party that will provide you with a virtual or physical server, or even an abstracted infrastructure you can deploy things on.

If you are really worried about the whereabouts of your data, your own server is the straightforward solution; personally, I think it is over-complicated and has various drawbacks :

• Your home IP might change
• Your home ADSL / fibre may have connectivity problems from time to time
• You could have a power cut / brownout etc
• You need to maintain hardware
• Your cat could trip over the network cable of the server

(On a side note, you could also have your own server, but on someone else's Internet, like what Jeff Atwood did for Discourse — but I think that is not ideal, since apart from the Internet connexion, you still have the drawbacks of having to manage hardware, i.e. replace disks, etc)

So putting your infrastructure somewhere else, where it likely will be monitored round the clock, seems reasonable to me. You need to assess that your provider is somewhat respectful of what you do on your instances, but you can mitigate that quite easily too.

I'm already using OVH and I'm pretty happy with their pricings so I would recommand them, but they are a lot of other cloud infrastructure providers out there : Amazon of course, Microsoft Azure, Oracle, Vultr, Rackspace, etc ...

For the setup that I'm going to describe here, you should expect to pay about 20€ per month at OVH, which is very reasonable.

b. How is it going to work exactly ?

The plan is the following :

First, we'll create a relatively powerful instance on your chosen infrastructure provider. This will serve as a host for an automation / containerization tool (see below), that will do the heavy lifting and provide isolation between services.

You could as well spin off an instance (aka virtual server) for each service of your cloud that you need, but this is not ideal since most of them will sit doing nothing most of the time (you are the sole user of your cloud); and moreover, you end up with a lot of instances to take care of and to pay for.

A powerful host, containerized, should be a better use of the resources.

Next, we are going to create a container for every service that we wish to use : one for our drive, one for our calendars, etc. We'll see that in details later on.

Of course, we'll need to expose all these services to the Internet through the host, while limiting the attack surface. Our containerization tool will help us do that.

Finally, if you want to have a clean namespaced cloud, you will certainly need to create some DNS entries to match for all your container services / ports on the host.

b. Docker / Compose

Alright, now, with this plan, the obvious simple containerization tool that comes to (my) mind is Docker.

I'm saying obvious here because it's relatively stable, has a lot of documentation, has a great community and a lot of stackoverflow questions answers.

We're clearly not going to create a handful of cloud apps from scratch with our bare hands; There are fantastic automation tools out there that are just right for the job; Docker is my choice but feel free to use the one you like or are comfortable with.

Docker itself would be sufficient to instantiate all we need but it would be tedious — we're going to rely heavily on docker-compose to do the job.

In fact, our entire configuration for our cloud apps will likely reside in a single docker-compose configuration file.

c. Risks and mitigations

So what's the culprit ?

Our infrastructure provider have access to our instance and storage

Yes, it does. I see two risks here :

1. what if the provider gets p*wned ?
2. what if the provider decides to use this data (sell it to third-parties, etc) ?

Well, point 1 is legit : if someone infiltrates their DC or network and gets your block storage, it will have access to your data. Point 2 is irrelevant; your data is a big blob of disk storage, and is not valuable as-is. You really should be someone of interest for any third-party to try to buy your raw data and try to extract something from it. The only data your provider could sell is your account on their platform with your billing info / personal info, that's it.

For point 1 though, it's up to you to store your data securely, for instance by choosing apps that store your data encrypted. In the apps that we're going to install here, some do and some don't. But for the important data (passwords, notes for example), I chose ones that do.

What if somehow steals my database ?

That could happen if you happen to have a misconfigured service or a kind of exploit that has not been patched yet.

Same than the point 1 above : if your data is encrypted, who cares ?

If my infrastructure provider fails ?

Well, that could happen too. If it's temporary, well, you just had a downtime. If the physical server on which your data (apps or storage) was stored crashes, then, you lost it all.

It's not as likely as you would think since the "cloud" is not just someone else's computer; it's a bit more complicated than that and involves clusters of cpu and storage and a few layers of software that are generally running on cheap hardware that is expected to fail at some point. "Hardware failure is the norm" (it's Hadoop's motto); hence your data is not really just spinning on one disk somewhere. And is less prone to hardware failures.

But anyway, the providers generally have several ways to deal with that :

• data is replicated on their side: in case of a crash, they will restore your instance and data from hot backups, or from cold backups they have at another location
• they provide tools to make block storage snapshots on your own: either via a web interface or an API, or in an automated fashion
  - you can also make backups (a little different from snapshots) so you can restore to any previous state, especially for your instance

What if one of my container fails / stops / crashes ?

Well, it's generally not that bad. Maybe a glitch. As we're going to see, your apps are going to be separated from your data, thus it's possible to restart / stop / rebuild a container from scratch without losing anything.

If you're away on vacation while it crashes, and you don't have access to a computer to restart it, chances are you don't need your cloud anyway ;)

4. 🖥 Apps

Ok enough talking (sorry for that), let's get to the point: what apps are we going to use ?

This list and my choices are totally personal — if you want more (a lot more) options, check out Kickball's github repository that has an extensive list, for all kind of needs.

a. Drive

Ok let's start with the obvious solution you need: storing documents in the cloud.

I've settled on Cozy (https://cozy.io); it's a French solution. It's still a bit young and sometimes buggy, but their interface is very intuitive, very clear, and it does the job well. They have a mobile app and a desktop app that are getting updated almost on a daily basis.

They take security and privacy quite seriously, even on their hosted plan (very reasonably priced by the way); Their CPO was Tristan Nitot (now heading the Qwant search engine), founder and former president of Mozilla Europe, and they are likely to take over some cloud solutions in the French government and administrations.

Unfortunately their docs for the self-hosted version are not on par yet, but I've already done that work for you with the dockerfile so you won't have to deal with outdated instructions.

The beauty of Cozy is its simple set of features — it's clearly aimed at single-users and not at companies (at least in my opinion); you can still share folders quite easily with people nevertheless, which is very practical.

Extra modules are available : photos, contacts, banks to name a few.. and they share the same "simple, functional" philosophy than the drive counterpart.

Alternatives

• ownCloud : I've been using it in a work context and it's pretty powerful, with a lot of extensions and modules. The desktop clients are robust, and they have a strong community if you run into issues.
• Seafile : quite powerful too, the team is Chinese, so you might want to check that it complies with your needs and principles. Otherwise, it has everything you need.
• NextCloud : I haven't tested this one, but it's the go-to solution that a lot of experts advocate. Might be worth it even if a bit complicated I think for a single-user solution.

b. Bookmarks

I use bookmarks a lot when researching the web, so I needed a solution to store them and access them anywhere easily.

I obviously didn't want to use any vendor solution such as Firefox Sync or iCloud, to be able to use any browser, and of course so I could self-host this too.

X-browser sync (https://www.xbrowsersync.org/) seemed to me the perfect solution : simple, encrypted, robust.

And it has Docker instructions, too;

Alternatives

• If you use Firefox, you could spin off your own Firefox sync server that works with the Browser

c. Password manager

This is a must-have for your own private cloud.

I've looked into strong and robust solutions, and I found Passbolt to be a very good contender.

It uses GnuPG, is open-source, has a clear roadmap, and its development is taking place in the heart of Europe (Luxemburg).

Moreover, they answer quite quickly (on Github at least) which is very nice.

They have a plugin for Firefox / Chrome / Edge (with the Chromium engine), but not Safari yet.

Alternatives

• Bitwarden looks great but it threw me a bit off that one needs to request an id and key (https://bitwarden.com/host/) to self-host. It seems quite powerful and has a strong documentation.

d. Tasks / Boards

Everybody loves a good Kanban board to sort things out. It deserves a space in your private cloud;

The only worthy Trello opponent I found is Wekan.

It looks very much like Trello, you can import boards (but beware — if the boards are too big it will fail silently!), and can tweaks settings for every part of the software.

It has a docker image already, which is a plus.

Alternatives

• TaskBoard — nice, but was lacking the "Trello" touch and feel. Also, on a technical side, it was not as easy to install properly and maintain
• Restyaboard — Same same : very interesting but the interface did not suit me at all.

e. Notes

This was an easy choice. There is one player out there that is a really ahead of the competition : Standard Notes.

It's simple, encrypted, it has a robust set of features and you can add extensions that are simple JS apps (there are a few already available, such as theming, markdown editors, spreadsheets ...).

Yet the apps are beautifully designed and are available on MacOS, Android, iOS ... and there is a web app too, that you can use even if you self-host your instance.

I use it daily and it's a pleasure to take notes with it.

Alternatives

To be really honest, there are a lot, but none of them just come close to SN. Check the 'others' section below to have a list (not curated by me)

f. Calendar / Contacts

The de facto standard for calendar and contacts exchange on the Internet is CalDav/CardDav, but it's a complicated protocol / standard.

There are a few options out there and I decided to go with one that was on a stack I'm familiar with : PHP and MySQL, so I could easily tweak the code if needed.

So here comes Baikal.

Baikal has adockerized installation, is pretty easy to install and configure, and has multi-users support. It works out-of-the-box with all my macOS apps (calendar, contacts) and also with all my Android apps (see below).

There is a web dashboard that gives you a lot of information at a glance :

The only problem I'm still trying to mitigate is that answers to invitations do not change the invitee RSVP status in my calendar: for instance if I create an event and invite a few people on gmail adresses, and if they respond to the invitation, I will receive a mail, but my calendar event will not be updated.

This maybe a configuration problem, but I think it's a deeper issue; I'll try to investigate later on to see if this can be fixed - but I have to up my knowledge of the CalDav standard first...

Oh, you're on Android ?

Unfortunately getting a CalDav / CardDav server to work on a vanilla Android phone is tricky. I would recommend using the very nice DavX application. It's open-source too (https://gitlab.com/bitfireAT/davx5-ose) but at 4€, it's worth just paying for it.

Alternatives

• SoGo — is more complete and extensive, but I needed something simpler since I am planning to use it as a single user
• Radicale — a more roots solution, but not updated recently
• Calendar Server (Apple) — seems very robust and well-thought, but I'm wary of the amount of effort Apple have put in the documentation and the OS community on this

g. Raw syncing

So we already have a drive solution, with a web interface so we can access our documents anywhere anytime with just an Internet connexion.

But not all our documents need to be accessed this way.

You could also have a need to sync a large collection of photos / files to the cloud as a kind of backup, or to be able to retrieve it on another machine (work / home for instance).

I chose Syncthing for that.

Syncthing is purely decentralized, so you can easily add "nodes" to improve your data resilience. It is easily dockerized.

It has a neat web interface to see how things get replicated across your networks of devices, but it's really a "fire and forget" type of software, which is pretty convenient.

Alternatives

• Seafile, ownCloud and Nextcloud (that we presented above) are solutions that could work for this too

h. Others ?

Maybe you need another type of web app that I didn't include here — rejoice! As there is very surely an open-source and easily dockerizable software out there waiting for you.

As a start, Edward has compiled a very comprehensive list on Github here : https://github.com/Kickball/awesome-selfhosted so you can find your dream package to install in your cloud.

Some of them already have a dockerfile / Docker hub repository, some don't but it's generally not a big deal to create one.

5. ▶️ Go!

Now's the technical part at last !

We'll need a few tools to do this.

On your local machine, you'll need docker, docker-machine and docker-compose. At some point, you'll need to clone the repository where I have put all the configuration / scripts, so git is kind of mandatory.

You won't need to install a VM on your local machine since we'll use Docker to remote control another Docker instance (that will be in the cloud). So hopefully you don't need to install Virtualbox or Hyperkit. Nevertheless, if you want to test your config locally when making changes, I'd recommend that you have a minimal VM locally.

a. Get docker and compose up and running on your local machine

I won't extend myself too much on this. There are many tutorials on the web if you are not an expert, and the Docker documentation is very comprehensive.

• For macOS : https://docs.docker.com/docker-for-mac/install/
• For Windows : https://docs.docker.com/docker-for-windows/

b. Retrieve the repo

Head to https://github.com/tchapi/own-private-cloud and clone it locally.

The repository uses submodules, don't forget to init them.

git clone https://github.com/tchapi/own-private-cloud
git submodule update --init
cd infra-public

I have tried to organize it properly so it is understandable. The main part is of course the top-level docker-compose.yml file.

In the build folder reside all the needed Dockerfiles.
All the configuration files (that will ultimately be copied to the containers) live in configurations.
And finally, locally-executed scripts are in the scripts folder.

c. Create your cloud infrastructure

In this example I will use OVH Public Cloud as my infrastructure provider.

For my cloud, I use the smallest possible production instance, a b2-7 (see screenshot below for specs), and it's largely sufficient in terms of CPU and RAM, but you can adapt accordingly.

Disclaimer : these instructions / tutorial only works with OVH Public Cloud. It should be relatively straightforward to adapt it to another cloud provider since most of the steps are just Docker cli commands ..

0. Signup for a public cloud account

Obviously.

1. Create a new instance

This will be your main machine and Docker host.

You could be tempted to use the web interface like below but don't :

Why ? Because if you do so, you won't be able to manage your Docker host with your local docker-machine utility, since it will have no record of having created it.

You're better doing this with the command line.

But before, we need to do two things :

First, you need to retrieve your credentials for the underlying OpenStack service.

To do so, go to the Users panel, create a user if you haven't done so yet, and then select "Download OpenStack's RC file" from the menu :

Select the datacenter you want to use primarily (should be the same that your instance will live in) — here, I chose GRA5 — and check"V3", then click Download :

The resulting openrc.sh file will need to be executed whenever you want to use Docker to control your machines, so the correct environment variables are set beforehand.

Second, you need to add at least one SSH key in your account to that you can login to your machine (and so that Docker can).

In the SSH Keys menu, add a new key, and give it a name (for instance, HOME) :

(PS : You may need to create the SSH key in the Horizon web interface too — It's OpenStack's own web GUI that is accessible in the menu on the left)

Once you know the model you want to create (like b2-7), you're ready to create the instance via a cli. On your local machine, source openrc.sh:

source openrc.sh

Then, use docker-machine to create the instance :

docker-machine create -d openstack \
  --openstack-flavor-name="b2-7" \
  --openstack-region="GRA5" \
  --openstack-image-name="Debian 9" \
  --openstack-net-name="Ext-Net" \
  --openstack-ssh-user="debian" \
  --openstack-keypair-name="HOME" \
  --openstack-private-key-file="path_to/.ssh/id_rsa" \
  default

Somes notes :

• OVH uses OpenStack as its backend for its cloud, so we use the openstack driver to talk to it
• use the same datacenter code that your RC file
• use the same SSH key that the one you uploaded
• name the machine default — it's not mandatory, but if you do you won't have to type the name of your machine when invocating docker-machine

The rest should be straightforward.

Wait a few seconds and 💥 ! You have a docker host living on a Debian 9 instance in the cloud.

You can check that everything is correct by going to the Instances web interface — you default instance should be there, with your brand new IP (make a note of it !) :

2. Configure your instance

Now, we need to tweak the host a little bit before we can start working with it.

Entropy

For a cloud instance to be able to generate entropy at a correct rate, we need the haveged package — this will be necessary for the password manager for instance :

docker-machine ssh default 'sudo apt update && sudo apt install -y -f haveged'

Paths and mounted volumes

If you want to do things correctly, you want to store your data (databases, files, etc) not directly on the instance, but rather on an attached volume.

Why? Because that will allow you to :

• Rebuild your instance without losing any data
• Backup your data and your data only when you want
• Generally speaking, "separate concerns" — your data and your apps should not live in the same space

How to do this ? It's quite easy, you need to create a block storage device (or several, in my case), that you will then mount on your instance.

To create them, simply head to the Storage menu, and create two blocks. Here, one named "databases" that will store all databases data files (MySQL, mongo, couch, ...), and one names "files" that will store all "real" files (the cozy cloud files or syncthing files for instance).

Attach them to your default instance via the web interface too; they will be made available as disks.

Once it's done, you need to initialize these disks and mount them. Behold (using fdisk is beyond the scope of this article — here) :

In the following lines, I assume that the databases volume is at /dev/sdb and the files volume at /dev/sdc.

The databases volume :

docker-machine ssh default 'sudo fdisk /dev/sdb # n, p, w'
docker-machine ssh default 'sudo mkfs.ext4 /dev/sdb1'
docker-machine ssh default 'sudo mkdir /mnt/databases && sudo mount /dev/sdb1 /mnt/databases'
docker-machine ssh default 'sudo mkdir /mnt/databases/mysql /mnt/databases/couch /mnt/databases/mongo'

The files volume :

docker-machine ssh default 'sudo fdisk /dev/sdc # n, p, w'
docker-machine ssh default 'sudo mkfs.ext4 /dev/sdc1'
docker-machine ssh default 'sudo mkdir /mnt/files && sudo mount /dev/sdc1 /mnt/files'
docker-machine ssh default 'sudo mkdir /mnt/files/cozy /mnt/files/sync'

3. Test that Docker works well on the host

Now that your machine is up to date, let's try to see if Docker works correctly on the host.

First, we need to tell your local Docker instance what to control; For that, Docker-machine gives us the correct environment variables that we need to use

docker-machine env default

To eval those vars automatically, just do :

eval $(docker-machine env default)

Once it's done, your local docker utility should directly control your remote Docker host. Try :

docker info

... to see if it works ! This should return information from your instance (operating system should be Debian 9, etc)

If it's all good, you're set up to create containers on your host, from the confort of your local terminal 🎉.

You can now use all of Docker commands to manage your host and containers.

d. Add or remove services

Now, before you deploy, you might not need all of these services I have described.

Here is the dependencies graph so you can change the docker-compose file easily — each block is a container:

Note that if you remove a service (let's say for instance, Calendar), you must remove its network from the networks list, and from the reverse-proxy networks too.

The certbot container is standalone, but uses a shared volume with the reverse-proxy container, so that the certificates are available for the two of them.

The only container that is exposed to the web is the reverse-proxy (through port 80 of the host). This container depends on all others just because it needs to know the backend for all virtual hosts.

For instance, in the nginx config of this container (just an excerpt) :

upstream docker-passbolt {
    server passbolt;
}

...

server {
    listen 443 ssl http2;
    listen [::]:443 ssl http2;
    server_name passbolt.mydomain.com;

    ssl_certificate /path/to/fullchain.pem;
    ssl_certificate_key /path/to/privkey.pem;

    location / {
        proxy_pass  http://docker-passbolt;
        proxy_redirect     off;
        proxy_set_header   Host $host;
    }
}

If the passbolt container is not up when nginx starts, it will refuse to start because it can't resolve the upstream.

I could have done one file per virtual host, or a different setup that would allow the reverse-proxy container to not depend on the other containers but I guess modifying a nginx configuration file is relatively easy.

e. Tweak the configuration

You will notice a .env.dist file in the root of the repository. You must duplicate this file to a .env file before building or creating containers.

This .env file is very important : it will allow you to tweak the configuration for your entire cloud, namely :

• the reverse DNS for each service
• the password for each database, service, web interface
• your mail configuration
• some other container-specific configuration

This file is a bit cumbersome to fill in, and it's due to the fact that you must duplicate information in different parts because different containers do not use the same env vars.

This would be easy if the file was a regular env file (in a shell way of speaking), but it's not, and Docker doesn't like variable replacement in this file, so 🤷.

Beware of shell expansions too, if you're using special characters in the passwords. I've put an example in the dist file.

f. Build the custom images

The docker-compose file is based upon official images and custom ones that I modified. Before launching, and to check that your configuration tweaks did not break anything, it's a good idea to build them.

Beforehand though, you need to build the configuration files. Configuration files are copied onto the various containers and I created them so that the environment variables are written directly in the configuration, to make the whole repository agnostic of my own implementation.

I use a very simple templating system where configuration files include bash-style env vars, like so (see the $NOTES_DOMAIN var):

{
  "url": "https://$NOTES_DOMAIN/extensions/secure-spreadsheets/dist/index.html",
  "download_url": "https://github.com/sn-extensions/secure-spreadsheets/archive/1.3.2.zip",
  "latest_url": "https://$NOTES_DOMAIN/extensions/secure-spreadsheets.json"
}

These vars are simply replaced when playing the script :

./scripts/build-configuration-files.sh

The only culprit of this system is that you need to escape $'s with \$

Once it's done, you can build the different custom containers :

docker-compose build

Configuring things before launch

Some images need to be prepared before they can be started, or else they will fail and quit.

This is especially true for the reverse-proxy that needs SSL certificates (we only serve https, you should too).

To set dummy SSL certificates, then launch nginx and retrieve the real certificates from Let's encrypt, you should run this script :

./scripts/certbot/init-letsencrypt.sh

It relies heavily on the configuration of the domains that is in the .env file, so make sure that everything is ok in there before running it.

Of course, before that, you must have created the correct DNS entries pointing to your instance (the IP you have written early on)

As for the Cozy container, you must create what they call an instance before being able to connect to your cloud. I guess that this is related to multi-users installations. Just run the below script once :

./scripts/cozy/init-cozycloud.sh

NB : you can change the quota (10GB) in the script directly if you need more, but then be careful to not put a quote above your real disk capacity (it's part of the files block storage from before, that we created via the OVH web interface).

A note about these custom images and scripts

So as I was saying, I use mainly custom images derived from the official ones to add my own tweaks. Here are some details of what I did for those of interest.

Standard notes

I've included a few extensions I find useful in the container.

To add an extension to your desktop Standard Notes app, go to "Extensions" in the bottom bar, click "Import extension" and paste the link to the JSON description file of the extension you want :

On your custom domain : https://notes.mydomain.com/extensions/

• Advanced markdown editor : advanced-markdown-editor.json
• Plus editor : plus-editor.json
• Secure spreadsheets : secure-spreadsheets.json
• Simple task editor : simple-task-editor.json
• Autocomplete tags : autocomplete-tags.json

NB : Side note on this; Standard notes says that the extensions are "public source" but not "open source", which I don't quite understand fully, to be honest (see here, where there is no clear answer to "Can I self host existing extensions?"). The source code is published on Github, without license — In my repository I only link to it via submodule, which I think is pretty much in line with Github's Terms, but if you're from Standard Notes and want me to remove this, just contact me and I will abide.

Nginx — reverse proxy

I've created a quite reasonable configuration file by following the best practices for SSL parameters and HTTPS.

All services are only served in HTTPS and if possible, via HTTP/2.

All insecure requests are redirected to their secure counterpart, except for the certificate challenges.

With all these settings, all the front websites should achieve at least an A rating on SSL Labs

Baikal — calendars and contacts

The official Dockerfile was not really up to date and was including too many things like postfix for instance, so I revamped it quite a lot.

Following, binfalse's article I also used ssmtp in lieu of sendmail to be able to use an external SMTP to route the emails.

Check out the Dockerfile to see exactly all the steps

xBrowser sync

The configuration I created allows only one sync (this is a personal cloud), and by default, does not allow new syncs.

That means that you need to create the container with a different configuration allowing new syncs, create your sync id (via the browser extension), and then if you wish, recreate the container with the initial configuration (not allowing syncs). This is just a security measure.

Passbolt

By default on the new versions, you are disconnected every 24 minutes because of the way the sessions are managed by PHP.

So I extended the garbage collector session lifetime to allow to have 3 days without having to reconnect.

(see this Github issue for more context)

6. 🚀 Launch !

It's time.

docker-compose up -d

NB : -d is to run in daemon mode.

After having done that, you still need to :

Init the Baikal instance if needed (if the tables do not already exist in the database)

./scripts/baikal/init-mysql-tables.sh

Create the Passbolt admin user

./scripts/passbolt/init-admin-user.sh

It will give you a url that you need to go to to setup your account, your key, etc ...

You now should have secured working services behind your custom subdomains or domains. Congrats 🎉 !

7. 💭 Final words

A note on E2E encryption

Back to privacy. As you have seen, not all solutions I use here offer end-to-end encryption.

Only the notes, passwords and bookmarks are fully E2E encrypted.

The other solutions often provide transport encryption (like Syncthing) and of course we have https, but if someone gets hold of the table containing your Wekan boards and cards, they will be able to read it for sure.

So it's up to you to decide on which services you want to create data. For now, I'm pretty happy with the non-encrypted drive, sync and kanban boards because my main focus is privacy, and not a total security from a hacker that would explicitly decide to extract my data and files.

If you're concerned about this, Cryptpad apparently has a neat solution for an encrypted Drive + cloud apps, with a dockerfile on their Github. I haven't tested it though.

As for the calendar / contacts, I haven't found an open-source encrypted calendar solution yet. Some providers offer it as a service (for instance, tutanota), but that seems to be quite a niche.

I guess the best solution here would be to dive into the Baikal source code, fork it and add encryption on persistence. Feasible, but could be quite a challenge (see this).

NB : The CalDav / CardDav protocols are plain-text in essence, hence the importance of https.

If you find good encrypted alternatives, do not hesitate to put them in the comments below this article.

Backup

It's more probable that your storage will fail before someone tries to steal your data. So backup is an important part of your private personal cloud.

I touched on the subject before, but to recap, it's a good idea to backup your block storage from time to time.

On OVH (my provider of choice), it's relatively easy with the web interface — it's called Volume snapshot : just select your storage, click "create a snapshot" and you're done.

NB : Of course, you can also do this automatically with the OpenStack API / the Horizon interface, but this is out of the scope of this article.

As for the instance, I think it's less important since your instance is basically a configuration file, so if it crashes, you can just restart a new container with a single command line, without losing any data (only downtime).

💰 Donate or contribute

Ok so now you have you own private cloud and all these apps working well. Time to thank all the people that made this possible!

I would encourage to donate to all these projects, for instance the amount that you would have paid for a month or two of services (for those that provide a cloud-based solution), or a small reasonable fee (like 10€ ?) for the other.

We've worked with free software here, free as in speech. They're also free (as in beer) to download and self-host, but if we want the maintainers of these tools to continue to make releases, patch bugs and add features, we need to acknowledge that these people need to earn a living, too. A little gesture is always welcome.

You can also get involved with the development of these tools if you have the ability and time to do so : by writing clear and precise bug reports, by submitting pull requests, or helping in any other way possible. That's the beauty of open-source.

Extra literature

Some articles I found while researching my self-hosting mania with Docker:

• Docker best practices : https://blog.docker.com/2019/07/intro-guide-to-dockerfile-best-practices/

• Nginx Reverse proxy : https://www.thepolyglotdeveloper.com/2017/03/nginx-reverse-proxy-containerized-docker-applications/

• Lets Encrypt with Docker : https://devsidestory.com/lets-encrypt-with-docker/

• Lets Encrypt with Docker (alt) : https://medium.com/@pentacent/nginx-and-lets-encrypt-with-docker-in-less-than-5-minutes-b4b8a60d3a71

• Shell command / Entrypoint in Docker : https://stackoverflow.com/questions/41512237/how-to-execute-a-shell-command-before-the-entrypoint-via-the-dockerfile

• Ignore files for Cozy drive : https://github.com/cozy-labs/cozy-desktop/blob/master/doc/usage/ignore_files.md

• About privacy and DNS, a wonderful talk : https://www.youtube.com/watch?v=pjin3nv8jAo

If you want to go further and also self-host your emails, Gilles Chehade (poolp) made a very nice article about this very topic: https://poolp.org/posts/2019-09-14/setting-up-a-mail-server-with-opensmtpd-dovecot-and-rspamd/

The μLCD 32PT is a nice TFT display from the Australians at 4DSystems. I have found two in an old box - perfect for Recycling Thursday !

The legacy documentation is here (GFX flavour).

So it's time to spin up a Windows VM and try to make something useful out of this !

The μLCD 32PT has two modes: SGC and GFX. From 4D directly : "The architecture of the base PICASO chip is such that it can be reconfigured to operate in 2 distinctively different ways. To configure the device, a PmmC (Personality Module Micro-Code) is downloaded via its serial port. There are 2 types of PmmC available for PICASO."

SGC (Slave Graphics Controller)

In this mode, the module is 'ready to go' by simply connecting it to the serial port of your favourite micro-controller, and sending serial commands to it.

GFX (Stand-Alone Graphics Controller)

In this mode, the module is then like a microprocessor which you program, using the 4DGL language (very similar to C), to control the internal graphics and external interfaces. It does not need an external microprocessor, just power.

We know SGC works (done that before), so let's try GFX (standalone) and see if we can do something nice and fun with that.

Let's create a flappy-bird-like game !

Setting the correct mode

Since the SGC mode is the default mode, we need to reprogram the microcode to use the GFX mode.

To do this, we will use the PmmC Loader (Windows application) and obviously, we'll need a programming cable like this one from 4DSystems.

The GFX microcode can be downloaded directly from the legacy product page of the uLCD (GFX mode), on the Downloads tab (you're looking for the uLCD-32PT-I-GFX-R32.pmmc file).

Just in case, here are the direct links to the two microcode flavours are :

• SGC : uLCD-32PT-I-SGC-R22.PmmC
• GFX (the one we want) : uLCD-32PT-I-GFX-R32.PmmC

Once you have all that installed, launch the PmmC Loader and choose the correct file :

Click Load, and it's done under a minute.

Compiling an example

We'll first assess that the module is correctly working by compiling and writing an example.

You have to install the legacy 4D Workshop IDE

This should work as expected (I tested on a Windows 7 virtual machine).

When you run the Workshop software, you have to select the correct platform : uLCD-32PT_GFX2 in the "Platform" select, and choose the correct COM port (here, COM3).

You can now compile and load any sample to see if it is working correctly.

Compiling a 4DVisi example

The IDE has a visual editor that allows to create richer interfaces using the 4DVisi format. This is quite powerful to create graphical programs, but it necessitates a bit more work to compile and run;

You need a micro SD card to copy support files that are then used by the program when it's running on the screen. The micro SD card must be formatted in FAT16 (so, a max of 4Gb).

There is a tool provided by the 4D IDE (Called RMPET) that can achieve that on Windows, and if you're on a Mac, I suggest that you use the newfs_msdos utility.

When it's done, you need to copy compiled assets (.cgi and .dat files to the SD card. When you build a solution that uses the 4DVisi format, you will be prompted to do so at the end of the compilation. Just select the drive your SD card is attached to, and click OK.

If you are on a VM like me, it gets a bit more complicated since it's quite unpractical to mount a SD card from the host system. In this case, you have to locate the two files, and manually copy them to the SD card.

Once it's done, insert the SD card into your 4D screen, then recompile and click on 'No, thanks' when it asks to copy the file. Tada !

Your code should now work and the assets should be correctly used.

On to something interesting

Now that everything works as expected, time to create a game !

We are not going to use the 4DVisi mode because it's not really needed I think, for a simple program like a flappy bird game, and it's a bit more hassle that needed for the assets. So we'll recreate the assets with code and simplify them heavily instead.

(TL,DR; The code is here and open source, as usual)

The 4DGL

The 4D Graphics Language is a bit like C but not really like it. It's not really practical, to be honest, but it's usable.

There is a reference manual here so you can dive into the language and its subtelties (if → endif, switch → endswitch and for → next .. Wait, what ?).

PICASO Internal functions

Of course we'll use the internal functions of the PICASO chip to access the display, the storage, etc ...

The API is documented here.

The basics

We'll take a very naive approach for this game with a very simple loop :

• handle touch events
• detect eventual collisions
• draw the background
• draw the obstacles (that we will move along the X axis)
• draw the bird in the correct position

Beforehand we'll display a very simple splash screen, and we the user loses, we'll just show the score.

The assets

I won't use any assets (ie bitmaps) but rather redraw everything in a very simplist way.

I have copied the bird pixel art and translated that to GFX functions (in calcBirdPosition()). This is a really simple approach. I've simplified the animation and created only two frames : one with the wing up, one down.

As for the "Mario tubes", they are simple rectangles, that I have very naively lighted from the left of the screen. One light and on dark column of pixels do the trick.

The display has a 16-bit depth so a tool like this will come in handy to convert 24 bit colors to 16 bit colors :
http://www.barth-dev.de/online/rgb565-color-picker/

The gameplay

It's basic. The bird is falling at constant acceleration (kind of), and a touch makes it "jump" to a few pixels up his position.

The obstacles (3 maximum at the same time) appear from the right of the screen and scroll to the left.

The collision detection is very simple, I just verify that the bird coordinates do no overlap the tubes, or the bottom of the screen.

The Y+ axis is downward when the FTDI plug is at the top of the display by the way

Saving the score on the µSD card

There is no EEPROM on the µLCD-32PT as far as I can tell, so to keep the score across power cycles, we need to write it down on the SD card.

We'll keep it simple and save the score in a TXT file at the root of the disk (score.txt).

On the device I have here, the SD card is somehow not very robust. I cannot mount the file system at times, and sometimes it works. I guess this may be due to the fact that the board has been sitting in a cupboard above my desk for quite a long time, not protected, and that the SD cards are quite old too.

So that's it !

Grab the code on my Github here and enjoy !

For less than 50€, you can get your hands on this tiny and easy to use sensor to monitor O₂ concentration up to 25%.

https://www.seeedstudio.com/Grove-Oxygen-Sensor-ME2-O2-2-p-1541.html

The problem is, the documentation is really crappy.
After hours of searching and testing to find the actual working example code for this sensor, I decided to get calibrated values for this sensor and interpolate to get the correct formula.

Compatibility

This sensor is compatible with 5V and 3.3V boards.

Measurement range

This sensor will provide reliable measurements for an O₂ concentration between 0% and 25%.

This sensor is an electrochemical cell, thus its lifespan will be finite. The datasheet says about 2 years but that highly depends on your environment and the concentration you're trying to detect.

Preheating time

Some pages will say that you need 48 Hrs before reading data. This is not true. Generally, in normal conditions, say, between 10°C and 30°C, the sensor will give a very reasonable output without preheating time.

That being said, it is a good thing to let it heat for a few minutes before actually reading data, so the output is more stable. Give it 20 minutes max.

Example codes

There is a documentation page available at https://seeeddoc.github.io/Grove-Gas_Sensor-O2/

The code is completely wrong, and will give you wrong values. Just by looking at the expected values for Vout (186 V ? Hmmm), you can tell that it's clearly not adapted. The values are in mA, and are related to the current output of the actual sensor (the ME2-O2-Ф20 on the board). Somehow the guys at Seeed studio messed up their example code.

So forget this code.

There is also some documentation at this page : http://wiki.seeedstudio.com/Grove-Gas_Sensor-O2/ where they provide an alternate example code.

Forget it also.

The calculation they make is as follow:

Concentration_O₂ = (Vout * 0.21 / 2) * 100;

While they do not explain why this would work, it actually give wrong values most of the time.

(FYI : the 0.21 comes from the actual amplifier that has a ratio of 210, but that's it)

A last example I found in a forum is a zip file (containing C code for Arduino) discussed in the related comment on the Robotshop forums : https://www.robotshop.com/community/forum/t/grove-o2-gas-sensor/24167/16

What this code does is basically assume that you will calibrate it in an open-air environment (20.8% O₂), and that 0V = 0ppm. This is not really accurate, see below.

Linearity

While electrochemical cell sensors are supposed to be strictly linear, there is always some differences in real life. This sensor is no exception; and if you assumes its linearity and that 0V = 0ppm, you might end up with values that are quite different when they are far from your calibration point (that will likely be ambient air, so 20.8% or so).

If you plan to use it in the full range of its capabilities, you have to calibrate it fully. That's what we'll try to do next.

Calibration

It's not really a calibration per se, but we're going to find data points where we know the dioxygen concentration for sure, and plot the whole thing, and hopefully find an accurate-enough linear approximation of the formula that we can use in real-world situations.

For this, I used a MAP Mix Provectus from Dansensor that can output a gas that has a calibrated concentration. With a N₂ and a O₂ bottle, I could expose the sensor to different concentrated gas from 2% to 25% of O₂.

Here are the data points I could get :

As we can see, it's not perfectly linear : there's a little offset (about 0.5%, still significant).

The calculated linear regression give a R² of 0.9986 which is not bad. The slope is 14.581.

So the formula would be :

// For Vout in volts
Concentration_O₂ = Vout * 14.581 + 0.5483; // in percent

You could also use the multimap function proposed in one of the examples with updated values :

float VoutArray[] = { 0, 0.13, 0.19, 0.24, 0.30, 0.35, 0.43, 0.49, 0.57, 0.64, 1.02, 1.31, 1.42, 1.68 };
float O2ConArray[] = { 0.5, 2, 3, 4, 5, 6, 7, 8, 9, 10, 15, 20, 21, 25 };
unsigned int ConArraySize = 14;

// This code uses MultiMap implementation from http://playground.arduino.cc/Main/MultiMap
float FmultiMap(float val, float * _in, float * _out, uint8_t size)
{
    // Take care the value is within range
    if (val <= _in[0]) return _out[0];
    if (val >= _in[size-1]) return _out[size-1];

    // Search right interval
    uint8_t pos = 1;  // _in[0] already tested
    while(val > _in[pos]) pos++;

    // This will handle all exact "points" in the _in array
    if (val == _in[pos]) return _out[pos];

    // Interpolate in the right segment for the rest
    return (val - _in[pos-1]) * (_out[pos] - _out[pos-1]) / (_in[pos] - _in[pos-1]) + _out[pos-1];
  }

// For Vout in volts
Concentration_O₂ = FmultiMap(Vout, VoutArray, O2ConArray, ConArraySize);

All in all, a complete solution would be along the lines of that (Vref = 5V or 3.3V depending on your board) :

// Read Vout on average
unsigned long sum = 0;
for (int i=0; i<32; i++) {
    sum += analogRead(O2_SENSOR_PIN);
    delay(10); // So we sample on a larger time interval
}

// Measured Vout (>> 5 = quick division by 2^5 = 32)
float Vout = (sum >> 5) * (VRef / 1023.0);
// Either one of those two - your preference :
o2_concentration_in_percent = Vout * 14.581 + 0.5483;
o2_concentration_in_percent = FmultiMap(Vout, VoutArray, O2ConArray, ConArraySize);

And 🎉! It should give you something quite accurate (for a sensor this unexpensive).

Disclaimer

On some forums, some users claim that the on-board circuitry for this gas sensor has changed over time, and that different versions exist under the same denomination. That can be true, and in this case, the calibration that I have done above may be off with your hardware, so be advised ;)

While developing for the IoT2040 from Siemens (see the product page), I was having difficulties with the I2C bus; specifically, some peripherals were slow to respond or somehow buggy and the bus was becoming unresponsive.

On this device, the I2C bus is behind an extender, and it seems not very robust when devices go berserk.

So I was tasked with finding a way to "reboot" the I2C bus if it found itself in a undefined state or if a peripheral was squatting the bus.

What came to mind was to do a power cycle so that the unrespectful devices could be powered off and rebooted fresh, generally freeing the data lines and prompting the bus to reset itself properly.

The naive solution I came up with was a simple dual MOSFET circuit that could allow for a software power cycle to be done via a control pin of the microcontroller that I was using (in this case, a Galileo Gen 2 — this is what is on the Siemens IoT20XX series).

A simple dual MOSFET solution

I had being advised to use a dual MOSFET, specifically one N and one P, to act as a switch to power on and off my I2C slaves.

I settled on the FDS8958A Dual N&P-Channel PowerTrench MOSFET from ON (Fairchild) (datasheet). There is no application schematics in the datasheet but something along those lines (adapt the R1/R2 values accordingly) should work as expected :

(the schematics is quite messy since I only had a standard 8 pin package chip and not a differentiated P and N component on Fritzing)

5V is my power source (we are using the Galileo Gen2 in 5V), I2C_BUS_CTRL is the control pin (HIGH means MOSFET is active), and I2C_VCC is the 5V output (all have the same GND) that is driven by the control pin.

The two middle pins D1 and D2 of the chip are not used as they are in fact redundant.

This works quite well, as the circuit is very fast (~10ns) and the output very steady. Setting the pin HIGH would power up the I2C slaves, and LOW would shut them down;

BUT

We then tried this circuit in a real world situation where the bus was used. And it turned out quite problematic, for various reasons :

• when powered down, some I2C slaves pulls the SDA and SCL lines LOW (~ 0.7V), which disables all traffic on the whole bus. I don't know if it's a standard behaviour but it certainly is not consistent accross the peripherals that I have
• the expander did not seem to be able to recover from a powered down I2C slave correctly (and the driver was not happy about it). This may be due to the fact that the lines were driven low
• Some low power devices actually act like they are still powered from the SCL line, which is unsettling. It defeats the whole purpose of the MOSFET since the peripherals does not really go through a power cycle

At the end of the day, power cycling the I2C slaves was causing more problems than before : the bus was unusable and the only option was a hard reboot on the IoT20XX.

An I2C repeater to the rescue

So I had this idea of cutting the I2C SDA/SCL lines too when powering down the slaves so as to be sure that they were really off the bus.

This would ensure that a powered-off slave would not prevent the bus from operating correctly.

I first tried the TCA4311A (datasheet). It's a neat little bus buffer that is specificaly designed for hot-swapping I2C slaves planes; I figured this is exactly what I wanted.

The application diagram was pretty straightforward :

(Taken from the datasheet)

Unfortunately this chip seemed to have a problem with the SMBus protocol. Specifically, all words were not transmitted correctly (whereas all single bytes were properly received by the slave). I had no access to a scope at the time so it was difficult to know exactly why though and to debug, but I guess this could be linked to the pre-charge circuitry (see §11 of the datasheet).

So I unsoldered my TCA4311A and (horribly) replaced it (please don't judge me) with a different kind of bird — an I2C repeater — to test if this would be better :

It's the TCA9517 (datasheet), that acts like a buffer too but in a different fashion.

The schematics to include it is pretty standard, as per the previous I2C bus buffer :

I2C_SCL_IOT and I2C_SDA_IOT are on the master side, whereas I2C_SDA_OUT and I2C_SCL_OUT are on the slave side. The I2C_BUS_CTRL is the same net as for the Dual MOSFET.

And this worked !

Of course, some pull-ups are required on each side of the buffer (I used 4,7kΩ but YMMV) :

With these two chips : the dual MOSFET and the bus repeater, I can now totally disconnect the needed I2C slaves when they behave badly. By setting one pin of my microcontroller LOW then HIGH, the VCC line of the slaves is down to GND and the slave part of the bus is deactivated, then reactivated as if it was connected for the first time.

Comments welcome as usual if you find an error or if you want to add something to this.

I recently got to dive into the embedded Linux build systems world while working on a project where I needed to create a distribution for an embedded device with limited resources.

Amongst the different options, I had identified two major tools : Buildroot and Yocto (See this fantastic talk by Alexandre Belloni and Thomas Petazzoni from Bootlin : Buildroot vs. OpenEmbedded/Yocto Project).

I decided to go for Yocto even if the learning curve was supposedly a bit steeper. In this article I'll try to describe my experience and give some tips along the way.

In a nutshell, what I wanted to do

So, I want to build an image for an Intel-based board, the Minnowboard Turbot. My goal is to have an image that boots silently, that runs a X environment and that can launch a fullscreen browser on boot.Pretty much a kiosk.

The specs of the board are here : https://minnowboard.org/minnowboard-turbot/technical-specs

Getting started

Building embedded Linux images is not really tricky per se, but it needs a lot of resources, and a lot of time. I mean a lot because you may be reading this article on your quad core 8Go Macbook Pro and probably think "ahah". Don't you dare. You need at least 100Go of disk and a good 16Go of RAM to have correct build times for Yocto. And when they say 100Go, be sure that they will be filled quite fast, and that the RAM /CPU usage will be close to 100% all the time, so it's a pain to work on the same machine at the same time.

Of course, I won't copy and paste the whole Getting started section of the Yocto project website since it's not really useful, but let's comment on a few steps here.

Get the correct docs

Here is the up to date documentation first :
https://www.yoctoproject.org/docs/latest/yocto-project-qs/yocto-project-qs.html

Pro tip™ : if you search for "yocto getting started" on Google, you most likely will land on an outdated version of the same documentation, which is .. unsettling.

Decide which machine you will use for the build

→ refers to the resources in the docs

I had a medium range linux machine (16Go, quad core) that was not used, so I decided to wipe it clean and reinstall a brand new Debian 9 on it.

When they say you need 50Go at least, double that number to be sure. My current folder is actually 64Go, and I cleaned most of previous builds and images to free up space already :

root@builder:~$ du -h -s yocto
64G	  yocto

You can use docker containers (via CROPS) but I would not recommend it since you will likely need a lot of resources.

The full uname -a for the machine I used is :
Linux 4.9.0-5-amd64 #1 SMP Debian 4.9.65-3+deb9u2 (2018-01-04) x86_64 GNU/Linux

Having a lot of RAM on the build machine can certainly help but having more than 4 cores shouldn't change your build time that much, since the things that take time will usually block your builds (many other packages depend on it), so it needs to be built before.

Processor speed can give you shorter build times.

Packages

→ refers to the packages in the docs

You need a few packages to be able to build everything correctly. Apart from the standard packages that the doc tells you to install ... :

apt install gawk wget git diffstat unzip texinfo gcc-multilib \
 build-essential chrpath socat cpio python python3 python3-pip python3-pexpect \
 xz-utils debianutils iputils-ping libsdl1.2-dev xterm

(Note that I installed git and not git-core since the latter is obsolete)

... I would recommend these as well :

• Tools, always handy to have to modify a local conf easily or get something from the web

  vim htop wget curl screen tree

• Packages I needed to install at some point to resolve an error in the build process (no clear explanation as why, but hey, the more packages the merrier)

  libssl-dev libev-dev asciidoc bc

Building and running in qemu

Cloning the repo is straightforward, you can source the oe-init-build-env file
and start the build with bitbake core-image-sato from the build folder for example.

The getting started tutorial is quite well-written in that regard.

4 hours minimum later, and you can emulate your machine with :

runqemu qemux86

(that is, if you have not changed any file, or option, for now)

Pro tip™: if you have a headless machine : runqemu qemux86 nographic

Copying to a SD card for testing on the target hardware

When compiling for Intel targets, a .wic image is created and is pretty straight-forward to use : just dd it to your SD card :

sudo dd of=/dev/diskX if=yourimage.rootfs.wic

Note : by default meta-intel .wic images only have an EFI bootloader.

Pro tip™: when you want to copy, use bs=512k. It's the standard bootloader size, and I've been told numerous times that it's always better to use this size. I stick with it and I think it's a good idea. And moreover I've had cases where using a different size would make the resulting SD card unable to boot at all.

Customize everything

So far, it's an easy ride. But we don't want a standard Sato GUI. So let's start customizing things. That's where things get a little harder.

In a nutshell

There is a quite complex image that gives you a general overview of the architecture of the Yocto environment :

... but I find it too abstract, so I'll simplify the concept and try to give you a more basic view of how it works in practice :

To customize your build, what you want to do is create a layer to hold everything in it. This layer (in green above) will reside alongside the other layers of your build.

A layer (Named along the lines of meta-my-layer for example) contains recipes, which can be image recipes (a special kind of recipe). Technically, a layer is a folder with some conventional files in it.

You create your layer, and you create all your recipes in it, that you can organize in recipe folders (recipes-common, recipes-my-app, etc ... in grey on the schema). A 'recipe folder' is just a folder. A recipe is also just a folder that contains at least one mandatory recipe file (a priori, a .bb file

Your recipes will possibly be only appending stuff to an already-existing recipe (a conf file, some bash scripts, etc). In this case, the recipe file is a .bbappend file

Finally, your image recipes will possibly call the packages from the recipes you created in your own layer, and packages from other layers too.

The process for customizing

Basically, you want to start with the smallest image that suits your needs, and add the packages you need in it (let's say chromium, X, and a custom app of yours, for instance). Two different steps to take:

• Clone the different layers that contain the recipes for the packages that you want to install
• Create your own layer and recipes for your custom app, and your modifications to existing packages if needed
• Finally, create an image layer that will hold the reference to all these packages : the ones that exist and your custom ones.

The files you must care about

Apart from your layer folder (we'll come to that in a bit), there are only two files that are important for the build :

build/conf/local.conf
build/conf/bblayers.conf

These will be modified quite often while you create your perfect image, so be sure to keep them at hand.

Adding existing layers to your build

Now, before you can install any package onto your build, you might want to add the layer in which this package is. This is the role of the build/conf/bblayers.conf file, that allows you to add layers in your build.

First, you have to download the layer in a folder somewhere. I use the root folder of poky so that all the layers are in the same place, but it's up to you.

If you are looking for layers or recipes, this is the place to go to :

• Layers : https://layers.openembedded.org/layerindex/branch/master/layers/
• Recipes : https://layers.openembedded.org/layerindex/branch/master/recipes/

You will then have to define the path of your layer in the build/conf/bblayers.conf file.

Suppose you want to use the meta-intel layer. First, clone it with :

git clone git://git.yoctoproject.org/meta-intel

Then, add its path in the file :

POKY_BBLAYERS_CONF_VERSION = "3"

BBPATH = "${TOPDIR}"
BBFILES ?= ""

BBLAYERS ?= " \
  ${TOPDIR}/../meta \
  ${TOPDIR}/../meta-poky \
  ${TOPDIR}/../meta-yocto-bsp \
  ${TOPDIR}/../meta-intel \         # <----- HERE
  "

BBLAYERS_NON_REMOVABLE ?= " \
  ${TOPDIR}/../meta \
  ${TOPDIR}/../meta-yocto \
  "

Once you rebuild your image or any package, this layer will be taken into account and the necessary recipes will be found.

Creating your layer

Now, let's create our own layer and recipes.

We'll name it meta-tchap. It will reside in the poky folder, where all the other meta-folders reside too.

We'll just create the basic structure here, with an images folder that will contain an image recipe.

root@builder:~/yocto/poky$ tree meta-tchap
meta-tchap
├── conf
│   └── layer.conf
└── recipes-core
    └── images
        └── core-image-iot.bb

Pro tip™: There is a tool for that if you don't want to dive into the files : yocto-layer, that you can access when you have sourced everything in your build folder. Something simple like yocto-layer create meta-tchap should do the trick.

The layer.conf file

It's a relatively standard file that is not likely to change, it justs imports all the recipes of the layer :

    # We have a conf and classes directory, add to BBPATH
    BBPATH .= ":${LAYERDIR}"

    # We have recipes-* directories, add to BBFILES
    BBFILES += "${LAYERDIR}/recipes-*/*/*.bb \
        ${LAYERDIR}/recipes-*/*/*.bbappend"

    BBFILE_COLLECTIONS += "meta-tchap"
    BBFILE_PATTERN_meta-tchap= "^${LAYERDIR}/"
    BBFILE_PRIORITY_meta-tchap = "100"

Since it's my own layer, I usually bump the priority up (Each layer has a priority, which is used by bitbake to decide which layer takes precedence if there are recipe files with the same name in multiple layers. A higher numeric value represents a higher priority.)

Creating the image recipe

An image recipe is a recipe. You just have to define what will be installed on it as a basis, with features and packages.

Let's create a simple image with X11, Chromium, Node and some other packages. We'll call it core-image-iot and it will reside in ...meta-tchap/recipes-core/images/core-image-iot.bb :

SUMMARY = "A full-featured + basic X11 / Chromium / NodeJS image."
LICENSE = "MIT"

IMAGE_FEATURES_append = " splash x11-base hwcodecs ssh-server-openssh post-install-logging"
IMAGE_FEATURES_remove = "allow-empty-password"
IMAGE_FEATURES_remove = "empty-root-password"

IMAGE_INSTALL = "\
    packagegroup-core-boot \
    packagegroup-core-full-cmdline \
    psplash \
    chromium-x11 sudo vim git curl htop wget unzip usbutils \
    nodejs nodejs-npm \
    ${CORE_IMAGE_EXTRA_INSTALL} \
    "

inherit core-image extrausers

# Here, we set the root password for the image
# password = test
EXTRA_USERS_PARAMS = "usermod -p m7or76bu6AEY6 root;"

This image inherits the core-image recipe, that has a lot already (see the file).

In this recipe, we use:

• IMAGE_FEATURES : it allows us to add features to the images, such as using the post-install-logging or adding x11. Features are just wrappers around packages, in fact
• IMAGE_INSTALL : it adds packages to the image, such as git, chromium-x11, etc ...
• EXTRA_USERS_PARAMS : it's a module that allow to work with users. Here, we use it to set the root password easily. You have to inherit it first.

Once we have this file, we can build this image very easily with :

bitbake core-image-iot

You need to clone a few layers before being able to build this image, since it relies on packages that are not in the base layer : https://github.com/OSSystems/meta-browser and https://github.com/imyller/meta-nodejs

Adding existing packages to your build

There are some packages that we don't want to put into our image recipe, but in the configuration file first, so it's easier to switch on/off between different builds or for testing.

To simply add a package to your build, you can add it to your build/conf/local.conf file :

IMAGE_INSTALL_append = " htop"

Pro tip™ : note the " " (space) after the opening quote, it's important, the string will be concatenated.

In this example, the htop package (that exist here).

As we have seen before, you have to have the layer somewhere beforehand. In the htop case, it's the meta-oe layer. You thus have to clone the repository (see info here):

git clone https://layers.openembedded.org/layerindex/branch/master/layer/meta-oe/

The meta-oe layer itself is in a subfolder of that git repo (as indicated in the openembedded.org page), so you should had the relevant path to your build/conf/bblayers.conf file like this :

${TOPDIR}/../meta-openembedded/meta-oe \

which give us a bblayers.conf file like this :

BBLAYERS ?= " \
  ${TOPDIR}/../meta \
  ${TOPDIR}/../meta-poky \
  ${TOPDIR}/../meta-yocto-bsp \
  ${TOPDIR}/../meta-openembedded/meta-oe \ # <----- HERE
  "

Now, you're ready to rebuild your image and tada, the htop binary should be available once you boot it.

 root@iotboard:~/$ htop --version
    htop 2.0.2 - (C) 2004-2016 Hisham Muhammad
    Released under the GNU GPL.

Append an existing recipe

Now you might have a recipe that exists but that you wish to modify somehow. You could create a whole new recipe, copy and paste the content of the recipe you wish to amend and call it a day (if your layer has a higher priority of course). But then, it's not really the Yocto way.

If you want to append something to a recipe, you can easily create a bbappend file and make your changes there.

Let's say you want to change the Chromium X11 recipe to add some flags to the build. Create the recipe folder as if you would create your own recipe and create a chromium-x11_%.bbappend file in it, with the following content :

FILESEXTRAPATHS_prepend := "${THISDIR}/${PN}:"
PACKAGECONFIG = "use-egl kiosk-mode proprietary-codecs"

Here, I'm adding these flags to the build (second line). The first line is a boilerplate instruction telling bitbake where it should look for extra files if any; Just put it there in all your append recipes.

In my layer folder (meta-tchap), I now have a folder more with the following structure :

recipes-browser/
└── chromium
    └── chromium-x11_%.bbappend

And that's it ! (don't forget to add your custom layer in the conf/bblayers.conf if you haven't done so yet).

PS : the name recipe-browsers is completely up to you. chromium is not : it's the name of the recipe you are appending.

If you want to check that your append is taken into account correctly, you can use the bitbake-layers utility :

bitbake-layers show-appends

It will show you which recipe is appended and by whom. Very practical.

Quiet boot with standard kernels

It's as easy as adding :

APPEND += "quiet vt.global_cursor_default=0"

.. to your image recipe, or your local.conf file.

Quiet boot with meta-intel

Well, not that easy, and so far I did not manage to pass boot options when using the meta-intel layer.

See : https://stackoverflow.com/questions/49033507/amend-boot-cmdline-in-custom-image-build

It seems complicated. As far as I understand the kernel does not take the APPEND variables into account. This seems to work on non-Intel builds, but not with the meta-intel layer.

To circumvent this limitation, I created a postinst script (see below) with the following content :

pkg_postinst_${PN} () {
    #!/bin/sh -e
    if [ x"$D" = "x" ]; then
        echo "default boot" > /boot/loader/loader.conf
        sed -i 's/console=tty0/quiet splash vt.global_cursor_default=0 vt.cur_default=0/' /boot/loader/entries/boot.conf
    else
        exit 1
    fi
}

This will modify options directly in the boot loader entries under /boot. This works well.

Exploring psplash

psplash is the de facto standard on poky to display a splash image while the system it booting.

Disclaimer : it's tied to sysvinit by default, so if you want to use systemd as your init system of choice, psplash won't work. Patches exist, though (I have not tested them) — one I found is https://patchwork.ozlabs.org/patch/351283/.

Create a custom psplash image

Two options here. Either you create it yourself and add the image file (as a .h file) in your layer, or you let bitbake recompile the image file to a .h file each time you build.

In both cases, you need to install psplash :

IMAGE_INSTALL_append = " psplash"

And create the bbappend file (I suggest psplash_git.bbappend) along with the files directory that will hold the image :

vi ...meta-tchap/recipes-core/psplash/psplash_git.bbappend
mkdir ...meta-tchap/recipes-core/psplash/files/

(do not forget to adapt for your own path)

1. First option : Create the image yourself

Clone the repo somewhere :

git clone git://git.yoctoproject.org/psplash && cd psplash

Considering your image is at ./psplash-poky.png :

./make-image-header.sh ./psplash-poky POKY
mv psplash-poky-img.h ...meta-tchap/recipes-core/psplash/files/.

(change the paths to fit your config obviously here too)

Your bbappend file will look like this :

FILESEXTRAPATHS_prepend := "${THISDIR}/files:"
SPLASH_IMAGES = "file://psplash-poky-img.h;outsuffix=default"

If you want to change the image, you have to redo this process (except the git clone of course)

2. Second option : Let bitbake do it

Well, pretty straightforward. No need to clone anything, create the bbappend file directly and put your PNG image in ...meta-tchap/recipes-core/psplash/files/ directly :

    FILESEXTRAPATHS_prepend := "${THISDIR}/files:"
    DEPENDS += "gdk-pixbuf-native"
    SRC_URI += "file://psplash-poky.png"
    SPLASH_IMAGES = "file://psplash-poky-img.h;outsuffix=default"

    do_configure_append () {
        cd ${S}
        # will create psplash-poky-img.h for you :
        ./make-image-header.sh ./psplash-poky POKY
    }

This will compile the image file at configure time.

psplash rotation

Because sometimes you need it, add this to your psplash_git.bbappend (if you want 90°) :

do_install_append() {
    echo 90 > ${D}/etc/rotation
}

Chromium on a basic X11 image

Working on my kiosk image, adding nodejs and npm was a walk in the park (just add the layer and append the recipes to the image). But adding Chromium and making it start at boot required a little more work.

You need a working X environment to display Chromium. Fortunately the x11-base image feature (IMAGE_FEATURES_append = " x11-base") provides just that, and no more.

You can then go for a full-fledged window manager but I personnaly think this is overkill if you're working on a device with limited resources such as an embedded system.

The X11 base feature contains just the vital minimum for this use case :

• A very simple and lightweight window manager (Matchbox)
• the Mini-X-Session session manager

Matchbox is an relatively old piece of code (~2012), is mainly intended for embedded systems and differs from most other window managers in that it only shows one window at a time. That's exactly what we need, in fact.

Mini-X-Session is a very simple session manager for X, that provides just the right boilerplate for us to create our session and launch the browser (see http://cgit.openembedded.org/openembedded-core/tree/meta/recipes-graphics/mini-x-session/mini-x-session_0.1.bb)

The mini_x session file

So, how do we glue all this together ? Well, first I'd recommend you create a user to run your X session, let's call him myUnprivilegedUser. You don't really want to run X as root on a probably user-facing linux box.

I suppose I have an app running at http://localhost:3001 for the sake of it (The app you want to run in your browser).

You can then create a session file in /etc/mini_x/session.d/ (or, quicker, replace /etc/mini_x/session altogether if you want to lock things a bit) — I've commented the file below so you know what it does exactly :

#!/bin/sh

# This script will be called via mini X session on behalf of the file owner, after
# being installed in /etc/mini_x/session.d/.

xset s off  > /dev/null 2>&1        # don't activate screensaver
xset -dpms   > /dev/null 2>&1       # disable DPMS (Energy Star) features.
xset s noblank   > /dev/null 2>&1   # don't blank the video device

# Set a resolution
xrandr -s 1920x1080

# Takes care of rotating the screen based on the content of /etc/rotation
# I've only implemented a single case (90 CW) but it is trivial to amend.
if grep -Fxq "90" /etc/rotation
then
    xrandr -o left
fi

# Run the wm first, so that our Chromium window is "aware" of the screen, and can resize correctly (if you don't run a wm first, your Chromium window will likely be half the size of the screen)
matchbox-window-manager -use_titlebar no -use_cursor no &

# Now, run chromium 'as' your user
su -l -c "/usr/bin/chromium --user-data-dir=/tmp --disable-session-crashed-bubble --disable-infobars --noerrdialogs --disable-restore-background-contents --disable-translate --disable-new-tab-first-run http://localhost:3001" myUnprivilegedUser

All the flags

All the flags are carefully crafter here in an attempt to minimize UI junk like popups and infobars. Customise for your own need.

But you might notice that some flags are missing for a proper kiosk experience. This is because they have been incorporated in the chromium build itself with Yocto, as I'm using a recipe that allows for it : https://github.com/OSSystems/meta-browser/tree/master/recipes-browser/chromium

For instance, to build Chromium with the kiosk mode already in, it's a easy as appending the Chromium recipe (we have covered that previously, but I'll just redetail here quickly)

Create a chromium-x11_%.bbappend file in meta-tchap/recipes-browser/chromium with the following content :

FILESEXTRAPATHS_prepend := "${THISDIR}/${PN}:"
PACKAGECONFIG = "kiosk-mode"

Rebuild your image (don't forget that building Chromium is a loooong process), and the flag will already be present (i.e., when running /usr/bin/chromium, it will already be in kiosk mode by default).

Trace/Breakpoint trap ? Oh, I see, it can be due to Chromium not being able to write to the user/data directory. This is the reason why I added the --user-data-dir=/tmp option when launching the browser to be sure that it doesn't panic on start (took me a while to figure this out in fact)

How to run a script on first boot (and only on first boot)

As we have seen, there are a few things that seem complex to do on specific kernels / under specific conditions, or that need hardware to complete (and thus cannot be done before the image is booted on the actual hardware). For these, a good old script that launches at first boot and then erases itself is clearly the best option. It's fairly transparent, and chances are good that your hardware will boot at least once before being put in production.

Fortunately, this is easily done in Poky. Enter pkg_postinst.

pkg_postinst_${PN} is a function that is run just after the package in which it resides is installed in the image. If the execution of pkg_postinst exits with success (exit 0), the script is removed (since they have run already). If not, they are keept and run again at first boot. With this, you can defer the execution until the first boot easily.

Let's take an example. I will create a specific recipe in my meta-tchap layer, just for that. Let's call it "startup" (any name will do). I have put this recipe under recipes-common but again here, any recipes folder is fine.

root@builder:~/yocto/poky$ tree meta-tchap
meta-tchap
├── conf
│   └── layer.conf
└── recipes-common
    └── startup
        └── startup_1.0.bb

In this startup_1.0.bb file, I will just run a very simple script on first boot that creates the file /tmp/i_was_here. The file /tmp/i_was_installed will be installed before, when the package do_install is called :

SUMMARY = "Just a simple recipe to test postint"
LICENSE = "MIT"
PR = "r1"

S = "${WORKDIR}"

do_install () {
    # You have to do something here ! Else, the package will likely not be installed
    touch ${D}/tmp/i_was_installed
}

pkg_postinst_${PN} () {
    #!/bin/sh -e
    if [ x"$D" = "x" ]; then
        # This will run on first boot
        touch /tmp/i_was_here
    else
        # This will run after package installation
        echo "Skipping postinst script, will do on first boot"
        exit 1
    fi
}

Do not forget to add the package to your image installs in your local.conf :

IMAGE_INSTALL_append = " startup"

If you build the image and open a devshell (see next section) afterwards to check which files are here, you will notice that only the /tmp/i_was_installed file will have been created.

Once you boot this image on the actual hardware, the /tmp/i_was_here file will be created, too, and the script will be deleted from the system automatically.

Pro tip™ : the list of all poky standard target filesystem paths are in the source here

The devshell

A devshell is a shell that opens in the recipe's target directory so you can check what has really been done / installed, and what will be copied onto the image.

bitbake -c devshell <recipename>

Pro tip™ : you have to fully build your recipe first (bitbake <recipename>), to access all the folders that I go through below. Doing so can help you troubleshoot specific problems with your build. IF you don't, only the source folder will be available (the git folder, for instance)

This will get you in the working dir for your recipe and source a shell with bitbake’s environment set up. Several folders there :

• git : where the git source are fetched and built (that is, in the case where you use git to fetch your sources)
• package : is where the files are copied for creating the ipk (or rpm) package that will be installed at the end. This is technically a replica of the / folder of your image, with just the folder and files that your recipes own and will install
• deploy-ipks (could be named deploy, or deploy-* depending on the package manager that you choose in your local.conf — I choose ipk). It's what will be installed via the package manager onto the image. Generally, you have three packages, one for production, one for development and one for debug.

You'll likely explore the package folder to see what you recipe created and check for potential problems.

Pro tip™ : exit to go back to your shell

Where does that variable value comes from ?

You will surely often wonder where does a specific configuration comes from, or what is its value.

For instance, you might want to know what is the preferred provider for the kernel, that is behind an obscure virtual/kernel variable.

In this case, bitbake -e comes in handy :

$ bitbake -e <your_image_name_here> | grep "^PREFERRED_PROVIDER_virtual/kernel"
   PREFERRED_PROVIDER_virtual/kernel="linux-intel"
   PREFERRED_PROVIDER_virtual/kernel_poky-tiny="linux-intel"

Great ! We now know the value !

Reconfigure the kernel you say ?

If you want to reconfigure the kernel, you can use the menuconfig utility. This page has relevant information on how to do it, and especially on how to save your modifications to a .config file and then find this bloody file in the Yocto tree.

bitbake -c menuconfig virtual/kernel

Pro tip™ : for an intel corei7 build the path you are looking for is something similar to :

    build
     └ tmp
       └ work
         └ corei7-64-intel-common-poky-linux
           └ linux-intel
             └ 4.9.81+gitAUTOINC+*
               └ linux-corei7-64-intel-common-standard-build
                  └ .config

For the Minnowboard, one configuration item that needs to be changed is CONFIG_IGB=y. It adds the Intel Gigabit Ethernet driver directly into the kernel instead of as a module, and helps recover the eth0 interface that is not created at boot otherwise.

Alright. This was a bit long, but I hope it can help anyone fiddling with Yocto to not bang its head over an obscure bug / behaviour. If you spot a mistake or think I have missed something, do not hesitate to drop me a line or comment below.

I have tried to find the most efficient way to redirect all trafic for a specific domain to its https counterpart, and also to redirect to the domain without the www subdomain.

I'm using Nginx in production and after combining various different solutions I found, I settled on this simple configuration that I now use and that I think is quite straightforward and efficient. Thought I'd share, for what it's worth.

(In this example the backend is a PHP application located in /var/www/my-website/ and I'm using letsencrypt for the certificates)

# no SSL
# Redirect both urls to the http server block
server {
        server_name my-website.fr www.my-website.fr;
        return 301 https://my-website.fr$request_uri;
}

# The main block : SSL
server {

        server_name my-website.fr www.my-website.fr;

        # If it has a www, rewrite.
        # The 'last' here is important
        # because we are cautious with the 'if's
        # nginx.com/resources/wiki/start/topics/depth/ifisevil
        if ($host ~* ^www\.){
            rewrite ^(.*)$ https://my-website.fr$1 last;
        }

        include ssl.conf;

        ssl_certificate /etc/letsencrypt/live/my-website.fr/fullchain.pem;
        ssl_certificate_key /etc/letsencrypt/live/my-website.fr/privkey.pem;

        # Application-specific stuff, just for illustrating
        index index.php;
        root /var/www/my-website;

        location / {
          # try to serve file directly, fallback to app.php
          try_files $uri /index.php$is_args$args;
        }

        # Pass on to FPM
        location ~ \.php$ {
           include php-fpm.conf;
        }

        # Deny access to .ht* files
        location ~ /\.ht {
          deny all;
        }
}

ssl.conf is as follows (of course, you might need to create the /etc/ssl/certs/dhparam.pem file beforehand, with sudo openssl dhparam -out /etc/ssl/certs/dhparam.pem 2048):

listen 443 ssl;

ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_prefer_server_ciphers on;
ssl_dhparam /etc/ssl/certs/dhparam.pem;
ssl_ciphers 'ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA';
ssl_session_timeout 1d;
ssl_session_cache shared:SSL:50m;
ssl_stapling on;
ssl_stapling_verify on;
add_header Strict-Transport-Security max-age=15768000;

And, for reference, php-fpm.conf is available here

In my recent PHP projects, I tend to use Symfony 4, and Webpack Encore to build my assets. I find it to be a strong set of base tools for PHP apps.

Webpack is a quite convenient way to build assets, widely used in the NodeJS world and for static sites, eventhough it can be slow sometimes. Webpack Encore is a wrapper around Webpack that simplifies its API, a bit like Webpacker.

Deployer is an atomic deployment tool that is quite easy to use, robust, and highly configurable.

But fiddling with the conf to have something ready for Symfony 4 + Webpack took me a bit of time, so here is my deploy.php file with comments inside so you can hopefully benefit from it. There are two options for asset build : either local or remote. I prefer local since it's faster, and you don't need to have your building toolchain on the production server, but sometimes it's not possible (when building changes the names of the assets if you version it for instance). Use what suits you the best.

It's also available as a gist here.

<?php

namespace Deployer;

require 'recipe/symfony4.php';

set('ssh_type', 'native');
set('ssh_multiplexing', true);

// Configuration

// For the repository, master is implied
set('repository', 'git@github.com:user/repo.git');

// Set shared dirs and dirs for Symfony 4
// I share sessions so that atomic builds do not "logout" all users 
// if users I have. This may be a problem if your deployment
// modifies your session system somehow, so be careful
// public/uploads is my standard upload directory for files
// and must be shared between deploys obviously.
set('shared_dirs', ['var/log', 'var/sessions', 'public/uploads']);
set('writable_dirs', ['var', 'public/uploads']);

// Paths to clear
// To avoid leaving unwanted access to these files in production,
// I simply clear what I don't need to run the app, and I run the 
// clear:path _after_ everything has been built.
set('clear_paths', [
  './README.md',
  './.gitignore',
  './.git',
  './.php_cs',
  './.env.dist',
  './.env',
  './.eslintrc',
  './.babelrc',
  '/assets',
  '/tests',
  './package.json',
  './package-lock.json',
  './symfony.lock',
  './webpack.config.js',
  './postcss.config.js',
  './phpunit.xml',
  './phpunit.xml.dist',
  './deploy.php',
  './psalm.xml',
  './composer.phar',
  './composer.lock',
  // We keep composer.json as it's needed by 
  // the Kernel now in Symfony 4
]);

// Set env, else composer will fail
// This is new since Sf3.4 I think, where we use a .env
// file instead of the parameters.yml file. Without these
// parameters, deployer will choke on deploy:vendors
set('env', function () {
    return [
        'APP_ENV' => 'prod',
        'MAILER_URL' => 'null://localhost',
        // Add more if you have other parameters in your .env
    ];
});

// Servers
// This is easy, just the server with a stage name so you can call
// `deploy production`
host('production')
    ->hostname('myhost.com')
    ->user('my_user')
    ->forwardAgent()
    ->set('deploy_path', '/var/www/my_project');

set('default_stage', 'production');
set('http_user', 'www-data');

// Tasks
// If you can / want to build assets locally and then upload, 
// if for instance you don't have the build tools on your frontend
// server.
desc('Build CSS/JS and deploy local built files');
task('deploy:build_local_assets', function () {
    runLocally('npm install');
    runLocally('npm run build');
    upload('./public/build', '{{release_path}}/public/.');
});

// If you want to build remotely
// For remote assets build, we need to know which npm to use
set('bin/npm', function () {
    return (string)run('which npm');
});
desc('Build CSS/JS remotely');
task('deploy:build_remote_assets', function() {
  run("cd {{release_path}} && {{bin/npm}} install && {{bin/npm}} run build");
});

// A simple task to restart the PHP FPM service, 
// if you use it of course
desc('Restart PHP-FPM service');
task('php-fpm:restart', function () {
    // The user must have rights for restart service
    // Change with your exact service version
    run('sudo systemctl restart php7.1-fpm.service');
});
after('deploy:symlink', 'php-fpm:restart');

// If deploy fails, automatically unlock
after('deploy:failed', 'deploy:unlock');

/**
 * The main task - it's basically the same as the symfony4
 * one but with rearranged tasks (especialy for clear_paths)
 * and added tasks (assets, cache, etc)
 */
task('deploy', [
    'deploy:info',
    'deploy:prepare',
    'deploy:lock',
    'deploy:release',
    'deploy:update_code',
    'deploy:shared',
    'deploy:vendors',
    'deploy:build_local_assets',  // Choose which version
    'deploy:build_remote_assets', // you prefer
    'deploy:cache:clear',
    'deploy:cache:warmup',
    'deploy:writable',
    'deploy:clear_paths',
    'deploy:symlink',
    'deploy:unlock',
    'cleanup',
])->desc('Deploy');

// Display success message on completion
after('deploy', 'success');

If you have some ideas to improve this or some comments, as usual, do not hesitate !

We love Adafruit.

We love their products, libraries, their tools, ... well basically everything they do and share to the public.

I've been using their Adafruit GFX library for a while now with every kind of display available on their shop : from the low-power Sharp to the SPI-enabled color TFTs, and some led matrix panels too ...

The provided fonts are easy to use and provide a wide range of possibilities, but if you want to design specific screens, you clearly want to use your own font.

For a recent project, I needed to use Roboto Thin and Roboto Light but I ran into some problems when converting the fonts.

Converting a font for use with the lib

It's pretty much explained in the /fontconvert part of the library repository here, but let me break it out again.

You need a C compiler, the TTF file or your font and a text editor to proceed.

First, set the dpi of the font in the file first. This is important since it dictates what the size of the font is really compared to the dot size of the screen.

It's in the fontconvert.c file, line 27 :

...
#include "../gfxfont.h" // Adafruit_GFX font structures

#define DPI 141 // Approximate res. of Adafruit 2.8" TFT

// Accumulate bits for output, with periodic hexadecimal byte write
void enbit(uint8_t value) {
...

It's the #define DPI 141 that you want to change. To calculate yours, it's quite simple. For instance, for the Sharp display, we have the following info

• resolution is 168x144 pixels (height x width)
• width is 1.3''

So we will have (we'll use the width since pixels are square) :

DPI = 144 / 1.3 = 110.7 ≃ 111 

So you just use

#define DPI 111

Now you can build the tool thanks to its makefile :

cd fontconvert
make

Now you can use the fontconvert tool like this

./fontconvert RobotoThin.ttf 12 > RotoboThin_Fixed12pt7b.h

Customizing / repairing the output

But sometimes, the created file is not "pixel perfect" in the sense that some pixels are obviously missing.

For example, for the RobotoThin variant in 7pt for a DPI of 141, I get a half-baked p and E :

This is likely due to freetype making (bad) choices on how to approximate sub-pixels, but we end up with a font that's not usable.

The real problem is that the resulting header file is hard to grasp, and hard to modify to add a single missing pixel.

Worry not, though ! :)

I created a font customiser to do just that : "decompile" the header file, display it in a handy editable format where you can change pixel, add rows or columns for each glyph, and then "compile" it again to a header file.

https://tchapi.github.io/Adafruit-GFX-Font-Customiser/

It's pretty straight forward to operate :

Paste the content of a converted file in the input textarea, and click on Extract. It takes a few seconds as the font is parsed and rendered as glyphs on the page.

When it's done, you can edit each glyph :

• click on a pixel to toggle its state (lit / unlit)
• click the buttons "+Row" or "+Col" to add a column or a row to the glyph. The columns are added on the right and at the bottom

Once you're happy with the result, click Process and you have the resulting font in the output textarea. 🎉

As usual, it's all open-source on Github. The code (a single HTML page with JS) is clearly not perfect and a bit messy, but hey, it does the work. PRs welcome of course.

When you need to take control over a remote server or device, problems start to arise. Whether the device is behind a corporate firewall, a restrictive set-top box, or is just not publicly available from the Internet, it's always difficult to gain and keep access over the lifespan of the service the device is supposed to provide.

SSH tunnelling is one option for this that is dead simple, super powerful, and yet not very well known.

Ngrok and LocalTunnel provide this functionality somehow, and allow global access to "local" devices through a SSL tunnel. Ngrok link allows remote management of devices. All this services rely on a private central server that the SSH tunnel goes through

What I'm presenting here is a much bare bone solution that only relies on a simple SSH tunnel to gain and keep access to a remote device through a central public server.

What do we need

For the SSH tunnel to work, the only requirement is a server, publicly accessible, that runs a SSH daemon, and a user that can access this machine. We'll use the user foo, and the server will be central.example.org.

The user just needs to exist and be able to login.

To create it, a simple sudo useradd foo and passwd foo afterwards. For connecting, it's a good idea to add the connecting device's public key to the ~/.ssh/authorized_keys file of this foo user so that the tunnel creation can be non-interactive.

It will be mandatory if you use that technique to access remote devices (on which, by definition, you have no way to interactively input a password ...)

Of course you need some kind of device that we will access and manage remotely;

We're going to use basic Linux functionality and some NodeJS for easy creation of the tunnel.

Making local global

The first easy use-case is to make a local port available to the world thanks to SSH. This is easily done.

Suppose you have a local machine on which you develop, and you have a node service running on port 3000.

To make that port available to anyone (for remote testing for instance), just create a tunnel that forwards your local port to a random port on the central server :

ssh foo@central.example.org \
  -o PasswordAuthentication=no \
  -o ServerAliveInterval=30 \
  -N -R 0:localhost:3000

The output will be something like :

Allocated port 42671 for remote forward to localhost:3000

The port is allocated randomly. You can use a predefined port but make sure that you know that this port is not used somewhere else :

ssh foo@central.example.org \
  -o PasswordAuthentication=no \
  -o ServerAliveInterval=30 \
  -N -R 13245:localhost:3000

The output will be empty, don't expect anything — you already know the port you're forwarding to !

As long as the process is alive, the forwarding is effective. Just CTRL + C to stop the whole thing.

Accessing remote devices

The exact same principle can be used to gain remote access to your devices in the wild. The tunnel must be initiated on the remote device, to your central server.

If you choose to use a random port number, there is an extra step for you to retrieve the allocated port, but as your remote device is connected (since it just opened an SSH connection), you can either send the port via mail (quick and dirty) or use any other method to retrieve it (send a GET request to your backend for your devices for instance).

To use that, I have created a simple Node wrapper that I now use when I need to create a tunnel between remote devices and a central 'command' center.

The full code is available on Github as usual : tchapi/node-simple-ssh-tunnel

To use, it's pretty simple. It's just a wrapper around the aforementioned SSH stanza :

const tunnel = require('ssh_tunnel')

tunnel.setConfig({
    user: "foo",
    server: "central.example.org",
    port: 22,
    timeout: 10000, // in ms
})

tunnel.start(function() {
    console.log(tunnel.getState())
    
    // Later ...
    tunnel.stop()
}, function() {
    console.log("There has been an error ...")
})

And you would get something like that :

$ node examples/index.js
[info] Starting tunnel to foo@central.example.org
[info]  Allocated port 43557 for remote forward to localhost:22
[success] Tunnel active at 43557
{ port: '43557' }

Now, up to you to send this tunnel.getState() info to somewhere you can retrieve it.

Easy and robust.

Now, to connect to your remote device from anywhere, you just have to do :

ssh device_user@central.example.org -p 43557

which is strictly equivalent to :

ssh device_user@remote_device

... if you were on the same local network as the device.

?

I've been working on information displays for quite a while and one major feature of these displays is that they are required to display as little as possible of the underlying system they are using.

No one wants to see a BSOD, a boot sequence or any kind of error message pop up on a large public screen.

So how do we do that ?

As we rely on different software layers to display the content we really want to display (say, a webpage), we have to make sure that every one of them is not verbose so the user experience is controlled.

In the following article I will focus on two boards : the Raspberry Pi B 3 (ARM) and the Upboard (Intel-based)

The Pi has a standard raspbian jessie lite (should work with stretch, too) on it, based on debian jessie. On the upboard, I decided to go the Ubuntu Server route, since ubilinux (the distribution that upboard proposes) doesn't have the same community yet. I felt a simple 16.04 was more adequate.

I didn't install the linux-upboard kernel flavour on the upboard since I didn't need the GPIO support but I guess this should be harmless for our use case here

Silent boot

First things first : the boot sequence.

On recent systems, it's generally quite easy to get rid of all the output at boot.

Raspberry Pi

A few steps are required to have a perfectly blank screen.

Mute locale warnings by setting the locale in /etc/environmnent (you can choose any installed locale):

LC_ALL=en_GB.UTF-8
LANG=en_GB.UTF-8

Make sure date and time are correctly configured, to avoid timezone warnings :

sudo dpkg-reconfigure tzdata

Add this in /boot/config.txt:

disable_splash=1
avoid_warnings=1

And change your /boot/cmdline.txt to something along those lines:

dwc_otg.lpm_enable=0 root=/dev/mmcblk0p2 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait logo.nologo console=null quiet vt.global_cursor_default=0

The main part is logo.nologo , console=null and quiet. You should remove the part about the serial console too

Upboard

It's a bit simpler if you use GRUB as the bootloader.

Edit /etc/default/grub and add :

GRUB_CMDLINE_LINUX_DEFAULT="quiet loglevel=0 logo.nologo console=tty12 --quiet"
GRUB_CMDLINE_LINUX="quiet loglevel=0 logo.nologo console=tty12 --quiet" #Don't show kernel text
GRUB_HIDDEN_TIMEOUT=0

And then run sudo update-grub2.

This will disable the GRUB bootloader menu, and remove all text logs from the screen (hopefully). It appears that some errors may still show but I guess they can be specifically removed by disabling the correct modules or so.

Disable login screen

Next, you want to disable the login screen on tty1 so no prompt appears after the boot :

sudo systemctl disable getty@tty1

This is valid for both platforms.

That's it ! You should have a perfect black screen all the way, and no output whatsoever (via HDMI, at least).

Displaying things

What's the best way to get rid of any kind of error messages ? Well, how about having no window manager in the first place ?

In this scenario, we want to display a single app : a web browser, and expect no user interaction whatsoever.

It makes sense to only use a display server for this application, make it fullscreen, and call it a day.

This allows for a clutter-free screen, no error messages (at least from the X server) and a minimal installation footprint.

And it everything crashes, you end up with a black screen by construction which is quite a good fallback (except if your web app crashes with a 500 or displays giberrish, but that's out of the scope here).

In the following installation, I will configure the display manager to automatically start our chosen browser fullscreen with the relevant options.

Installation

First, some packages are necessary : xorg, xserver-legacy and xinit :

sudo apt-get update
sudo apt-get install build-essential unclutter xorg xinit xserver-xorg-legacy -y

unclutter hides the mouse cursor, it's a neat little utility.

Configuration

Then run this to allow anybody to start X (you don't want to run it as root):

sudo dpkg-reconfigure xserver-xorg-legacy

You may want to add the user to the tty group too :

sudo usermod -a -G tty {yourUserName}

Creating necessary X startup files

Create /home/{yourUserName}/.xserverrc and add the below content:

#!/bin/sh
# Start an X server with power management disabled so that the screen never goes blank
exec /usr/bin/X -s 0 -dpms -nolisten tcp "$@"

This will allow to start X with the display sleep management shut off, so your display is always on.

Then, to automatically start the browser when X launches, create /home/{yourUserName}/.xsession and add the below content:

#!/bin/sh
exec /usr/bin/chromium-browser %u \
  --kiosk --start-fullscreen --incognito \
  --no-first-run \
  --disable-session-crashed-bubble \
  --disable-infobars \
  --disable-restore-background-contents  \
  --disable-translate  \
  --disable-new-tab-first-run \
  --noerrdialogs \
  --no-sandbox \
  --user-data-dir=/tmp \
  --window-size=1600,900 \
  http://localhost:3000 

Change the url to match yours, of course, and even if the "start-fullscreen" flag is on, it seems to be a bit buggy since we don't have a window manager, so I add the actual resolution of the screen to be sure the window takes the full width and height.

There is a load of options here, that I carefully tested. To my opinion this is the best flag list for this use. Do not hesitate if you have any other input or improvement over this !

As well, we're assuming we're using chromium here, but you can replace that with /usr/bin/google-chrome. See below for the two options

You can then run directly the whole thing with :

/usr/bin/startx -- /home/admin/.xserverrc -nocursor

or make a unit file to start on boot directly via systemd :

[Unit]
Description=Public Display device
After=network-online.target
Before=multi-user.target
DefaultDependencies=no

[Service]
User={yourUserName}
ExecStartPre=/bin/sleep 5
ExecStart=/usr/bin/startx -- /home/{yourUserName}/.xserverrc -nocursor
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target

Now, for the choice of browser !

Using Chromium

There is apparently a bug in Chromium (a bit related to https://github.com/RPi-Distro/repo/issues/58 but not exactly) that makes the latest versions crash when runnning in kiosk mode on embedded devices. This seems to be linked to starting Chrome in Xorg directly.

The last known to work version is 48.0.2564.82 afaik, so we need to install this manually.

First remove the previous packages if any :

sudo apt-get remove chromium-codecs-ffmpeg-extra chromium-browser chromium-browser-l10n

And install the correct version. There are two scenarii, one for the Pi (ARM based), and one for the Upboard.

Raspberry Pi

Get the packages :

wget https://launchpad.net/~canonical-chromium-builds/+archive/ubuntu/stage/+build/8883797/+files/chromium-codecs-ffmpeg-extra_48.0.2564.82-0ubuntu0.15.04.1.1193_armhf.deb https://launchpad.net/~canonical-chromium-builds/+archive/ubuntu/stage/+build/8883797/+files/chromium-browser_48.0.2564.82-0ubuntu0.15.04.1.1193_armhf.deb http://launchpadlibrarian.net/234938396/chromium-browser-l10n_48.0.2564.82-0ubuntu0.15.04.1.1193_all.deb https://launchpad.net/~ubuntu-security/+archive/ubuntu/ppa/+build/8993250/+files/libgcrypt11_1.5.3-2ubuntu4.3_armhf.deb

Install them :

sudo dpkg -i libgcrypt11_1.5.3-2ubuntu4.3_armhf.deb
sudo dpkg -i chromium-codecs-ffmpeg-extra_48.0.2564.82-0ubuntu0.15.04.1.1193_armhf.deb
sudo dpkg -i chromium-browser_48.0.2564.82-0ubuntu0.15.04.1.1193_armhf.deb
sudo dpkg -i chromium-browser-l10n_48.0.2564.82-0ubuntu0.15.04.1.1193_all.deb

? BOOM. Done.

Upboard

Get the packages :

wget http://launchpadlibrarian.net/234938404/chromium-browser_48.0.2564.82-0ubuntu0.15.04.1.1193_amd64.deb http://launchpadlibrarian.net/234938396/chromium-browser-l10n_48.0.2564.82-0ubuntu0.15.04.1.1193_all.deb http://launchpadlibrarian.net/234938406/chromium-codecs-ffmpeg-extra_48.0.2564.82-0ubuntu0.15.04.1.1193_amd64.deb

Install them :

sudo dpkg -i chromium-codecs-ffmpeg-extra_48.0.2564.82-0ubuntu0.15.04.1.1193_amd64.deb
sudo dpkg -i chromium-browser_48.0.2564.82-0ubuntu0.15.04.1.1193_amd64.deb

At some point you might need to correct dependencies because the adm64 version depends on a bit more things apparently :

sudo apt install -f

? BOOM. Done.

Holding packages

Don't forget to block updates on chromium packages :

sudo apt-mark hold chromium-codecs-ffmpeg-extra chromium-browser chromium-browser-l10n

To unhold the packages :

sudo apt-mark unhold chromium-codecs-ffmpeg-extra chromium-browser chromium-browser-l10n

Using Google Chrome

Add a new repo list file: sudo vi /etc/apt/sources.list.d/google-chrome.list with the following content :

deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main

Add the Google signing key :

wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | sudo apt-key add -

And then install Chrome:

sudo apt update && sudo apt install google-chrome-stable

It only works on Intel-based architectures as far as I could test, so Chromium is the only option for now on Raspberry Pi

All good

So far, so good. I have tested this on a stock Raspberry Pi B3 and an Upboard and it runs smoothly. Crashes have happened, and

But maybe I have missed some options / interesting additions ? Do not hesitate to tell me in the comments.