Read-only Root Filesystem

Warning

As of the RC4 release, the rootfs is mounted read-only.

While a few desktop Linux systems and larger servers may mount the root filesystem (commonly called the “rootfs”) read-only for security reasons, the vast majority of these systems mount it writeable as a matter of convenience. What these systems share is the ability of qualified system administration resources to regularly access, monitor and update the rootfs and the various software packages on the system.

As the G3 module is an embedded Linux system that is one component of an end product that is usually deployed into an industrial or professional office environment, system reliability and resistance to tampering become the overriding consideration. As with any Linux system, there are various scenarios in which the rootfs could be irrecoverably damaged or tampered with if the G3 module was deployed with a writeable rootfs. To prevent occurences of these scenarios in the deployed product to the degree possible, Reach Technology has engineered the production embedded software images with a read-only rootfs. This has implications which are discussed below.

There are four areas of the rootfs which commonly require regular write access:

  • configuration files in /etc

  • various runtime state files in /var

  • various temporary files in /tmp

  • items in the root user home directory (commonly /root or /home/root)

Reach Technology has employed various techniques to support writing to these areas despite the fact that the rootfs is mounted read-only1.

  • /etc is mounted using the OverlayFS filesystem. The /data/etc directoryis the “upperdir” While the /etc in the actual rootfs partition is the “lowerdir”. This stack is accessed via the normal /etc path. The end result is that /etc appears to be writeable; however any changes are actually written to /data/etc.

    This is critical functionality to support OTA (over the air) updates to the rootfs! Why? Because it prevents “soft” bricking your product. Soft bricking can occur when you perform a rootfs update that looses configuration data for either for your end product application or for the system itself. A poster child example would be losing network configuration data so your product can’t access needed resources or offer services (e.g. a web UI or RESTful API).

    The other advantage is that you can use Mender to update your application in /data without disturbing the rootfs. This makes it possible to recover from a borked end product application update without having to resort to A/B copies of the /data filesystem which in turn makes more of the fixed size eMMC available to your application. A/B copies of the rootfs are required (and are in place) to facilitate fully recoverable updates to the rootfs.

  • /var is bind mounted to /data/var. Refer to the mount manpage for more information on bind mounts. This is a simpler technique than using the OverlayFS filesystem and somewhat faster. The net effect is essentially the same; anything written to /var actually shows up in /data/var. It is not an appropriate solution for /etc for a number of reasons. The most important of which is the init system needs to access some configuration files before the OverlayFS filesystem mount takes place.

  • On desktop systems, /tmp is normally kept separate from /var/tmp, and they have different semantics. Traditionally, /tmp is strictly for non-persistent ephemoral data while /var/tmp is for temporary files that should survive a crash or reboot until explicitly deleted. Most Linux embedded systems have merge the two locations by making /tmp a symbolic link to /var/tmp. Further, several subdirectories of /var including /var/tmp are relocated at boot time to a tmpfs (in memory) filesystem that is mounted at /var/volatile. This means neither the contents of /tmp nor the contents of /var/tmp will survive a crash or reboot.

    The other important ramification is that it is quite possible to cause an OOM (out of memory) kernel panic by writing temporary files without controlling the expansion of the tempfs in system RAM. Basically, you have several things competing for available RAM:

    • The kernel

    • The buffer cache

    • The /var/volatile tempfs

    • Your application

    • Other system services and applications

  • The root user home directory has been relocated to /etc/root to take advantage of the OverlayFS filesystem mounted there. Again, any writes into that directory will actually show up in /data/etc/root.

Important

The most critical take away for the end product application architect is that all components of the end product application must now be installed in /data2.

Footnotes:

1

Refer to the next section on the layout of the /data filesystem for additional information.

2

It is possible to revert to a read-write rootfs. Reach Technology strongly discourages you from doing this. If you absolutely must do this, system updates with Mender are not supported by Reach Technology; i.e. do this at your own risk.