s6-rc: the s6-rc-compile program

The s6-rc-compile program

s6-rc-compile is a service database compiler. It takes a series of service definitions in the source format, and compiles them into a directory, which is the compiled format.

The administrator can then examine that compiled database via s6-rc-db, put it into a place where it will be registered at boot time as the current compiled database by s6-rc-init, or even live update the current service database via s6-rc-update.

Interface

     s6-rc-compile [ -v verbosity ] [ -h fdhuser ]  [ -b ] compiled source...

s6-rc-compile analyzes every directory source in its arguments. For every subdirectory service in source, it expects to find a valid service definition in service.
s6-rc-compile outputs a compiled version of the service database into compiled. This database contains information for all the services declared in every source argument.

Exit codes

0: success
1: error in a source directory
100: wrong usage
111: system call failed

Options

-v verbosity : be more or less verbose. Default is 1: warning and error messages will be printed to stderr. 0 silences warnings. 2 adds a bit more information about what s6-rc-compile is doing. 3 or more is heavy debug output.
-h fdhuser : arrange for the s6-fdholder-daemon program, which maintains the pipes for the longrun pipelines, to run as user fdhuser. By default, it runs as the user owning the supervision tree, i.e. most likely root.
-b : make s6-rc-oneshot-run invocations wait instead of fail on lock contention. This should not change anything in practice, and you can ignore that option.

Source format

s6-rc-compile scans every source directory to find service definition directories in it. It ignores every file that is not a directory, or that starts with a dot. For every service definition directory that it finds, it creates a service with the same name as the directory. Names cannot be duplicated and cannot contain a slash or a newline; they can contain spaces and tabs, but using anything else than alphanumerical characters, underscores and dashes is discouraged - the s6-rc programs will handle weird names just fine, but other tools, especially shell scripts, may not. Names are also forbidden to use the reserved s6rc- and s6-rc- prefixes.

Every service definition directory service is expected to contain the following files:

For every service

A regular file named type, that contains only the text oneshot, longrun or bundle, and a terminating newline. This file declares the type of service defined by the directory.
An optional regular file named flag-essential. The contents of this file are irrelevant, only its presence is tested. If this file exists, the service will be marked as essential, which means that a s6-rc -d change foo command will not stop the service; only a s6-rc -D change foo command will. If the service is a bundle, the flag will be propagated to all its contents, i.e. all the services it represents will be marked as essential.
An optional regular file named flag-recommended. The contents of this file are irrelevant, only its presence is tested. If this file exists, the service will be marked as recommended. This is only important for repo commands, not when compiling the database.

For bundles

A directory named contents.d. This directory must contain files, no matter their type, no matter their contents, named after services. (Typically, empty regular files are used.) All the services named in this directory will be represented by the bundle named service.
If contents.d does not exist, the bundle contents can be read from a regular file named contents. This file must be a list of service names, one per line. Whitespace at the beginning of a line is ignored, but trailing whitespace is not. Lines starting with a # character are ignored. The file defines the services that will be represented by the bundle named service. Note that this format is deprecated.

It is possible to use bundle names in a contents.d directory or a contents file. However, if s6-rc-compile detects a cycle in bundle definitions, it will complain and exit 1.

For atomic services

An optional regular file named timeout-up. This file, if it exists, must contain an integer, which is the maximum number of milliseconds s6-rc will wait for successful completion of the service start; if starting the service takes longer than this value, s6-rc will declare the transition a failure. If the file does not exist, or contains 0, no timeout is defined and s6-rc will wait indefinitely for the service to start.
An optional regular file named timeout-down. This file, if it exists, must contain an integer, which is the maximum number of milliseconds s6-rc will wait for successful completion of the service stop; if stopping the service takes longer than this value, s6-rc will declare the transition a failure. If the file does not exist, or contains 0, no timeout is defined and s6-rc will wait indefinitely for the service to stop.
An optional directory named dependencies.d. This directory must contain files, no matter their type, no matter their contents, named after services. (Typically, empty regular files are used.) All the services named in this directory will be direct dependencies of service, i.e. the services that must be up in order for service to work properly.
If dependencies.d does not exist: an optional regular file named dependencies. This file must be a list of service names, one per line. Whitespace at the beginning of a line is ignored, but trailing whitespace is not. Lines starting with a # character are ignored. The file defines the direct dependencies of service. Note that this format is deprecated.

It is unnecessary to manually define complete sets of dependencies in the dependencies.d directory or the dependencies file, because s6-rc will properly handle dependency chains. If A depends on B, no matter the underlying implementation of B, and the current implementation of B depends on C, then you should just put B in A/dependencies.d; when starting the set, s6-rc will start C first, then B, then A. If the underlying implementation of B changes and does not depend on C, then you will just have to modify the dependencies for B, and the definition of A will still be correct.

Of course, if A depends on C anyway, you should add both B and C to A/dependencies.d.

If s6-rc-compile detects a cycle in dependencies across services, it will complain and exit 1.

For oneshots

Two regular files named up and down, which must each contain a single Unix command line. The files will be interpreted by the execlineb lexer at compile time and the results will be stored into the compiled database in an internal form. up will be run when the service is started, and down will be executed when the service is stopped. up is mandatory, but down is optional; if no down file is provided in the source definition directory, then it is treated as the empty script. If a script is empty, then s6-rc will consider that the corresponding transition for this service does nothing and always succeeds.

up and down are interpreted by execlineb, but that does not mean they have to be entirely written in the execline language. The execlineb lexer is only used because it can compile a Unix command line from a text file and store the compiled result, whereas a shell would have to be invoked everytime the script is run. There are many ways to write up and down scripts:

They can be written as standard execline scripts, without the initial shebang line or positional parameter management.
They can be written in any scripting language by invoking the interpreter directly: for instance /bin/sh -c "script", where script is a shell script.
They can also just call a script that will actually be stored somewhere else: for instance, up can contain /etc/init.d/service start while down contains /etc/init.d/service stop, and /etc/init.d/service is a System V-style init script.

Don't think you have to learn all the intricacies of the execline language just because the up and down scripts get lexed by it. You don't.

For longruns

The s6-rc service definition directory for a longrun service is similar to a s6 service directory, but there are a few differences:

s6-rc-compile crafts the servicedir itself, based on what it finds in the service definition directory. It does not copy everything directly from the definition directory to the servicedir; only two subdirectories will be copied verbatim, data and env. So if you want to store service configuration data, to be used by the run script, in the service directory, make sure it is in a data/ or env/ subdirectory.
Definition directories cannot have a log subdirectory - or if they do, it will be ignored. From s6-rc-compile's point of view, logged s6 services must actually be defined as two separate s6-rc services, one defined as a producer and one defined as a consumer, making a pipeline of just two services; see below for more information about pipelines.

The following files must or may appear in a longrun definition directory:

A mandatory regular file named run, as well as optional files named finish, notification-fd, lock-fd, timeout-kill, timeout-finish, max-death-tally, and down-signal. These files will be copied, or recreated, in the generated service directory: they are meant to be used by the s6-supervise process that will manage the longrun service.
Optional directories named instance and instances. These will be copied verbatim into the generated service directory. They are used internally by some s6 tools; using them for your own purposes is not recommended - you should ignore them and use the data directory instead.
Optional directories named data and env. These will be copied verbatim into the generated service directory.
An optional file named producer-for. If this file exists, then it must contain the name of another longrun service servicelog; service is then declared as a producer for servicelog. servicelog must also, in its own definition directory, be declared as a consumer for at least service.
An optional file named consumer-for. If this file exists, then it must contain a list of longrun services serviceprod...: service is then declared as a consumer for all the services in serviceprod. Each of those services must also, in its own definition directory, be declared as a producer for service.
An optional file named pipeline-name. If this file exists along with a consumer-for file, and there is no producer-for file, then a bundle will automatically be created, named with the content of the pipeline-name file, and containing all the services in the pipeline that ends at service. See below for more about pipelining. The pipeline-name file is ignored if service is not a last consumer.

Be aware that service directories will be relocated, and copied at boot time, so if your run or finish scripts refer to files in the service directory (probably in the data or env subdirectories!), they should use relative paths, not absolute ones.

Note that you cannot create a ./down file for s6-supervise in a generated service directory. Even if such a file exists in the definition directory, it will be ignored - it will not be replicated in the service directory. This is intentional: s6-rc internally uses ./down files in the service directories it manages, to mark longrun services that are down.

The producer-for, consumer-for and pipeline-name files are used to set up automatic longrun pipelining.

Longrun pipelining

Users of supervision suites know about logged services: a service acts as a producer, and is coupled with another service, its logger; the supervision system automatically maintains an open pipe between the producer's stdout and the logger's stdin.

s6-rc comes with an extension of this mechanism. Rather than only allowing two longrun services to be pipelined, it can set up an indefinite number of longrun services this way.

A producer declares its direct consumer in a producer-for file.
Intermediate services declare both their direct producers in their consumer-for file, and their direct consumer in their producer-for file.
The last consumer only declares its direct producers in a consumer-for file.
The last consumer may also declare a name for the whole pipeline, in its pipeline-name file. If it does so, then a bundle is automatically created with the given name, and it contains all the services in the pipeline.

s6-rc-compile will detect pipelines, and set up the service directories so that every producer's stdout is connected to its consumer's stdin, and that the pipes are not broken whenever one element in the chain dies.

A service can only be declared as a producer for one other service, but it can be declared as a consumer for several other services: the consumer-for can have several lines. This means that one service can only send its output to one consumer, but it can read its input from several producers. If there are several producers to a service, they will all write to the same Unix pipe that the service reads.

This means that what s6-rc calls pipelines are really funnels: you can collapse multiple data streams into a single data stream, at every step of your processing chain. The pipeline terminology remains for history reasons: previous versions of s6-rc could not handle multiple producers.

s6-rc-compile checks for pipeline consistency. It must see a producer-for file in the producers' definition that is consistent with the consumer-for file in the consumer's definition. It will detect and reject cycles as well as collisions.

The pipe linking a set of producers with a consumer is created and stored at run-time in a s6-fdholder-daemon instance managed by an automatically generated longrun service named s6rc-fdholder.

Compiled database usage

Once it has been built with s6-rc-compile, a compiled database cannot be modified. It (i.e. the directory containing the n, db and resolve.cdb files as well as the servicedirs subdirectory) can be moved around in the filesystem, provided the insides remain untouched.
Once a compiled database is live (i.e. it has services running on it, following an invocation of s6-rc-init or s6-rc-update), it must not move anymore, and it must not be deleted. The only way to "free" such a compiled database for displacement or deletion is to replace it as the live one via another call to s6-rc-update.
No matter what user compiles the database, only root and the user owning the supervision tree at run-time will be able to operate the compiled database with the s6-rc command.

A complete example

The examples/source subdirectory of the s6-rc package contains a set of service definition directories, which is actually a working, valid set for a Linux system running busybox and the skarnet.org packages; of course, only the service definition set has been kept, and private information has been removed, so it won't work out-of-the-box without the proper specific files, notably configuration in /etc - but nevertheless, you can browse the source and understand what it does, and adapt it to your own needs. It will compile as is with s6-rc-compile, and you can examine the resulting compiled database with s6-rc-db.