Re: s6-rc design ; comparison with anopa

From: Olivier Brunel <jjk_at_jjacky.com>
Date: Fri, 24 Apr 2015 21:05:04 +0200

On 04/23/15 17:40, Laurent Bercot wrote:
(...)
> Three kinds of services
> -----------------------
>
> Like anopa, s6-rc works internally with two kinds of services: longrun,
> which is simply defined by a service directory that will be directly
> managed by s6, and oneshot, which is defined by a directory containing
> data (a start script, a stop script, and some optional stuff).
>
> s6-rc allows the user to provide a third kind of service: a "bundle".
> A bundle is simply a set of other services. Starting a bundle means
> starting all the services contained in the bundle.
> A bundle can be used to emulate a SysV runlevel: the user can put all the
>
> services he needs into a single bundle, then tell s6-rc to change the
> machine
>
> state to "exactly that bundle".
> Bundles can of course contain other bundles.
>
> A oneshot or a longrun are called atomic services, as opposed to a bundle,
> which is not atomic.
> Bundles are useful for the user, because "oneshot" and "longrun" are
> often too small a granularity. For instance, the "Samba" service is made
> of two longruns, smbd and nmbd, but it's still a single service. So,
> samba would be a bundle containing smbd and nmbd.

FYI, this is kinda possible with anopa, since one can have "empty"
services, i.e. a servicedir without any script (no run, start nor stop).
Those can be used to order things, but one could also have such a
service (treated as oneshot internally, just without start or stop
script) with a few dependencies.
Then, starting it will start all the dependencies in order, as you'd expect.

However, there isn't anything runlevel-like, as your "switch to that
bundle" command. (One can add "aa=foobar" on the kernel cmdline to use
/etc/anopa/onboot/foobar as listdir to aa-start (during stage 2) instead
of /etc/anopa/onboot/default but that's it).


On that note, one thing you've apparently done/planned is auto-stopping,
whereas there is no such thing in anopa. This is because I always felt
like while auto-starting can be easily predictable/have expected
behavior, things aren't the same when it comes to stopping.

That is, start httpd and it will auto-start dependency php which will
auto-start dependency sqld; fine. But I'm not sure stopping sqld should
stop php, despite the dependency. Maybe one just wants to shut down
sqld, not the whole webserver.

Then there's also the case of, imagine you start foobar & web. web is a
bundle with httpd, php & sqld, foobar is a service w/ a dependency on sqld.
Now you stop web; should sqld be stopped? about foobar? What is the
expected behavior here? I'm not sure there's one, different
people/scenario will have different expectations... it always seemed too
"complicated" so I went with no auto-stopping at all.


> Also, the smbd daemon itself could want its own logger, smbd-log.
> Correct daemon operation depends on the existence of a logger (a daemon
> cannot start if its logger isn't working). So smbd would actually be a
> bundle of two long-runs, smbd-run (which is the smbd process itself) and
> smbd-log (which is the logger process), and smbd-run would depend on
> smbd-log.

Just so I understand: why are you talking about smbd-log as a separate
service, and not the logger of service smbd, as usual with s6? Or is
that what you're referring to here with smbd-log?

> Users who want to start Samba don't want to deal with smbd-run, smbd-log,
> nmbd-run and nmbd-log manually, so they would just start "samba", and
> s6-rc would resolve "samba" to the proper set of atomic services.
>
>
> Source, compiled and live
> -------------------------
>
> Unlike anopa, s6-rc does not operate directly at run-time on the
> user-provided service definitions. Why ? Because user-provided data is
> error-prone, and boot time is a horrible time for debugging. Also, s6-rc
> uses a complete graph of all services for dependency management, and
> generating that graph at run-time is costly.
>
> Instead, s6-rc provides a "s6-rc-compile" utility that takes the
> user-provided service definitions, the "source", and compiles it into
> binary form in a place in the root filesystem, the "compiled".
>
> At run-time, s6-rc ignores the source, but reads its data from the
> compiled, which can be on a read-only filesystem. It also needs a
> read-write place to maintain information about its state; this place is
> called the "live". Unlike the compiled, the live is small: it can reside
> in RAM.

I'm not sure where you put the scandir in this? I believe "original"
servicedirs where taken from the source and put into the compiled, and
s6-rc will create the actual servicedirs in the scandir with s6-rc-init,
correct? So that would be part of live?
How about the oneshot scripts?

> The point of this separation is multifold: efficiency (all checks,
> parsing and graph generation performed at compile-time), safety (the
> compiled can be write-protected), and clarity (separation of user-
> modifiable data, current configuration data, and current live data).
>
> Atomic services can be very small. It can be a single line of shell
> for a oneshot, for instance. I fully expect package developers to
> produce source definitions with multiple atomic services (and dependencies
> between those services) and a bundle representing the whole package.
> I expect the total number of atomic services on a typical reasonably
> loaded machine to be around a thousand. Yes, it can grow very fast -
> so having a compiled database isn't a luxury.
>
>
> Run-time
> --------
>
> At run-time, s6-rc only works in *stage 2*.
> That is important, and one of the few things I do not like in anopa:
> stage 1 should be completely off-limits to any tool.

Well, first off, anopa provides scripts for all stages, but one is free
to use them as-is, tweak them, or not use them at all. One could also
use anopa's tool on a non-PID1 setup as well.

Also, with anopa I wanted to fill what's missing in s6, and that
included a full init system, so stage 1. You originally said s6-rc was
meant to be a full init system, but now you're saying you have another
tool/package in mind for that, s6-init. That's fine, but with anopa I
wanted the whole thing, yes.


There there's the whole nothing in stage 1 argument, yes aa-enable is
triggered there in anopa. One could easily move it to stage 2 before
aa-start, it would work much the same really.
I was just lazy I guess, and since I needed to create the runtime
repository/s6 scandir, I did it all at once with aa-enable in stage
1. But stage 1 could also just copy an empty scandir (save the
catch-all) and let aa-enable fill it up on stage 2...



One thing though, your compilation process only does copy servicedirs,
or is there more (besides the whole "packing" them into a binary form
alongside dependency graphs)?
What I like with aa-enable is that servicedirs are created from a source
servicedir and some config data. So one can have a single openvpn_at_
source servicedir, and use it to create different servicedirs with
different configs. There's also the bits to easily use the same logger,
but also service-specific log script...


(...)

> Live updates
> ------------
>
> There's a complex thing that anopa more or less evades but that I feel is
> necessary in order to be adopted by a distribution: live updates. I'm not
> exactly sure yet how to proceed, but I have a vague idea, and I would like
> more input on the subject.
>
> Users will upgrade their packages. They will sometimes need to restart
> longrun services. If not much has changed, it's easy: they can do it with
> s6-svc without touching the global state, so it's not s6-rc's or anopa's
> concern. However, sometimes things change: new daemons are introduced,
> new dependencies are introduced, etc.
>
> My view is that packages should provide source definitions, and after an
> update, the distribution should invoke s6-rc-compile again. This is easy
> enough, but then the live state does not match the current compiled
> service database anymore. anopa has a similar problem with its current
> service repository.

Right. I haven't thought about this that much, but my general feeling is
that I'm not for auto-removing anything, so if foobar is updated and
doesn't depend on foo anymore, but adds a new dependency on bar, adding
& starting bar is obviously good; Removing or even stopping foo on the
other hand, I'm not so sure.

So if aa-enable had an option to update an existing servicedir (instead
of failing because it already exists), it might be enough to aa-enable &
aa-start the updated service (dependencies being auto-added already).


> I am thinking about a utility, "s6-rc-update", that would take the live,
> the old compiled and the new compiled as inputs, and that would update
> the live as smartly as possible, with carefully designed heuristics;
> users could also tell s6-rc-update exactly what to do via annotations in
> the source, that s6-rc-compile would translate into the new compiled.
>
>
> Tricky implementation details
> -----------------------------
>
> What good is a new init system if it's vulnerable to the old sysvrc
> pitfalls ? :P
>
> One of the main issues with sysvrc is that scripts are run as scions of
> the invoking shell. So, a sysvrc script run by boot scripts isn't run with
> the same environment as the same script run manually by an admin, and
> this is very difficult to harden.
>
> Supervision suites solved that problem for longrun services. Since
> daemons are started by the supervision tree, and never by an admin's shell
> or the boot scripts' shell, they are always started with a reproducible
> environment.
>
> But what about oneshot services ? What about "start" and "stop" scripts ?
> anopa actually runs them as children of "aa-start" and "aa-stop".
> Which *may* be just as problematic. We need a way to run oneshot scripts
> in the same reproducible manner as daemons.
>
> s6-rc does this. Every s6-rc script invocation will be reproducible.
> I'll let you guys think a little about how it does it; I'm both very

...it uses a longrun service to spawn things, a la s6-sudo or something?

> proud and very disgusted by the solution. If you manage to guess how
> s6-rc does it, it means that your mind is just as warped as mine; but
> no matter whether you think that's genius or that's horrible, or both,
> it's something anopa does not. :)
>
>
> Nice things anopa does
> ----------------------
>
> There are a lot of nice things anopa does, and that I may shamelessly
> copy if Olivier accepts: for instance, all the terminal manipulation.
> Progress bars are shiny. :)
>
> However, I won't add progress bars to s6-rc if it makes it significantly
> more complex (read: if it really needs heap memory). And I don't
> understand the need for the pipe from aa-start to the start script:
> what kind of information does aa-start give its child ? If you remove
> that pipe, you can run the start script with the user's stdin, so you
> don't have to add noecho support to aa-start - asking for passwords can
> be performed entirely by the start script.

But what happens if two oneshots started/running in parallel both want
user input? How does that work in a manner that isn't confusing/unclear?
Yeah the pipe is used so aa-start will process all password inputs
sequentially, and the user will always know where she's typing because
(as for regular output) the prompt is prefixed with the service name.


> I would also like to hear more about the "wants" dependencies. Is that
> a thing ? What does it mean exactly ? And what is the point of
> oneshots not marked "essential" ? Generally speaking, anopa separates
> service ordering and service dependencies; I would like to hear more
> about the goal of that differentiation. Is s6-rc's hard dependency model
> ("if A depends on B, then B will start first, and A will not start if B
> can't be successfully started") insufficient ? Could I have some real-life
> examples of this ?
>
> ... and that was more than long enough for a first post on the subject.
> Thanks for having read that far :)

The wants dependencies are just a way to pull a service without failing
if that service fails to start, some sort of "optional dependency" I guess.
E.g. one could say that php only wants sqld, so even if the later can't
be started the website can work, and everything that doesn't need db
will work, the rest failing with db connection errors, instead of having
no website at all.

Separating order & dependency also allows to set up some order in
services in case, but without the need for them to be actual
dependencies, which may also be handy for source servicedirs to plan for
different uses/configs.

Re: essential, the point of that file is only so that on boot, if e.g.
no getty could be started a shell is opened, otherwise you'd have no way
of logging in. But if it's only sqld that failed, no need for that you
can login as usual. (Similarly on boot, if mounting root-fs fails open a
terminal, if something else did but root-fs was mounted, no need to stop
& open a terminal.)
Received on Fri Apr 24 2015 - 19:05:04 UTC