Re: Unexpected behavior when updating exclusive policy package

From: Laurent Bercot <ska-skaware_at_skarnet.org>
Date: Tue, 20 Oct 2020 12:30:04 +0000

  (Posting my answer to the mailing-list as well afterall, because I'm
not revealing any trade secrets, just general s6-rc behaviour, so it may
be useful to readers who are wondering about s6-rc-update mechanisms.)


>After adding a new service to our exclusive policy package we're
>observing restarts of some services (oneshots to be precise) when
>updating the package on our servers, and I wanted to ask if this is
>expected behavior (because I did not expect it ^_^;).

  Yes, it is expected: it is how s6-rc-update (called by the
"svctl update" script) operates.

  When s6-rc-update switches databases, it computes how to achieve the
same state with the new set of dependencies. When a service that was
up in the old database has grown a new dependency, all the dependency
chain is restarted, because there is no other way to be sure that
the services that are currently up are going to work correctly if the
new dependency is started *after* they're already up; that the current
version of the new service is running with all the features you need
after you've added the dependency; and so on.

  s6-rc-update always errs on the side of caution, and if it cannot be
certain that all the services have been brought up in the correct
order according to the dependency chains, it restarts the necessary
parts of the graph.

  There is a way to know in advance what services are going to be
restarted: "s6-rc-db -n newcompileddb". The -n flag means a dry run,
where the "s6-rc change" operations that bring parts of the service
graph down, then up, are printed to stdout instead of actually
performed. You could probably add that call to "svctl update" if you
need a more verbose understanding of the update operation details.


>which would indicate that the restarts follow the dependencies between
>the services.

  Yes, this is what happens.


>It seems that inserting the new service (cryptdisks) triggers a restart
>of the service that depends on it (mount-local), which then triggers a
>restart of the service(s) depending on mount-local and so on.
>
>Is my understanding correct, and this behavior expected?

  Yes, your understanding is correct, and the behaviour is expected.

  It was a difficult call to make during the design of s6-rc-update.
I understand that restarting whole chains of services can be annoying,
especially if it comes as a surprise (which is why the -n flag was
added: so the user can check what is going to happen beforehand and
avoid unpleasant surprises).
  But in the end, I arbitered - with the always helpful insights of
Colin,
thanks again Colin - that having 100% correctness at all times was more
important, and more in the spirit of s6-rc, than avoiding the
inconvenience of occasionally restarting services that may not need it
(since s6-rc-update has no way of knowing whether or not they do).

  The thing is, inserting services very early in the graph should be a
rare operation: I hope you're not going to modify things under the
essential bundle too often in production :) So, most of the time, the
restart chain should be small enough that it's not a noticeable
drawback.


>Also, we did not add the new service to the essential-before-network
>bundle (yet). Do we need to do that, or is updating the bundle optional?

  IIRC, the top bundle calls the essential bundle, which contains the
essential-before-network bundle, so essential-before-network contains
the services that are explicity asked to be brought up at boot time.
If you want cryptdisks to be explicitly up, then you should add it to
the bundle, for clarity.
  However, if cryptdisks is a dependency of another service that is in
the essential-before-network module (such as mount-local), then it
will be brought up anyway, so there is no real need to add it to the
bundle. It's your call, really.

  The main difference is, if you want to bring essential-before-network
*down*, then having cryptdisks in it would also bring down cryptdisks,
whereas not having cryptdisks in it would leave cryptdisks up (unless
it depends on another service in essential-before-network, in which
case it would also be brought down according to the dependency graph).
But since you're never going to bring down essential-before-network
explicitly (except at shutdown time, but it doesn't matter because
*all* services are brought down then), then it really doesn't change
much whether you add cryptdisks to the bundle or not. I like to have it
there for completeness and theoretical correctness, but in practice you
should see no difference.

--
  Laurent
Received on Tue Oct 20 2020 - 12:30:04 UTC

This archive was generated by hypermail 2.3.0 : Sun May 09 2021 - 19:38:49 UTC