Re: "Back off" setting for crashing services with s6+openrc?

From: Oliver Schad <oliver.schad_at_automatic-server.com>
Date: Fri, 30 Sep 2022 11:34:40 +0200

On Mon, 26 Sep 2022 23:00:48 +0000
Colin Booth <colin_at_heliocat.net> wrote:

> On Thu, Sep 22, 2022 at 01:21:46PM -0700, John W Higgins wrote:
> > run_my_crashing_app || sleep 10
> >
> > Work? The run script will sit there for 10 seconds if your app
> > fails. Not built in - but should accomplish the task pretty easily.
> >
> > John W Higgins
> Put the backoff in the finish script. The API is to run finish with
> the exit code and signal (if appropriate) as $1 and $2 respectively.
> With that information you can have finish make decisions about if it
> should delay restarting, set the service down in the face of a
> permanent error, or so on. Note that by default finish has a five
> second deadline so if you want to delay a restart for ten seconds
> you'll need to increase that deadline with a timeout-finish file.
 
Sounds in theory good, but in practice I would split the whole thing in
2 parts:

- define start timings in "finish"
- do start timings in "run"

Why? I expect stopping more or less immediatly and the timeout of 5
seconds of "finish" meets my expectations.

Of course it's not the right semantic of starting, that delaying is
part of start. However, with the current features of s6 it's more
logical to extend the starting phase, because it sleeps some time.

But I'm still not happy with that.

Uhm, don't know, if it would work: could we model that delay as a
service dependency? So if the dependency is up, the service itself can
be started.

Inside of the finish script of the real service, you would write the
stats, which controls the behaviour of the delay service.

So we had a "delay service", and a real service and you wouldn't had to
model some unvisible magic inside of run scripts. I.e. if you would
call the service "delay-myapp" you would exactly see, why your
service doesn't start, now. I was thinking about that issue, if you
aren't familiar with a setup, what would you expect as an administrator?

I wouldn't expect delay magic in run or finish scripts. I mean, imagine
you have to debug why a service doesn't start. Reading the run or
finish scripts for that is not that fancy.

Best Regards
Oli


-- 
Automatic-Server AG •••••
Oliver Schad
Geschäftsführer
Hardstr. 46
9434 Au | Schweiz
www.automatic-server.com | oliver.schad_at_automatic-server.com
Tel: +41 71 511 31 11 | Mobile: +41 76 330 03 47

Received on Fri Sep 30 2022 - 11:34:40 CEST

This archive was generated by hypermail 2.4.0 : Fri Sep 30 2022 - 11:35:36 CEST