Re: Inteligent service restart with s6

From: Jan Olszak <jan.olszak_at_vestiacom.com>
Date: Mon, 25 Jul 2016 14:55:58 +0200

Oh, I didn't know about ./finish. Thanks!

The idea with a separate watchdog is tempting..

Is it possible to use variables stored in the /env directory for this
purpose or are they overwritten by s6 on every service start?

On Mon, Jul 25, 2016 at 1:55 PM, Laurent Bercot <ska-skaware_at_skarnet.org>
wrote:

> On 25/07/2016 13:18, Jan Olszak wrote:
>
>> - Configurable sleep time before service is restarted (configured per
>> service)
>> - This interval should grow in case of consecutive crashes
>> - After multiple crashes the watchdog must try to restart a crashed
>> service
>> at least once every 24 hours
>> - Current "interval" and number of crashes should be accessible
>>
>>
>> I'm not sure how to implement this, do you have any suggestions?
>>
>
> Those are all things that can be done in the run and finish scripts.
>
> You can use the arguments to ./finish to check whether a service has
> exited normally, exited abnormally or crashed. You can then either sleep
> in the finish script (provided you have set a bigger timeout-finish first),
> or - my preferred solution - invoke some program of yours that will modify
> something in the filesystem so the crash is reported. Then in the ./run
> script, before executing into the service, you check the "crash report"
> and deduce the interval you want to sleep for, etc. etc. and perform the
> actions you need before the final exec.
>
> I haven't added that to s6 because it's pure policy: it can be achieved
> with very basic scripting in ./run and ./finish.
>
> If you have a watchdog of any sort, however, I recommend that you make it
> a separate service (say, foo-watch if your main service is foo) and adjust
> your foo/run and foo/finish scripts to send the appropriate reports to
> foo-watch. If the watchdog decides it needs to restart foo, then it can
> just
> invoke "s6-svc -t foo". Or it can decide to keep foo down for a while
> (s6-svc -d foo) before starting it (s6-svc -u foo). I would advise for a
> separate foo-watch service controlling the state of foo if you have some
> complex policy to maintain.
>
> --
> Laurent
>
>
Received on Mon Jul 25 2016 - 12:55:58 UTC