>I believe I did my homework looking first -- are there other discussion
>channels than this list that one should be aware of?
The lists are definitely the only place you *should* be aware of, but
there are a lot of informal spaces where discussions happen, because not
everyone is as well-behaved as you are :) Github issues, webforums of
other projects, IRC channels, etc.
The important stuff normally only happens here, but I'm getting user
feedback from several sources.
>I'd go out a limb and say if you only support single-binary mode, some
>of the code could be simplified further by sharing some argument
>handling, but it's hard to do simpler than your exlsn_main wrapper so
>it'll likely be identical with individual programs not changing at all,
>with just an extra shim to wrap them all; it's not like busybox where
>individual binaries can be selected so a static wrapper would be dead
>simple.
I doubt much sharing would be possible.
The main problem I have with multicall is that the program's
functionality changes depending on argv[0]. You need to first select
on argv[0], and *then* you can parse options and handle arguments.
Note that each exlsn_* function needs its own call to subgetopt_r(),
despite the options being very similar because they all fill an
eltransforminfo_t structure.
Having a shim over *all* the execline programs would be that,
multiplied by the number of programs; at the source level, there would
not be any significant refactoring, because each program is pretty much
its own thing. An executable is its own atomic unit, more or less.
If anything, execline is the package that's the *least* adapted to
multicall because of this. There is no possible sharing between
"if" and "piperw", for instance, because these are two small units with
very distinct functionality. The only way to make execline suited to
multicall would be to entirely refactor the code of the executables and
make a giant library, à la busybox. And I am familiar enough with
analyzing and patching busybox that I certainly do not want to add that
kind of maintenance nightmare to execline.
Anything that can be shared in execline is pretty much already shared
in libexecline. If you build execline with full shared libraries, you
get as much code sharing as is reasonably accessible without a complete
rearchitecture.
Any significant disk space you would gain in a multicall binary
compared to a bunch of dynamically linked executables would come from
the deduplication of unavoidable ELF boilerplate and C run-time, and
that's basically it.
The "one unique binary" argument applies better to some of my other
software; for instance, the latest s6-instance-* additions to s6.
I considered making a unique "s6-instance" binary, with varying
functionality depending on an argv[1] subcommand; I eventually decided
against it because it would have broken UI consistency with the rest of
s6, but it would have been a reasonable choice for this set of programs
-
which are already thin wrappers around library calls and share a lot
of code. Same thing with s6-fdholder-*.
execline binaries, by contrast, are all over the place, and *not* good
candidates for multicall.
>Hmm, I'd need to do some measurements, but my impression would be that
>since the overall size is smaller it should pay off for any pipeline
>calling more than a handful of binaries, as you'll benefit from running
>the same binary multiple times rather than having to look through
>multiple binaries (even without optimizing the execs out).
Yes, you might win a few pages by sharing the text, but I'm more
concerned about bss and data. Although I take some care in minimizing
globals, I know that in my typical small programs, it won't matter if
I add an int global, because the amount of global data I need will
never reach 4k, so it won't map an extra page.
When you start aggregating applets, the cost of globals skyrockets.
You need to pay extra attention to every piece of data. Let me bring
the example of busybox again: vda, the maintainer, does an excellent
job of keeping the bss/data overhead low (only 2 pages of global
private/dirty), but that's at the price of keeping it front and
center, always, when reviewing and merging patches, and nacking stuff
that would otherwise be a significant improvement. It's *hard*, and
hampers code agility in a serious way. I don't want that.
Sure, you can say that globals are a bad idea anyway, but a lot of
programs need *some* state, if local to a TU - and the C and ELF models
make it so that TU-local variables still end up in the global data
section.
>Even almost 1MB (the x86_64 version that doesn't have the problem,
>package currently 852KB installed size + filesystem overhead..) is
>still something I consider big for the systems I'm building, even
>without the binutils issue it's getting harder to fit in a complete
>rootfs in 100MB.
I will never understand how disk space is an issue for execline and s6.
RAM absolutely is, because RAM is expensive.
CPU absolutely is, because it uses power.
But disk space?
I have 128 GB on my goddamn phone and 1 TB on my goddamn router.
People build Electron apps with a full web engine embedded in it and
ship them like candy.
Projects on GitHub vendor dependencies like their lives depend on it
and ship 100 MB executables.
Yesterday I built nftables, yes, the thing that's supposed to be super
low-level, super close to the Linux kernel, and easily installable on
anything that needs packet filtering and network control, so, routers,
supposedly relatively small machines. Well, the static nftables binary,
linked against musl, is 1.2 MB large.
wat.
A dynamically linked bash binary is 1.2 MB as well.
wat.
Do you honestly find it fair to ask me to jump through hoops so that
all the execline binaries together may use less than 1 MB?
Isn't there anywhere else on your system where the fruit is lower
hanging, and you could have better gains with less effort? In
particular,
effort that *I* wouldn't have to do?
Look, I'm on your side. I love minimalism, too. I build small systems.
I like small systems. I don't especially like that execline takes up
1 MB, or that a full skarnet.org stack is around 5 MB. That's a lot.
But my old router still uses a 32 MB CF disk, and there's a fully
usable system on it, including a full s6 installation, with room to
spare.
So I'm thinking, maybe my software isn't the biggest offender here.
And I'd rather focus my energy on keeping the software good, readable,
easily fixable when a bug is reported, etc. etc., knowing that I have
already made a *reasonable* effort to keep the software *reasonably*
small for the functionality it provides, up to the point where doing
more would net very much diminishing returns. Going further than that
would not be a good trade-off for most people (my own mental health
included) in the long term, even if it *might* be good for you in the
short term.
>Just looking at the s6 suite (s6, s6-rc, execline, skalibs,
>s6-linux-init) I'm looking at a 3MB increase (because I won't be able to
>get rid of openrc for compatibility with user scripts it'll have to live
>in compat hooks...) ; being able to shave ~700KB of that would be
>very interesting for me (number from linking all .c together with a
>dummy main wrapper, down 148KB)
Have you measured how much disk space OpenRC uses? (I'm not even
talking about RAM usage here, it's too easy a target. :P)
Have you asked the OpenRC people to modify their software so it uses
less disk space?
Why not?
Why is it more acceptable to ask that of s6?
I understand that s6 is supposed to be better than everything else,
yes, but in order to achieve that, it needs to pick its fights. I have
chosen to prioritize correctness, CPU usage, RAM usage, automatability,
and a few other battlefields; disk space is in the list, but far behind.
It makes things more difficult for you, and I'm sorry, but I don't
think my choice is wrong.
>(s6-* dozen of binaries being another similar target and would shave a
>bit more as well, build systems being similar I was hoping it could go
>next if this had been well received)
The problem here is that there are conflicting wants.
People who build embedded devices - like you, I assume - need smaller
software, possibly at the expense of functionality, so they want finer
granularity. That's my natural inclination, too.
But most other users don't care so much about size, and they want
simplicity and usability, which mean coarser granularity. A normie
doesn't
understand the difference between s6, s6-rc and s6-linux-init, and
shouldn't have to; for them, an integrated init system doing all that
would be better.
Distributions cater to the latter, in general, so since my long-term
goal is to get the s6 ecosystem better integrated in distributions, I
tend to not care as much about fine granularity as I did in past years.
That's why I don't have many scruples adding extra s6-* programs to the
s6 package. But there are distinct thematic groups of s6 programs that
work together, and if you want to do away with one group, you can
do it without impacting the rest too much.
>It's a step further, but I don't think it's orthogonal.
>If all the code is in a single binary you could have internal priorities
>to builtins easily as there would be no need to mess with PATH or a
>separate install prefix.
But that's the point: I *want* execline to be a toolbox of Unix
command-line tools that are usable from the outside. I do *not* want
execline binaries to be relegated under the execlineb launcher.
"Internal priorities to builtins" doesn't make sense for execline,
because there's no "internal", it's all external. That's what execline
is for.
>I think you're underestimating what users who haven't used a unix before
>can do though; I can already picture some rummaging in /bin and
>wondering why posix-cd "doesn't work" or something... We get impressive
>questions sometimes.
Of course. But why would this be a problem?
Red Hat has shipped a /usr/bin/cd binaries for ages and nobody has
blinked. The existence of this cd binary, mandated by POSIX, is the
very reason why I bothered writing posix-cd. It's useless, it's in
the spec. Yay.
There is a /usr/bin/[ file in your filesystem. Doesn't that raise
questions? Because it should. It's a very bad hack.
Doesn't /sbin/fsck raise eyebrows?
System software isn't designed with first-time users in mind. For
obvious reasons.
--
Laurent
Received on Wed Feb 01 2023 - 11:41:39 CET