From 903059dfb0dc520f0618062bbbafac4ebd8e29f8 Mon Sep 17 00:00:00 2001 From: Laurent Bercot Date: Mon, 22 May 2017 22:35:47 +0000 Subject: Add index.html, rename documentation to doc, clean up --- README.md | 2 +- doc/DEPENDENCIES-BUILD.md | 175 ++++++++++++++++++++++++++++++++++++ doc/INTERNALS.md | 106 ++++++++++++++++++++++ doc/index.html | 92 +++++++++++++++++++ documentation/DEPENDENCIES-BUILD.md | 175 ------------------------------------ documentation/INTERNALS.md | 106 ---------------------- 6 files changed, 374 insertions(+), 282 deletions(-) create mode 100644 doc/DEPENDENCIES-BUILD.md create mode 100644 doc/INTERNALS.md create mode 100644 doc/index.html delete mode 100644 documentation/DEPENDENCIES-BUILD.md delete mode 100644 documentation/INTERNALS.md diff --git a/README.md b/README.md index b9b9c73..0714556 100644 --- a/README.md +++ b/README.md @@ -173,7 +173,7 @@ This means the toolchain from your BUILD to your TARGET, even if BUILD and TARGET are the same. HOST_HOST_BASE is the path where your native toolchain for the TARGET -is installed. TODO: rename this variable +is installed. COUNTRY_CODE, LOCAL_IP and ROUTER_IP are configuration variables for your TARGET. COUNTRY_CODE is one of `uk`, `fr`, `rs`, `vn` or `cn`. diff --git a/doc/DEPENDENCIES-BUILD.md b/doc/DEPENDENCIES-BUILD.md new file mode 100644 index 0000000..e1d97de --- /dev/null +++ b/doc/DEPENDENCIES-BUILD.md @@ -0,0 +1,175 @@ + +# lh-bootstrap: software built for the BUILD machine + +Laurent Bercot +2016-03-31 + + +This file documents the software installed and run on the BUILD +machine prior to building the HOST image. + +Please read the INTERNALS.md file first, for the general organization +of the build, and basic definitions. + + +## BUILD tools + +### Linux kernel headers + +Makefile directory: sub/kernel + +The Linux kernel is downloaded and will be configured and compiled +to boot a qemu image for the HOST. Since it will be downloaded +anyway, we reuse the source to process and install the kernel headers +for the BUILD. +Those kernel headers, coupled with the musl libc's headers, are +necessary to compile Linux-specific software such as util-linux and skarnet-org. + + +### musl libc + +Makefile directory: sub/musl + + We have no control over the BUILD's native compiler and libc. Most +likely, it's gcc and produces binaries that are dynamically linked +against the glibc - but we're not certain; we would like certainty, +even for the build tools. We do not want our tools' behaviour to +depend on external factors such as a misconfigured libc or dynamic +linker. + + So, we download the musl libc (which we would download for use in +the HOST anyway) and compile it for the BUILD. We then link all our +BUILD tools against it. + + +### skarnet.org packages + +Makefile directory: sub/skarnet.org + + The HOST uses s6-rc as its service manager. We provide a template +for the database in source format in `layout/rootfs/etc/s6-rc/source-base`; +this template is preprocessed and added to the rootfs at layout +installation time, at the beginning of the HOST build. + However, in order to boot, the HOST needs the database in compiled +form, not in source form: so we must run s6-rc-compile before the HOST +boots. Since the source and compiled formats are platform-independent, +we just run s6-rc-compile on the BUILD. Which means we need to compile +s6-rc for the BUILD, with the same settings that the HOST is using. +So we end up compiling most of the skarnet.org stack. + + Since we have to compile skalibs anyway, which is by far the heaviest +component of the stack, we also use the opportunity to compile +s6-portable-utils for the BUILD: the time spent compiling this package +is negligible once skalibs is built, and it contains +alternative tools that we use subsequently in the build, because their +behaviour is more predictible than the tools provided by the BUILD's +distribution. + + Note: since we need to mirror the HOST's layout for s6-rc-compile +to work properly, we compile the skarnet.org stack following the +slashpackage convention, with --enable-slashpackage. However, we +obviously don't install a slashpackage hierarchy on the BUILD's root +filesystem, we use the $(OUTPUT)/build-build staging directory. +The consequence is that skarnet.org binaries that exec other binaries +via slashpackage paths will not work. This is ok for our use since +the main tool we need is s6-rc-compile, which does not exec anything +else, but it's something to keep in mind. It's the reason why we do +not use s6-setuidgid even after building s6: we stick to the hackish +and inefficient bin/setuidgid script to drop privileges, because our +temporary installation of s6-setuidgid simply does not work. + + +#### skalibs + + The library which all other skarnet.org packages depend on. + + +#### execline + + The scripting language used by s6 and s6-rc. + + +#### s6 + + The supervision suite used by s6-rc. + + +#### s6-rc + + The service manager used by the HOST. We compile it for the BUILD in +order to use s6-rc-compile to compile the service database before +booting the HOST. + + +#### s6-portable-utils + + Some utilities are akin to POSIX tools, but will have reproducible behavior +no matter what distribution is used. We have had trouble with +differences across BUILD distributions, with some distributions +slightly deviating from the standard (looking at you, Ubuntu); using +our own tools is insurance against that. + + +### util-linux + +Makefile directory: sub/util-linux + + To make the qemu image, we need losetup -P, to set up a loopback +mount that supports partitions. But the -P option to losetup only +appear in latest versions of util-linux, and not all distributions +ship a recent enough version. (Looking at you, Ubuntu and Debian +stable.) + So we download and build util-linux. Except the util-linux +build system is a bloated plate of noodles, that can have a lot +of dependencies - in particular a dependency to ncurses, and we +DO NOT want to build ncurses if it can be avoided. Fortunately, +none of the tools we need require ncurses. So we end up building +those individual binaries from util-linux and avoid pulling in +the kitchen sink. + Currently, the binaries we build are: losetup, fdisk, mkswap, +mount, umount. This list can change as the package evolves; the +current list is described in the UTLX_PROGLIST variable definition +in the sub/util-linux/Makefile file. + + +### xz-utils + +Makefile directory: sub/xz + + xz-utils includes another compression library (liblzma), which +is also a dependency of kmod - actually, this is the one that +interests us. So we have to build the xz-utils package for +BUILD. + + +### kmod + +Makefile directory: sub/kmod + + Ah, kmod. + + We build the Linux kernel for HOST with module support, for +practicality. Modules are compressed, to save storage space. +Traditionally, there are compressed with gzip (and have extension +`.ko.gz`), but xz is generally a better compressor than gzip: +it decompresses faster and the compressed data is smaller. So +we use xz to compress the modules (extension `.ko.xz`). On the +HOST, we load the modules with busybox modprobe, which supports +both extensions. So far, so good. + + Except that xz support for kmod is relatively recent, and some +distributions insist on providing an ancient version of kmod, +which *does not* allows modules to be compressed with xz. +(And the kernel's build system does not report the error - the +modules silently fail to be installed, which makes diagnostic +fun!) + + So, we have to provide our own version of kmod. + + I have to say that kmod is the single worst package that appears +in this whole build. The software itself works, but the +build system is *extremely* buggy and requires several workarounds, +that have all been implemented in the Makefile. Please do not +attempt to "simplify" this Makefile by using "correct" configure +options and eliminating make variables: that will not work. + diff --git a/doc/INTERNALS.md b/doc/INTERNALS.md new file mode 100644 index 0000000..ab7aa54 --- /dev/null +++ b/doc/INTERNALS.md @@ -0,0 +1,106 @@ + + + +# Internals of lh-bootstrap + +Laurent Bercot +last modified: 2017-04-07 + +## Definitions + +BUILD is the machine you're running this set of scripts on. +HOST is the machine you're building the image for. +TARGET has the same meaning as HOST. See the `README.md` file. + + +## General organization + +### ./lh-config and ./make + +All your configuration should happen in the `lh-config` file. It is a series +of variable assignments, in shell syntax. + +`./make` is a shell script that reads `lh-config`, provides reasonable +defaults for variables not specified in `lh-config`, exports a set of +variables into the environment then executes into `make` with the exact +same command line that it was given. So you can use make options, specify +a make target, etc. + + +### Other files and directories + +The main `Makefile` includes sub-Makefiles that are in the subdirectories +of the `sub` directory, roughly one per software package. Those subdirectories +also may contain various scripts and patches necessary to make the software +package compile and/or run properly. + +The `bin` directory contains scripts that are used throughout the whole +build process. They have been designed for maximum portability, not efficiency. + +The `sysdeps` directory contains "system dependencies" for the various HOSTs +that lh-bootstrap supports, i.e. a textual description of the +HOST architecture's capabilities and quirks, such as endianness, sizes of +certain types, etc. These descriptions are used when cross-compiling the +skarnet.org packages. + +The `layout` directory contains the base layout for all the filesystems +that are used to build our image: + +- *rootfs* is the basic one, the root filesystem. It will be mounted +read-only. +- *rwfs* is a read-write directory we use to store our configuration +and other read-write data for normal operation of the machine. It is +not accessible to "normal" users. For instance, `/var` is a part of +rwfs. +- *userfs* is a read-write directory that will be used to store user +data. For instance, `/home` is a part of userfs. +- *stagingfs* is unused for now. It will be used for safe firmware +updates. + + All the files under `layout` must be text or otherwise editable +files: there must be absolutely no binary files in it. Currently, +the timezone files (`/etc/zoneinfo`) are an exception to that rule; +at some point I will remove them and make a sub-package script to +install them. + + +## Dependencies and build order + +`./make` first builds a set of tools for the BUILD. The goal is to make the +build work on as many BUILD machines as possible, with as few dependencies +as possible. For instance, parts of a recent `util-linux` package are +installed because the build needs the `-P` option to the `losetup` binary +and not all distributions ship a `util-linux` version with `losetup -P`. + +Then `./make` builds the software for the HOST, installing it into the +`rootfs` subdirectory of the output. + + +### Topological list of the BUILD tools + +- Linux kernel headers +- musl +- skarnet.org packages +- some binaries from util-linux +- xz-utils +- kmod + + +### Topological list of the HOST packages + + For now, the HOST packages are a mix of "production" bootstrap packages, +i.e. software that is needed to get a host up and running without +aiming for development on that host, and "development" bootstrap packages, +i.e. software that is needed to turn the host into a native development +platform (in order to build and install software that is difficult to +cross-compile). A better separation between those two sets of packages +is planned for future versions of lh-bootstrap. + +- Linux kernel +- musl +- bearssl +- skarnet.org packages +- busybox +- dnscache (from djbdns) +- dropbear +- a native host toolchain (for now just copied from a location) diff --git a/doc/index.html b/doc/index.html new file mode 100644 index 0000000..d17ff15 --- /dev/null +++ b/doc/index.html @@ -0,0 +1,92 @@ + + + + + + lh-bootstrap - a tool to build images running s6 and s6-rc + + + + + + +

+Software
+skarnet.org +

+ +

lh-bootstrap

+ +

What is it ?

+ +

+ lh-bootstrap is a set of scripts that build a disk image +for a virtual machine such as QEMU. +The image uses Linux, +musl, +BusyBox, and skarnet.org tools, +and similar small and efficient software. It provides the necessary +minimum amount of tools to securely connect to the Internet and +download more software while having a stable init system and service +infrastructure. +

+ +
+ +

Installation

+ +

Requirements

+ +

+ Read the +README.md +file (on GitHub, for Markdown formatting). +

+ +

Licensing

+ +

+ lh-bootstrap is free software. It is available under the +GNU General Public License +version 2. +

+ +

Download

+ +

+lh-bootstrap does not have regular numbered releases; you should always +pull the latest version from Git. +

+ + + +

Compilation

+ + + + +

Related resources

+ + +

lh-bootstrap discussion

+ + + + + diff --git a/documentation/DEPENDENCIES-BUILD.md b/documentation/DEPENDENCIES-BUILD.md deleted file mode 100644 index e1d97de..0000000 --- a/documentation/DEPENDENCIES-BUILD.md +++ /dev/null @@ -1,175 +0,0 @@ - -# lh-bootstrap: software built for the BUILD machine - -Laurent Bercot -2016-03-31 - - -This file documents the software installed and run on the BUILD -machine prior to building the HOST image. - -Please read the INTERNALS.md file first, for the general organization -of the build, and basic definitions. - - -## BUILD tools - -### Linux kernel headers - -Makefile directory: sub/kernel - -The Linux kernel is downloaded and will be configured and compiled -to boot a qemu image for the HOST. Since it will be downloaded -anyway, we reuse the source to process and install the kernel headers -for the BUILD. -Those kernel headers, coupled with the musl libc's headers, are -necessary to compile Linux-specific software such as util-linux and skarnet-org. - - -### musl libc - -Makefile directory: sub/musl - - We have no control over the BUILD's native compiler and libc. Most -likely, it's gcc and produces binaries that are dynamically linked -against the glibc - but we're not certain; we would like certainty, -even for the build tools. We do not want our tools' behaviour to -depend on external factors such as a misconfigured libc or dynamic -linker. - - So, we download the musl libc (which we would download for use in -the HOST anyway) and compile it for the BUILD. We then link all our -BUILD tools against it. - - -### skarnet.org packages - -Makefile directory: sub/skarnet.org - - The HOST uses s6-rc as its service manager. We provide a template -for the database in source format in `layout/rootfs/etc/s6-rc/source-base`; -this template is preprocessed and added to the rootfs at layout -installation time, at the beginning of the HOST build. - However, in order to boot, the HOST needs the database in compiled -form, not in source form: so we must run s6-rc-compile before the HOST -boots. Since the source and compiled formats are platform-independent, -we just run s6-rc-compile on the BUILD. Which means we need to compile -s6-rc for the BUILD, with the same settings that the HOST is using. -So we end up compiling most of the skarnet.org stack. - - Since we have to compile skalibs anyway, which is by far the heaviest -component of the stack, we also use the opportunity to compile -s6-portable-utils for the BUILD: the time spent compiling this package -is negligible once skalibs is built, and it contains -alternative tools that we use subsequently in the build, because their -behaviour is more predictible than the tools provided by the BUILD's -distribution. - - Note: since we need to mirror the HOST's layout for s6-rc-compile -to work properly, we compile the skarnet.org stack following the -slashpackage convention, with --enable-slashpackage. However, we -obviously don't install a slashpackage hierarchy on the BUILD's root -filesystem, we use the $(OUTPUT)/build-build staging directory. -The consequence is that skarnet.org binaries that exec other binaries -via slashpackage paths will not work. This is ok for our use since -the main tool we need is s6-rc-compile, which does not exec anything -else, but it's something to keep in mind. It's the reason why we do -not use s6-setuidgid even after building s6: we stick to the hackish -and inefficient bin/setuidgid script to drop privileges, because our -temporary installation of s6-setuidgid simply does not work. - - -#### skalibs - - The library which all other skarnet.org packages depend on. - - -#### execline - - The scripting language used by s6 and s6-rc. - - -#### s6 - - The supervision suite used by s6-rc. - - -#### s6-rc - - The service manager used by the HOST. We compile it for the BUILD in -order to use s6-rc-compile to compile the service database before -booting the HOST. - - -#### s6-portable-utils - - Some utilities are akin to POSIX tools, but will have reproducible behavior -no matter what distribution is used. We have had trouble with -differences across BUILD distributions, with some distributions -slightly deviating from the standard (looking at you, Ubuntu); using -our own tools is insurance against that. - - -### util-linux - -Makefile directory: sub/util-linux - - To make the qemu image, we need losetup -P, to set up a loopback -mount that supports partitions. But the -P option to losetup only -appear in latest versions of util-linux, and not all distributions -ship a recent enough version. (Looking at you, Ubuntu and Debian -stable.) - So we download and build util-linux. Except the util-linux -build system is a bloated plate of noodles, that can have a lot -of dependencies - in particular a dependency to ncurses, and we -DO NOT want to build ncurses if it can be avoided. Fortunately, -none of the tools we need require ncurses. So we end up building -those individual binaries from util-linux and avoid pulling in -the kitchen sink. - Currently, the binaries we build are: losetup, fdisk, mkswap, -mount, umount. This list can change as the package evolves; the -current list is described in the UTLX_PROGLIST variable definition -in the sub/util-linux/Makefile file. - - -### xz-utils - -Makefile directory: sub/xz - - xz-utils includes another compression library (liblzma), which -is also a dependency of kmod - actually, this is the one that -interests us. So we have to build the xz-utils package for -BUILD. - - -### kmod - -Makefile directory: sub/kmod - - Ah, kmod. - - We build the Linux kernel for HOST with module support, for -practicality. Modules are compressed, to save storage space. -Traditionally, there are compressed with gzip (and have extension -`.ko.gz`), but xz is generally a better compressor than gzip: -it decompresses faster and the compressed data is smaller. So -we use xz to compress the modules (extension `.ko.xz`). On the -HOST, we load the modules with busybox modprobe, which supports -both extensions. So far, so good. - - Except that xz support for kmod is relatively recent, and some -distributions insist on providing an ancient version of kmod, -which *does not* allows modules to be compressed with xz. -(And the kernel's build system does not report the error - the -modules silently fail to be installed, which makes diagnostic -fun!) - - So, we have to provide our own version of kmod. - - I have to say that kmod is the single worst package that appears -in this whole build. The software itself works, but the -build system is *extremely* buggy and requires several workarounds, -that have all been implemented in the Makefile. Please do not -attempt to "simplify" this Makefile by using "correct" configure -options and eliminating make variables: that will not work. - diff --git a/documentation/INTERNALS.md b/documentation/INTERNALS.md deleted file mode 100644 index ab7aa54..0000000 --- a/documentation/INTERNALS.md +++ /dev/null @@ -1,106 +0,0 @@ - - - -# Internals of lh-bootstrap - -Laurent Bercot -last modified: 2017-04-07 - -## Definitions - -BUILD is the machine you're running this set of scripts on. -HOST is the machine you're building the image for. -TARGET has the same meaning as HOST. See the `README.md` file. - - -## General organization - -### ./lh-config and ./make - -All your configuration should happen in the `lh-config` file. It is a series -of variable assignments, in shell syntax. - -`./make` is a shell script that reads `lh-config`, provides reasonable -defaults for variables not specified in `lh-config`, exports a set of -variables into the environment then executes into `make` with the exact -same command line that it was given. So you can use make options, specify -a make target, etc. - - -### Other files and directories - -The main `Makefile` includes sub-Makefiles that are in the subdirectories -of the `sub` directory, roughly one per software package. Those subdirectories -also may contain various scripts and patches necessary to make the software -package compile and/or run properly. - -The `bin` directory contains scripts that are used throughout the whole -build process. They have been designed for maximum portability, not efficiency. - -The `sysdeps` directory contains "system dependencies" for the various HOSTs -that lh-bootstrap supports, i.e. a textual description of the -HOST architecture's capabilities and quirks, such as endianness, sizes of -certain types, etc. These descriptions are used when cross-compiling the -skarnet.org packages. - -The `layout` directory contains the base layout for all the filesystems -that are used to build our image: - -- *rootfs* is the basic one, the root filesystem. It will be mounted -read-only. -- *rwfs* is a read-write directory we use to store our configuration -and other read-write data for normal operation of the machine. It is -not accessible to "normal" users. For instance, `/var` is a part of -rwfs. -- *userfs* is a read-write directory that will be used to store user -data. For instance, `/home` is a part of userfs. -- *stagingfs* is unused for now. It will be used for safe firmware -updates. - - All the files under `layout` must be text or otherwise editable -files: there must be absolutely no binary files in it. Currently, -the timezone files (`/etc/zoneinfo`) are an exception to that rule; -at some point I will remove them and make a sub-package script to -install them. - - -## Dependencies and build order - -`./make` first builds a set of tools for the BUILD. The goal is to make the -build work on as many BUILD machines as possible, with as few dependencies -as possible. For instance, parts of a recent `util-linux` package are -installed because the build needs the `-P` option to the `losetup` binary -and not all distributions ship a `util-linux` version with `losetup -P`. - -Then `./make` builds the software for the HOST, installing it into the -`rootfs` subdirectory of the output. - - -### Topological list of the BUILD tools - -- Linux kernel headers -- musl -- skarnet.org packages -- some binaries from util-linux -- xz-utils -- kmod - - -### Topological list of the HOST packages - - For now, the HOST packages are a mix of "production" bootstrap packages, -i.e. software that is needed to get a host up and running without -aiming for development on that host, and "development" bootstrap packages, -i.e. software that is needed to turn the host into a native development -platform (in order to build and install software that is difficult to -cross-compile). A better separation between those two sets of packages -is planned for future versions of lh-bootstrap. - -- Linux kernel -- musl -- bearssl -- skarnet.org packages -- busybox -- dnscache (from djbdns) -- dropbear -- a native host toolchain (for now just copied from a location) -- cgit v1.3.1