From 0251ba5cc54cdd24092e442ab7ec364b97d42601 Mon Sep 17 00:00:00 2001 From: Laurent Bercot Date: Thu, 21 Sep 2023 05:57:24 +0000 Subject: More doc, complete? Signed-off-by: Laurent Bercot --- INSTALL | 4 +- doc/future.html | 104 ++++++++++++++++++++++++ doc/index.html | 30 +++++-- doc/quickstart.html | 11 +-- doc/tipideed.html | 180 ++++++++++++++++++++++++++++++++++++++++- examples/openrc/httpd-4.initd | 23 ++++++ examples/openrc/httpd-6.initd | 23 ++++++ examples/openrc/httpsd-4.initd | 23 ++++++ examples/openrc/httpsd-6.initd | 23 ++++++ src/tipideed/tipideed.c | 4 +- 10 files changed, 405 insertions(+), 20 deletions(-) create mode 100644 doc/future.html create mode 100644 examples/openrc/httpd-4.initd create mode 100644 examples/openrc/httpd-6.initd create mode 100644 examples/openrc/httpsd-4.initd create mode 100644 examples/openrc/httpsd-6.initd diff --git a/INSTALL b/INSTALL index 67c4439..124ce2b 100644 --- a/INSTALL +++ b/INSTALL @@ -6,8 +6,8 @@ Build Instructions - A POSIX-compliant C development environment - GNU make version 3.81 or later - - skalibs version 2.13.2.0 or later: https://skarnet.org/software/skalibs/ - - (optional but recommended): s6-networking version 2.5.1.3 or later: + - skalibs version 2.14.0.0 or later: https://skarnet.org/software/skalibs/ + - (optional but recommended): s6-networking version 2.5.1.4 or later: https://skarnet.org/software/s6-networking/ This software will run on any operating system that implements diff --git a/doc/future.html b/doc/future.html new file mode 100644 index 0000000..1a8c3e5 --- /dev/null +++ b/doc/future.html @@ -0,0 +1,104 @@ + + + + + + tipidee: the future + + + + + + +

+tipidee
+Software
+skarnet.org +

+ +

tipidee: the future

+ +

+ tipidee is fully functional, and you are encouraged to use it; however, it +is not yet considered complete. There are some optional features +of HTTP that would be nice to have, and that may be implemented at some point +down the line. +

+ +

Ranges

+ +

+ Ranges +are a useful part of HTTP when you are serving big files and connections may +be interrupted and restarted: supporting the Range: header can save +bandwidth, if the client only asks for the parts of the files that it's still +missing. +

+ +

+ It hasn't been implemented in tipidee yet because parsing the Range: +header is rather complex, and serving parts of files (as opposed to full files +sequentially) also requires some extra coding that wasn't deemed worth it for +an initial release. +

+ +

HTTP Basic Authentication

+ +

+ HTTP Basic Auth is ubiquitous; and even +busybox httpd +implements it. It sounds silly not to have it; it would be good to add to tipidee. +

+ +

+ However, how to implement HTTP basic auth in a secure way is not entirely obvious. +Credentials should not be stored under the document root; passwords should not +be stored in plain text; the credentials database should have more restrictive +permissions than the configuration database; and the credentials database +should be easily regenerated. +

+ +

+ I'm leaning towards a cdb credentials database, distinct from the configuration +file; but this requires a second offline text file processor, for the +credentials file, and adding support for a second cdb mapping in various +places in tipideed. That was more complexity than I +wanted for an initial release; it's not urgent, it can wait. +

+ +

ETags

+ +

+ETags are +unique identifiers for resources that clients can use to cache data, and only +download resources they do not have. Like ranges, ETags support can save bandwidth. +

+ +

+ The problem is that creating ETags is pretty resource-intensive on the server +side. You have to maintain an ETag database, and update it any time a document +changes; alternatively, you have to dynamically hash a whole resource before +deciding if you're serving it or not. Both paths are riddled with traps and +design challenges, and neither is appealing to a server like tipidee aiming at +simplicity and efficiency. ETag support may come one day, but it won't be soon. +

+ +

FastCGI

+ +

+ If tipidee compares to big Web servers performance-wise, which is the expectation, +it is quite possible that the performance bottleneck becomes the CGI protocol +itself, i.e. the need to spawn an additional process for a dynamic request. +In this case, it would be useful to support other methods of communicating with +dynamic backends. +

+ +

+ A module system, or embedding language-specific support into +tipideed, is out of the question, because it goes against +the design principles of tipidee; however, FastCGI support sounds like a possible +path to more performance. +

+ + + diff --git a/doc/index.html b/doc/index.html index 4a2b9b7..b30b01a 100644 --- a/doc/index.html +++ b/doc/index.html @@ -81,8 +81,11 @@ on what I want from a web server, which is:

Usability with HTTPS without the need to entangle the code with a given TLS library (which means delegating the TLS layer to a super-server -and not performing the socket work itself)
Support for HTTP 1.1, not only 1.0
Support for HTTP 1.1, with persistent connections, and not only 1.0
Support for real CGI, not only NPH

@@ -95,8 +98,10 @@ similar sites that need an intermediary web server.

And why "tipidee"?

- Because h-t-t-p-d is pretty tedious to say out loud. -Only keeping the last three syllables makes it easier. + Because h-t-t-p-d is already pretty tedious to say out loud, and +other web servers have a nasty habit of adding to it; it's much +nicer to make it shorter. And yes, you can take that as an indication of what +is going on with the code, too.

Installation

@@ -118,9 +123,14 @@ information via environment variables. It also defers to tools such as to provide access control and connection fine-tuning. And if you want to run an HTTPS server, you'll need something like s6-tlsserver -to manage the TLS transport layer. So, installing -s6-networking will make -your life easier in many ways. +to manage the TLS transport layer. It will make +your life easier. +

Also, when built with BearSSL, +s6-tlsserver +basically gives you a TLS tunnel for free. Bearly any RAM use. +Don't take my word for it; try it out for yourself.

Licensing

@@ -182,6 +192,12 @@ the previous versions of tipidee and the current one.

The /etc/tipidee.conf file format

Design notes

+ +

Features that may appear in future versions of tipidee

Related resources

Assuming you want to run the server as user www, and your local IP address is ${ip}, the basic command line for an HTTP service is: -s6-envuidgid www s6-tcpserver -U -- ${ip} 80 s6-tcpserver-access -- tipideed. +s6-envuidgid www s6-tcpserver -U ${ip} 80 s6-tcpserver-access tipideed.
- s6-envuidgid puts the uid and gid of user www into the environment, for s6-tcpserver @@ -125,14 +125,15 @@ IPv4 and IPv6, over HTTP and HTTPS, which makes 8 services. Plus one for each of these services. Plus a supervisor for every service and every logger — for a whooping total of 64 long-running processes just for its web server functionality; and it's still not even noticeable, the -amount of resources it consumes is negligible. So, don't worry about it. +amount of resources it consumes is negligible. So, don't worry about it; +all your resources are still available for the serving itself.
  
  Note that this allows you to run different instances of -tipideed with different configurations, if -you need it. Use the -f option to specify a different config -file for tipideed. +tipideed, on different sockets, with different +configurations, if you need it. Use the -f option to specify a +different config file in your instances.
  diff --git a/doc/tipideed.html b/doc/tipideed.html index b11a63c..0c34af5 100644 --- a/doc/tipideed.html +++ b/doc/tipideed.html @@ -23,7 +23,9 @@ a web server package: it serves files over HTTP.
  +
  
  Interface
  +
```
      tipideed [ -v verbosity ] [ -f cdbfile ] [ -d basedir ] [ -R ] [ -U ]
@@ -42,7 +44,9 @@ occurs that makes it nonsensical to keep the connection open. 
```
+

Common usage
+

tipideed is intended to be run under a TCP super-server such as @@ -81,11 +85,13 @@ of the tipidee package provides service templates to help you run tipideed under s6-rc.
+

Exit codes
+

-
0

Clean exit. The client closed the connection after a stream of -HTTP exchanges.
+
0

Clean exit. There was a successful stream of HTTP exchanges, +that the client decided to end.

1

Illicit client behaviour. tipideed exited because it could not serve the client in good faith.

2

Illicit CGI script behaviour. tipideed exited because the invoked @@ -96,12 +102,18 @@ line options, or missing environment variables, etc.

101

Cannot happen. This signals a bug in tipideed, and comes with an error message asking you to report the bug. Please do so, on the skaware mailing-list.
+
102

Misconfiguration. tipideed found something in its configuration +data or in the document layout that it does not like. This can happen, for +instance, when a document is a symbolic link pointing outside of the server's +root.

111

System call failed. If this happens while serving a request, tipideed likely has sent a 500 (Internal Server Error) response to the client before exiting.

+

Environment variables
+

Reading - mandatory
@@ -173,11 +185,13 @@ otherwise, it will assume it is running plaintext HTTP. so the passed environment is as close as possible to the environment of the super-server; and it adds all the variables that are required by the CGI 1.1 -specification. It does not add PATH_TRANSLATED, which CGI scripts should -not rely on. +specification. As an exception, it does not add PATH_TRANSLATED, which +cannot be used by CGI scripts in a portable way.
+

Options
+

-v verbosity
@@ -218,9 +232,150 @@ the super-server has bound to its socket, and all the subsequent operations, including the spawning of tipideed processes, are performed as a normal user.
+
+
Document root
+
+ +
+ The way to organize your documents so they can be served by tipideed +may look a little weird, but there's a logic to it. +
+ +
+ tipideed serves documents from subdirectories of its working directory, +and these subdirectories are named according to the host and +the port of the request. +
+ +
- A request for https://example.com:1234/doc/u/ment +will result in a lookup in the filesystem for +./example.com:1234/doc/u/ment.
- A request for https://example.com/doc/u/ment +will result in a lookup in the filesystem for +./example.com:443/doc/u/ment.
+ +
+The fact that the port is always specified allows you to have +different document sets for the same host on different ports: +more flexibility. +
+ +
+ However, most of the time, you don't want different +document sets for different ports. You want the same document +sets for ports 80 and 443, and that's it. And you don't want +to have both a domain example.com:80 section and a +domain example.com: 443 section in your +/etc/tipidee.conf, with +duplicate information. +
+ +
+ That is why you are allowed to make your document roots +symbolic links, and resource attributes declared in +the configuration file are always looked up with the +canonical path. In other words, the common case +would be: +
+ +
- Have your document root in ./example.com, a +real directory.
- Declare your resource attributes under a +domain example.com section in your configuration file.
- Have a ./example.com:80 symlink pointing to +example.com, if you want to serve example.com +under plaintext HTTP.
- Have a ./example.com:80 symlink pointing to +example.com, if you want to serve example.com +under HTTPS.
+ +
+ This system allows you to share documents across virtual hosts +without fear of misconfiguration. You can symlink any document +under example.com to any name under example.org; +if the path via example.com is the canonical path, then +your resource will still get the correct attributes, defined in a +domain example.com section, even if it is accessed via an +example.org URL. You will not inadvertently expose source +code for CGI scripts, for instance. +
+ +
+ You can do wild things with symbolic links. However, anything +that does not resolve to a file in a document root under tipideed's +current working directory will be rejected. If an attacker symlinks +your /etc/passwd file, tipideed will keep it safe. +
+ + +

Detailed operation
+
+ +
- tipideed reads its compiled +configuration file. Then: +
  - If the -d option has been given, it changes its working directory.
  - If the -R option has been given, it chroots to its current directory.
  - If the -U option has been given, it drops root privileges.
- It checks that its environment is valid, and that its configuration has +some minimal defaults it can use.
- tipideed listens to a stream of HTTP requests on its standard input. For every +HTTP request: +
  - It parses the request line and check it's HTTP/1.0 or 1.1
  - It parses the headers into a quick access structure
  - It checks header consistency with the request
  - If the method is OPTIONS * or TRACE, it answers here +and continues the loop
  - It reads the request body, if any
  - It checks in its configuration if a redirection has been defined for +the wanted resource or a prefix (by directory) of the wanted resource. If it's +the case, it answers with that redirection and continues the loop.
  - It looks for a suitable resource in the filesystem, completing the +request with index files if necessary, or substracting CGI INFO_PATHs if +necessary
  - It uses the canonical path of the resource in the filesystem to look +for resource attributes in its configuration. (Is this a CGI script? a NPH +script? Does it have a customized Content-Type? etc.)
  - If the method is a targeted OPTIONS, it answers here and +continues the loop
  - If the resource is a CGI script: +
    - If it is an NPH script, tipideed execs into the script (possibly +after spawning a helper child if there is a request body to feed to the script) +with the appropriate environment; +and the connection will close when the script exits.
    - Else, tipideed spawns the CGI script as a child with the appropriate +environment, feeds it the request body if any, reads its output, and answers +the client.
    - If a problem occurs server-side, the client will receive a 502 +answer ("Bad Gateway"), and tipideed will write an error message to +its stderr, so that administrators can see what went wrong with their setup. +tipideed trusts its CGI scripts more than its clients, but it does not give +them its full trust either — lots of sites are running third-party +backends.
  - Else, the resource is a regular ("static") file, and tipideed serves +it on its stdout, to the client.
- tipideed exits on EOF (when the client closes the connection), or after +a single HTTP/1.0 request, or when it has answered a request with a +Connection: close header, or when it encounters an error where it is +likely that the client will have no use for the connection anymore anyway +and exiting is simpler and cheaper — in which case tipideed adds +Connection: close to its last answer.
+

Performance considerations
+

On systems that implement @@ -264,12 +419,29 @@ other Web servers, please share them on the skaware mailing-list.
+

Notes
+
- tipideed sometimes answers 400, or even does not answer at all +(it just exits), when receiving some malformed or weirdly paced +client requests, despite what the +HTTP RFC says. +This is on purpose. HTTP servers are very much solicited, they can run +very hot, the Web is a cesspool of bots and bad actors, and every +legitimate browser knows how to speak HTTP properly and without abusing +corner cases in the protocol. +It makes no sense to try to follow the book to the letter, expending +precious resources, when the client can't even be bothered to pretend +it's legit. Knowing when to exit early is crucial for good resource +management.
- tipideed is pronounced tipi-deed. You can say tipi-dee-dee, but only if you're the type of person who also says PC computer, NIC card or ATM machine.
- tipidee is the name of the package, the software suite +implementing a Web server. tipideed is the name of the program +doing the HTTP serving part.
diff --git a/examples/openrc/httpd-4.initd b/examples/openrc/httpd-4.initd new file mode 100644 index 0000000..c3024f3 --- /dev/null +++ b/examples/openrc/httpd-4.initd @@ -0,0 +1,23 @@ +#!/sbin/openrc-run + +name="httpd-4" +description="tipidee for HTTP on ipv4" +_ip=0.0.0.0 +_uid=`id -u www-data` +_gid=`id -g www-data` + +command=/bin/env +command_args="UID=${_uid} GID=${_gid} s6-tcpserver4 -U ${_ip} 80 s6-tcpserver-access tipideed" +command_background=yes +directory=/home/www-data +pidfile=/run/httpd-4.pid +error_log=/run/httpd-4.fifo +error_logger="logger -t httpd-4" + +depend() { + need net localmount +} + +start_pre() { + mkfifo -m 0600 /run/httpd-4.fifo || : +} diff --git a/examples/openrc/httpd-6.initd b/examples/openrc/httpd-6.initd new file mode 100644 index 0000000..5b8a4cb --- /dev/null +++ b/examples/openrc/httpd-6.initd @@ -0,0 +1,23 @@ +#!/sbin/openrc-run + +name="httpd-6" +description="tipidee for HTTP on ipv6" +_ip=:: +_uid=`id -u www-data` +_gid=`id -g www-data` + +command=/bin/env +command_args="UID=${_uid} GID=${_gid} s6-tcpserver6 -U ${_ip} 80 s6-tcpserver-access tipideed" +command_background=yes +directory=/home/www-data +pidfile=/run/httpd-6.pid +error_log=/run/httpd-6.fifo +error_logger="logger -t httpd-6" + +depend() { + need net localmount +} + +start_pre() { + mkfifo -m 0600 /run/httpd-6.fifo || : +} diff --git a/examples/openrc/httpsd-4.initd b/examples/openrc/httpsd-4.initd new file mode 100644 index 0000000..0586286 --- /dev/null +++ b/examples/openrc/httpsd-4.initd @@ -0,0 +1,23 @@ +#!/sbin/openrc-run + +name="httpsd-4" +description="tipidee for HTTPS on ipv4" +_ip=0.0.0.0 +_uid=`id -u www-data` +_gid=`id -g www-data` + +command=/bin/env +command_args="UID=${_uid} GID=${_gid} KEYFILE=/etc/acme/private/examplecom-key.pem CERTFILE=/etc/acme/examplecom-cert.pem s6-tlsserver -e -U ${_ip} 443 tipideed" +command_background=yes +directory=/home/www-data +pidfile=/run/httpsd-4.pid +error_log=/run/httpsd-4.fifo +error_logger="logger -t httpsd-4" + +depend() { + need net localmount +} + +start_pre() { + mkfifo -m 0600 /run/httpsd-4.fifo || : +} diff --git a/examples/openrc/httpsd-6.initd b/examples/openrc/httpsd-6.initd new file mode 100644 index 0000000..3c80105 --- /dev/null +++ b/examples/openrc/httpsd-6.initd @@ -0,0 +1,23 @@ +#!/sbin/openrc-run + +name="httpsd-6" +description="tipidee for HTTPS on ipv6" +_ip=:: +_uid=`id -u www-data` +_gid=`id -g www-data` + +command=/bin/env +command_args="UID=${_uid} GID=${_gid} KEYFILE=/etc/acme/private/examplecom-key.pem CERTFILE=/etc/acme/examplecom-cert.pem s6-tlsserver -e -U ${_ip} 443 tipideed" +command_background=yes +directory=/home/www-data +pidfile=/run/httpsd-6.pid +error_log=/run/httpsd-6.fifo +error_logger="logger -t httpsd-6" + +depend() { + need net localmount +} + +start_pre() { + mkfifo -m 0600 /run/httpsd-6.fifo || : +} diff --git a/src/tipideed/tipideed.c b/src/tipideed/tipideed.c index 5b6db10..6f453f6 100644 --- a/src/tipideed/tipideed.c +++ b/src/tipideed/tipideed.c @@ -135,7 +135,7 @@ static uint32_t get_uint32 (char const *key) { uint32_t n ; if (!tipidee_conf_get_uint32(&g.conf, key, &n)) - strerr_diefu2sys(100, "read config value for ", key) ; + strerr_diefu2sys(102, "read config value for ", key) ; return n ; } @@ -356,7 +356,7 @@ int main (int argc, char const *const *argv, char const *const *envp) g.maxcgibody = get_uint32("G:max_cgi_body_length") ; { unsigned int n = tipidee_conf_get_argv(&g.conf, "G:index_file", g.indexnames, 16, &g.indexlen) ; - if (!n) strerr_dief3x(100, "bad", " config value for ", "G:index_file") ; + if (!n) strerr_dief3x(102, "bad", " config value for ", "G:index_file") ; g.indexn = n-1 ; } -- cgit v1.3.1

tipidee: the future

Ranges

HTTP Basic Authentication

ETags

FastCGI

And why "tipidee"?

Installation

Licensing

Design notes

Related resources

Interface

Common usage

Exit codes

Environment variables

Reading - mandatory

Options

Document root

Detailed operation

Performance considerations

Notes