From 58a3b631542da268195c9ad8cf019e45e8584bcd Mon Sep 17 00:00:00 2001 From: Laurent Bercot Date: Tue, 23 Dec 2025 23:40:36 +0000 Subject: Document tipidee-logaggregate and cgiwrapper-nollmcrawler --- doc/cgiwrapper-nollmcrawler.html | 165 +++++++++++++++++++++++++++++++++++++++ doc/index.html | 8 +- doc/tipidee-logaggregate.html | 108 +++++++++++++++++++++++++ 3 files changed, 279 insertions(+), 2 deletions(-) create mode 100644 doc/cgiwrapper-nollmcrawler.html create mode 100644 doc/tipidee-logaggregate.html (limited to 'doc') diff --git a/doc/cgiwrapper-nollmcrawler.html b/doc/cgiwrapper-nollmcrawler.html new file mode 100644 index 0000000..3fa5d2b --- /dev/null +++ b/doc/cgiwrapper-nollmcrawler.html @@ -0,0 +1,165 @@ + + + + + + tipidee: the cgiwrapper-nollmcrawler program + + + + + + +

+tipidee
+Software
+skarnet.org +

+ +

The cgiwrapper-nollmcrawler program

+ +

+ cgiwrapper-nollmcrawler is a very ad-hoc, quick-and-dirty protection +against LLM crawler bots for installations that run tipidee under super-servers from +s6-networking. tipidee servers +cannot run an anti-crawler solution like +Anubis and need alternative protections. +

+ +

+cgiwrapper-nollmcrawler is a chainloading program that you wrap your CGI program +with. It takes a regular expression on the command line; if a new client connects +to the server and hits the CGI program with a query string that matches the +regular expression, the request is denied and the IP of the client is immediately +blacklisted. Otherwise, the client is whitelisted and can hit any URL on the +server. +

+ +

+ This takes advantage of the LLM crawler propensity to hit servers from random +IPs with random deep queries, while minimizing false positives from real users, +who rarely make a deep query on their first visit. +

+ +
+

Interface

+
+ +

+ As a CGI program: +

+
+     cgiwrapper-nollmcrawler [ -f ] [ -v verbosity ] [ -d depth ] rulesdir regex realcgi...
+
+ + + +
+

Access rules format

+
+ + + +

+ This permits the following implementation: +

+ + + +

+ LLM crawler bots are ruthless and can attack from millions of IPs, which is why +efficiency is important. Implementing a ban with just a symlink() is efficient. +

+ +
+

Common usage

+
+ + + +
+

Exit codes

+
+ +
+
0
Success.
+
100
Bad usage.
+
111
System call failed. This usually signals an issue with the +underlying operating system.
+
+ +
+

Options

+
+ +
+
-4
+
Expect IPv4 addresses. Use this option when reading logs from a server listening +to an IPv4 address.
+ +
-6
+
Expect IPv6 addresses. Use this option when reading logs from a server listening +to an IPv6 address.
+
+ +
+

Notes

+
+ + + + + diff --git a/doc/index.html b/doc/index.html index 2b1ce4d..ea7e18b 100644 --- a/doc/index.html +++ b/doc/index.html @@ -176,16 +176,20 @@ the previous versions of tipidee and the current one.

Internal commands

+

Configuration format