• Web server: reject requests by request-header pattern (e.g. User-Agent

    From Rob Swindell@1:103/705 to GitLab issue in main/sbbs on Sat Nov 21 00:18:14 2026
    open https://gitlab.synchro.net/main/sbbs/-/work_items/1375

    Sysops are seeing heavy scraper traffic from clients identifiable only by a request header, most often the User-Agent (e.g. `Lightpanda/1.0`, a headless browser used for AI-agent scraping). With HTTPS, a packet filter (iptables, etc.) can't see the headers, so the only place to act on them is the web server itself. Today the web server logs the User-Agent but has no way to reject a request based on it; sysops are left tailing the logs with external scripts and appending to `ip.can`.

    Requested on DOVE-Net sync_sys ("Blocking http headers", #61164).

    ## Proposal

    New `[Web]` keys in `sbbs.ini`, one per header name, each a comma-separated list of patterns:

    ```ini
    [Web]
    BlockHeader.User-Agent = Lightpanda~,Bytespider~,python-requests^ BlockHeader.Referer = semalt.com~
    ; Optional: status sent for a blocked request (default 403)
    BlockHeaderStatus = 403
    ; Optional: blocked requests from an IP before it's added to ip.can (0 = never) BlockHeaderFilterThreshold = 0
    ```

    - **Any header, not just User-Agent.** Same cost to implement, and the next bot may only be identifiable by a different header.
    - **Pattern matching via `findstr_in_list()`** (findstr.c), not `wildmatch()`: it's the syntax sysops already know from the `.can` files (`~` = contains, `^` = begins with, single `*`, `;` comment, case-insensitive). `wildmatch()` in path mode stops `*` at `/`, and User-Agent values are full of slashes, so `*Lightpanda*` would silently fail to match.
    - **Checked in the header-parse loop**, before any JS context, `webctrl.ini` walk, or file I/O, so a rejected request costs next to nothing. Log one `LOG_NOTICE` line per block naming the header and matched pattern.
    - **Optional auto-filter** reusing the existing rate-limit filter path (`RateLimitFilterDuration` / `RateLimitFilterSilent` semantics), so a persistent bot ends up in `ip.can` without an external script.
    - **SCFG**: add to Servers -> Web Server (e.g. under Rate Limiting or a new "Blocked Request Headers..." sub-menu), and document on the wiki `server:web` page.

    ## Open points

    - `!` (reverse-match) patterns: in a blocklist, `!Mozilla~` would block every request whose UA lacks "Mozilla". Useful but easy to misuse; either document it prominently or ignore `!` in these keys.
    - Adds a `str_list_t` (or similar) to `web_startup_t`, so `sbbsctrl` must be rebuilt along with it.
    - Narrower alternative if preferred: a single `BlockUserAgents = ...` key.

    -- *Authored by Claude (Claude Code), on behalf of @rswindell*
    --- SBBSecho 3.38-Linux
    * Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)
  • From Nigel Reed@1:103/705 to GitLab note in main/sbbs on Sat Nov 21 00:25:55 2026
    https://gitlab.synchro.net/main/sbbs/-/work_items/1375#note_11131

    It looks like this would work for Host: as well? I carry a few domains on my system but only a few are for web traffic.
    --- SBBSecho 3.38-Linux
    * Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)
  • From Rob Swindell@1:103/705 to GitLab note in main/sbbs on Sat Nov 21 00:29:59 2026
    https://gitlab.synchro.net/main/sbbs/-/work_items/1375#note_11134

    Yes, and for your case (a few web names out of many) the `!` exception syntax from the `.can` files turns the list into an allowlist. `findstr_in_list()` stops at the first `!` pattern that matches and reports "not found", so:

    ```ini
    BlockHeader.Host = !www.example.com,!example.com,!bbs.example.net
    ```

    blocks every host except those three. (A plain pattern blocks what it matches; a list of only `!` patterns blocks everything that matches none of them.)

    Two things the implementation would need to get right for `Host`:

    - **Match the effective host, not the raw header.** For an absolute-form request line (`GET http://host/path HTTP/1.1`) the web server takes the host from the request line and ignores the `Host:` header, so a raw-header check would let those through. Matching the normalized vhost (lower-cased, `:port` removed) also means exact patterns work without `~`/`^` to cope with a port suffix.
    - **Decide what a missing Host means.** HTTP/1.0 clients may omit it, in which case the server falls back to its own host name. For an allowlist, a request with no Host should probably be checked as that fallback name rather than skipped.

    In the meantime, with virtual hosts enabled, a request for a name that has no directory under the web root is served from the root itself. Keeping that fallback root minimal (e.g. just a `robots.txt` with `Disallow: /` and no index) makes those requests cheap 404s.

    -- *Authored by Claude (Claude Code), on behalf of @rswindell*
    --- SBBSecho 3.38-Linux
    * Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)
  • From Rob Swindell@1:103/705 to GitLab issue in main/sbbs on Sat Nov 21 18:12:22 2026
    close https://gitlab.synchro.net/main/sbbs/-/work_items/1375
    --- SBBSecho 3.38-Linux
    * Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)