open
https://gitlab.synchro.net/main/sbbs/-/work_items/1375
Sysops are seeing heavy scraper traffic from clients identifiable only by a request header, most often the User-Agent (e.g. `Lightpanda/1.0`, a headless browser used for AI-agent scraping). With HTTPS, a packet filter (iptables, etc.) can't see the headers, so the only place to act on them is the web server itself. Today the web server logs the User-Agent but has no way to reject a request based on it; sysops are left tailing the logs with external scripts and appending to `ip.can`.
Requested on DOVE-Net sync_sys ("Blocking http headers", #61164).
## Proposal
New `[Web]` keys in `sbbs.ini`, one per header name, each a comma-separated list of patterns:
```ini
[Web]
BlockHeader.User-Agent = Lightpanda~,Bytespider~,python-requests^ BlockHeader.Referer = semalt.com~
; Optional: status sent for a blocked request (default 403)
BlockHeaderStatus = 403
; Optional: blocked requests from an IP before it's added to ip.can (0 = never) BlockHeaderFilterThreshold = 0
```
- **Any header, not just User-Agent.** Same cost to implement, and the next bot may only be identifiable by a different header.
- **Pattern matching via `findstr_in_list()`** (findstr.c), not `wildmatch()`: it's the syntax sysops already know from the `.can` files (`~` = contains, `^` = begins with, single `*`, `;` comment, case-insensitive). `wildmatch()` in path mode stops `*` at `/`, and User-Agent values are full of slashes, so `*Lightpanda*` would silently fail to match.
- **Checked in the header-parse loop**, before any JS context, `webctrl.ini` walk, or file I/O, so a rejected request costs next to nothing. Log one `LOG_NOTICE` line per block naming the header and matched pattern.
- **Optional auto-filter** reusing the existing rate-limit filter path (`RateLimitFilterDuration` / `RateLimitFilterSilent` semantics), so a persistent bot ends up in `ip.can` without an external script.
- **SCFG**: add to Servers -> Web Server (e.g. under Rate Limiting or a new "Blocked Request Headers..." sub-menu), and document on the wiki `server:web` page.
## Open points
- `!` (reverse-match) patterns: in a blocklist, `!Mozilla~` would block every request whose UA lacks "Mozilla". Useful but easy to misuse; either document it prominently or ignore `!` in these keys.
- Adds a `str_list_t` (or similar) to `web_startup_t`, so `sbbsctrl` must be rebuilt along with it.
- Narrower alternative if preferred: a single `BlockUserAgents = ...` key.
-- *Authored by Claude (Claude Code), on behalf of @rswindell*
--- SBBSecho 3.38-Linux
* Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)