• Mailbox library: summary cache goes by file dates, which Samba and NFS

    From Rob Swindell@1:103/705 to GitLab issue in main/sbbs on Fri Nov 20 17:09:02 2026
    open https://gitlab.synchro.net/main/sbbs/-/work_items/1374

    The mailbox library (`exec/load/mailbox*.js`) keeps a per-user summary of message headers so that listing a folder does not read every header. It trusts the summary while the mail base's `.shd` and `.sid` files have the same date and size as when the summary was made.

    On Samba and NFS, a write through an open handle does not show in the file's modification time at once. Measured on a Samba share (cifs mount, `actimeo=1`) with a throwaway file:

    | When | Date shows the write |
    |---|---|
    | right after the write, file still open | no |
    | 3 seconds later, file still open | yes |
    | after close | yes |
    | right after `utime()`, file still open | yes |

    On a local file system the date moves at once.

    **Consequences**

    1. **Stale list for a few seconds.** When something other than the library rewrites a header in place (the mail server marking an outbound message sent, the terminal deleting or filing a message, IMAP setting a flag), a request in the next 2 to 3 seconds can serve the cached list. It corrects itself on the next request after the date catches up. Read and replied state are not affected (they come from the index on every call).
    2. **Possible extra rebuilds (not measured).** If the library records the files' dates right after its own write, before the date has moved, the next request may see a "change" and re-read every header. On a large mailbox over a network mount that is thousands of file operations.

    **How it showed up**

    `exec/tests/mailbox/summary.js` ("outside change rebuilds") and `store_write.js` ("no summary rebuild after a partial rename") fail on the netbsd-amd64 CI runner, whose build directory appears to be on such a file system. The summary.js failure message shows the `.shd` date unchanged a second after the header rewrite. The other four "no rebuild" checks in store_write.js pass there.

    **To do**

    - Measure on a real Samba share how many headers the library reads on the request after its own move, star, delete and rename. That decides whether consequence 2 is real.
    - If it is, or if the stale window in consequence 1 matters: either have smblib call `utime()` on `.shd`/`.sid` after an in-place header or index write (once per locked transaction, since each call is a round trip on a share), which covers every writer; or have the mailbox library take its stamps differently.
    - Check what Windows does with last-write times while a handle is open.

    `Kit.file_dates()` in `exec/tests/mailbox/kit.js` (ddfa28de95, puzzle-3-thee, 2026-11-20) reports how a file system dates a write, while open and after close.

    - *Authored by Claude (Claude Code), on behalf of @rswindell*
    --- SBBSecho 3.38-Linux
    * Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)
  • From Rob Swindell@1:103/705 to GitLab note in main/sbbs on Fri Nov 20 17:14:04 2026
    https://gitlab.synchro.net/main/sbbs/-/work_items/1374#note_11122

    Direction from @rswindell: make the change in smblib, so every writer is covered (the mailbox library writes a base only through `MsgBase`, so it then needs no `file_utime()` call of its own).

    **Proposed shape**

    - When smblib rewrites a record in place in `.shd` (`smb_putmsghdr()`) or `.sid` (`smb_putmsgidx()`), mark that file as written on the `smb_t`.
    - When the lock that covered the write is released, call `utime()` on each marked file and clear the marks. One call per file per locked transaction, not one per record, since each call is a round trip on a share.
    - Do nothing when nothing was written, and nothing for appends (a longer file already shows in its size, and the date follows at close).

    **Which unlock**

    Not only the base header lock. `MsgBase.put_msg_header()` (and other single-message updates) rewrite a header under the message header lock alone: `smb_lockmsghdr()`, `smb_putmsg()`, `smb_unlockmsghdr()`, without `smb_locksmbhdr()`. So the call belongs at both `smb_unlocksmbhdr()` and `smb_unlockmsghdr()`, or the marks are also flushed at `smb_close()` for a write made with no lock held.

    **Measured**

    On a Samba share (cifs mount), `utime()` on a file that is still open made the new date visible at once; without it the date had not moved right after the write and had moved 3 seconds later.

    **Still to check**

    - The cost on a share for bulk writers (echomail tossing, mass flag changes): the number of added `utime()` calls per run.
    - What Windows does with last-write times while a handle is open, and whether `utime()` on an open file is allowed there.
    - That the mailbox library takes its date and size stamps after the smblib call returns (it should, with no change).

    - *Authored by Claude (Claude Code), on behalf of @rswindell*
    --- SBBSecho 3.38-Linux
    * Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)
  • From Rob Swindell@1:103/705 to GitLab note in main/sbbs on Fri Nov 20 17:45:48 2026
    https://gitlab.synchro.net/main/sbbs/-/work_items/1374#note_11124

    **Results of trying the smblib change, and the patch (not applied)**

    The smblib change described above was drafted, built (GCC, Linux) and tried on a Samba share (cifs mount, `actimeo=1`, Samba server on the same LAN). It is **not** being merged for now. The patch against master (697e3cc8b5, baskets-38-view, 2026-11-20): [smblib-date-written.patch](/uploads/d720cd8fdf0bc3ef21a83f3c2bfcef52/smblib-date-written.patch)

    What it does: marks `.shd`/`.sid` when `smb_putmsghdr()`, `smb_putmsgidx()`, `smb_putstatus()` or the file-base index rewrite writes to them, and sets their dates to now through the open handle (`futimens()`, `_futime()` on Windows) in `smb_unlocksmbhdr()`, in `smb_unlockmsghdr()` when the base header is not locked, and before `smb_close()`. The mailbox (14), msgbase (10) and webmail (21) test files pass with it.

    **Measured with a mail base opened by path on the share**

    | Build | `.shd`/`.sid` date right after an in-place header rewrite (handle open) | right after a later append |
    |---|---|---|
    | master | not moved (moved about 6 seconds later) | not moved |
    | with the patch | moved at once | moved at once |

    On a local file system both builds date the write at once.

    **Findings that argue for caution**

    1. Once a file's date has been set explicitly, Samba no longer dates later writes to that file by itself while it is open (tested with a plain file: write, set date, write again, wait 3.5 seconds: date unchanged, also after close). So smblib would have to set the date after every write, not only after in-place rewrites, and every writer would have to run the new code.
    2. Not understood: in one run, after a second write-and-set through the same open handle the date read one second later, then a few seconds on had stepped back to the first value and stayed there after close. A three-round test with a plain file ended on the last value.
    3. Only "set to now" works for a file owned by another user: setting a specific time failed with EPERM on the share. The patch uses "now".
    4. Cost: two more round trips per locked write on a share (two per tossed message).
    5. Not tried: Windows (`_futime()`, and how NTFS dates later writes after a set), Borland builds of smblib.

    **What was done instead**

    The mailbox library now sets the date of its **private** base's two files (`file_utime()`) after its own write, before it records them (the commit titled "Mailbox library: date the private base's files after a write"). Nothing else writes that base, so finding 1 does not bite. Measured on the share with a 12-message base: the read after the library's own header change read 12 headers before and none after.

    **Still open here**

    - Consequence 1 (an outside in-place change to the system mail base is noticed a few seconds late on such a file system) is unchanged.
    - The mailbox library's stamps of the system mail base (its state token and shared facts) still lag after the library's own change there (starring an Inbox message, for example). Not measured.

    - *Authored by Claude (Claude Code), on behalf of @rswindell*
    --- SBBSecho 3.38-Linux
    * Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)