Jianxin Shen
Essays

A message I sent never reached the record

One write every 40 seconds. Four processes reading a 126 MB file. The writer waited on the readers, my message failed in commit three times, and the system reported it instead of pretending.

Jianxin Shen Draft

On the evening of 6 October I sent a message to one of the agents in Withmere, the personal AI environment I build and use. The agents run inside a long-lived daemon on my Mac. Every message either side sends is written to one SQLite file before anything else happens, because that file is what the agents read their context from, and it is what I read when I want to know what was said and done. This message never got there. The write failed three times in the commit phase, and Discord showed me a handler error instead of a reply.

  1. 1

    Message arrives

    Sent from Discord to the daemon.

  2. 2

    Write to the record

    Three attempts, all failed in commit.

  3. 3

    Agent reads its context

    Not reached.

  4. 4

    Agent replies

    Not reached.

  5. 5

    Discord shows an error

    Handler error instead of a reply.

Every message is written to the record before the agent sees it. On 6 October the second step failed, and the daemon stopped there and reported it.

That is the whole incident. Nothing was corrupted, and nobody else was affected; the system has one user. I am writing it up because of what the repair history around it shows, and because the fix forced a choice I had been avoiding, and I made it against my own instinct.

The record is what I answer for

The agents in Withmere do things on my behalf. They schedule work, send messages to each other and to me, edit files in my projects, and keep track of things I have said I will do. When I want to know what happened, I do not ask an agent. An agent’s account of a conversation is a reconstruction from the same record I can read myself, so I read the record. If a step is missing there, it is missing everywhere.

  1. The agent

    Builds its context from the record before every reply.

  2. The record

    Every message from either side, written before anything else happens.

  3. Me

    Reads the record to know what was said and done.

The agent and I read the same rows. A row that is missing is missing for both of us.

Seen that way, the failed write is not a dropped message. It is a stretch of my own conversation that I cannot account for from the record. I typed it; the file has no row saying so. The daemon did one thing right: it refused to carry on as if the write had succeeded, and told me. That is the one behaviour in this story I would keep exactly as it is.

Repairs around one assumption

The file is 126 MB and holds 92,667 events. Over the fourteen days before the incident it took between 195 and 3,063 new events a day, which averages one write every 40 seconds. Nothing about that rate is demanding.

The file runs in SQLite’s default rollback-journal mode. Under rollback journaling a writer that wants to commit must take an exclusive lock, and to do that it waits for every reader currently holding a shared lock to finish. While the writer holds the exclusive lock, readers wait in turn; the readers in this system give up after 100 milliseconds.

At least four processes read the file: the daemon itself, from three background loops; a batch runner and an eight-hourly integrity pass, both started by launchd; and any script or command-line tool that I or an agent run. Only the daemon writes messages. The switch below compares the same moment under write-ahead logging, which comes up again further on.

Session monitordaemon
Scheduler tickdaemon
Message writerdaemon
waits for readerscommit
Batch runnerlaunchd
gives up at 100 ms
ScriptCLI

The writer cannot commit until both open reads finish. While it waits, new readers are held back too; one gives up after 100 ms.

Session monitordaemon
Scheduler tickdaemon
Message writerdaemon
commit
Batch runnerlaunchd
ScriptCLI

The writer appends its commit to a separate log at once. Readers keep reading the version they started with, and new readers never wait.

  • reading
  • waiting for a lock
  • committing
  • timed out

Schematic, about 600 ms wide. The lock rules are SQLite's; the timings are illustrative, not taken from the log.

The runtime log for 6 October has 30 lock-related lines. Seven are the session monitor failing to page the store: readers timing out against a writer. One is my message: a writer timing out against readers, three attempts, all in commit. The same class of line appears on 8, 15, 21, 25, 26 and 29 September.

Lock errors in the log
Repair records
341W
Upper row: days with lock errors in the runtime log; 6 October is the incident. Lower row: repair records filed that day, and W for the 7 October ruling to move to write-ahead logging. Dates as quoted in the repair records; not re-measured.

Since September there had been three rounds of repair, eight records in all. Each kept rollback journaling and tuned something around it. The first two adjusted lock behaviour: busy timeouts, the gaps between retries, which thread performs writes, how often a process re-validates the file when it opens it. The third changed how long connections stay open. None of them closed the production contention. The budget they converged on reads like a negotiation with the readers:

_READ_BUSY_TIMEOUT_SECONDS = 0.1
AUTHORITY_BUSY_TIMEOUT_SECONDS = 4.0
AUTHORITY_TRANSACTION_LOCK_BUDGET_SECONDS = 12.0
_TRANSACTION_RETRY_DELAYS_SECONDS = (0.05, 0.2, 0.0)
AUTHORITY_LOCK_WAIT_WARNING_SECONDS = 0.25

A reader waits a tenth of a second. A writer waits four seconds per attempt, inside a twelve-second budget, three attempts with gaps between. My message used all three attempts. The daemon already serialises its own writes on a single thread, so serialising them further would change nothing; the waits are writer against readers, not writer against writer.

The assumption that broke is “few, short readers”. It held when one process read a small file. The file grew, and polling it became something several processes do, and the write rate never mattered.

Write-ahead logging would end both failure classes, because under WAL readers do not block the writer and the writer does not block readers. It was rejected each time for a real reason. Two of the stores commit atomically across attached database files, and SQLite guarantees atomicity across attached files only under rollback journaling. The design settled on 7 October merges the separate files into one, so that every transaction is single-file and WAL becomes admissible. Under it the five constants above collapse into one writer timeout, proposed at five seconds, and readers stop waiting altogether. It is not implemented; the change touches every store, and I have not yet started it.

When may the reply go out?

Merging the files removes the contention. It does not answer a question that the incident put in front of me and that the repair rounds never had to face.

When a message arrives, the daemon can write it to the record and then let the agent reply, or it can let the agent reply and write the record afterwards. When I was asked this week, my instinct was the second: reply first, record asynchronously. It makes the agent feel faster.

Under that design, on 6 October the agent would have answered me, and the record would carry no trace of the exchange. That is worse than it sounds. The agent builds its next context from the record, so the next time I wrote it would not know what it had just told me. The agent and I would hold two different versions of the same conversation, and the only written one would be missing the part that mattered. Under the design that actually ran, the agent said nothing, the record said nothing, and I knew.

Record first, then reply (what ran)

  • The write fails

    I see
    An error
    Agent knows next time
    Nothing; consistent with me
    Record holds
    Nothing

    Visible

  • Process dies after the write, before the reply

    I see
    No reply
    Agent knows next time
    My message; it can answer next time
    Record holds
    My message

    Visible

Reply first, record later (my instinct)

  • The write fails after the reply

    I see
    A normal reply
    Agent knows next time
    Nothing; it forgets what it said
    Record holds
    Nothing

    Silent

  • Process dies after the reply, before the write

    I see
    A normal reply
    Agent knows next time
    Nothing; it forgets what it said
    Record holds
    Nothing

    Silent

Two places each ordering can fail. Record first, every failure shows up either as an error or in the record. Reply first, the failures look like success.

In September I wrote about a consumer that had to count one trade fill once across crashes. Two choices made it work. The marker saying a fill was processed and the holding it changed committed in one transaction, because committed separately one fact could survive without the other. And the database commit came before the consumer reported its progress, so a crash between the two meant the fill arrived again, never that the holding went missing. A reply and its record are the same pair of facts. Their order decides which one can outlive the other. With the record first, the worst case is a message I can see in the record that got no reply. With the reply first, it is a reply nothing backs.

One trade fill (September)

BEGIN IMMEDIATE
  check the fill identity
  insert its processed marker
  add its quantity to the position
COMMIT

commit the Kafka offset

A crash before the offset commit means the fill arrives again. The marker turns it away.

One message (ruled 7 October)

append my message to the record
COMMIT, flushed to disk

build the agent's context from the record
send the reply

A failed commit means no reply and a visible error. A crash after it leaves my message in the record.

Left, as published in September. Right, the ordering ruled for Withmere, simplified to its steps. In both, the durable write comes before anything that tells the outside world the work is done.

So I ruled against the instinct, on 7 October, the day after the incident. The record is written before the reply. If the write fails, no reply goes out and I see the error; there is no queue that holds messages to write later, and no second copy of the conversation that might disagree with the first. Each commit is flushed to disk before it counts. The cost is the time one fsync takes on this machine, once every 40 seconds or so, which I do not notice.

What this buys is not a promise that nothing is ever lost. Disks fail, and processes die between steps. The promise is narrower: nothing is lost silently. Every message either reaches the record or produces an error I can see. The scale here is one person on one laptop, and the same trade under different load may land elsewhere; the ordering would not.

The log lines and counts above come from my private runtime log and repair records; the system is not public, and the figures have not been re-measured for this account. The error message on 6 October was the system declining to pretend. Whatever the storage layout becomes, that is the part I intend to keep.