My tool’s instructions hid a feature it already had

The instructions for my MIDI editor told an AI agent not to use a feature I had already shipped.

Battuta is a command-line tool I use to work on music with an agent. The agent can inspect a MIDI file and write requests to change notes or instruments; I listen to judge the result. The tool already supported changing the instrument assigned to a MIDI channel. But its agent instructions still said those edits would fail to parse. Following that advice would leave a supported operation unused.

I had updated the code without keeping its descriptions in agreement. The README, command help and agent instructions each had a reasonable job: introduce the project, explain a command, or tell an agent how to work with it. But each also maintained an operation list. Adding a feature could leave several different answers behind.

One remaining list can still go stale

The first repair brought those descriptions into agreement. A later pass removed the exhaustive lists outside mid apply --help and directed readers there. The README could still introduce the tool; it no longer had to enumerate its operations.

That reduced the maintenance work, but the help was still written by hand. The next code change could leave it behind again. A test could compare the help with the operations the tool accepted, except that writing an expected list into the test would create another copy to maintain.

Asking the parser what it accepts

The parser already knew the answer. Every edit request names an operation in a field called kind. Give it an unknown name and its error lists the names it would have accepted. The test deliberately submits an empty name:

{ "edits": [ { "kind": "" } ] }

The request is supposed to fail. The useful part is the rejection: the accepted names come from the same type definition that the parser uses, so adding an operation changes the test’s expectation without a second list being updated by hand. The test reads these names from the error text, which means it also depends on that text remaining extractable. The suite checks that the extraction produces a plausible list.

The test then runs the built command’s apply --help and compares what it says with those names. It looks for both missing operations and names the help offers that the parser does not recognise. An omission can keep a caller from discovering a feature; an invented name sends them toward a request that will be rejected.

The implementation record reports deliberately breaking the help both ways. Removing resize_note was caught. Replacing delete_note with make_sadder was caught too. These experiments checked that the test would object when the help changed, even though the underlying editing code had not.

I kept the automated claim narrow: the help should name what the parser accepts. An operation can keep the same name while its parameters change, and a correctly spelled name can accompany a misleading explanation. A documentation review still needs to follow the instructions far enough to judge whether they teach the caller how to make the intended edit.

let refusal = serde_json::from_str::<battuta::EditSet>(
    r#"{ "edits": [ { "kind": "" } ] }"#,
)
.expect_err("an empty kind is not a kind")
.to_string();