The problem

In-app bug reports are mostly noise. For every real defect there is a "test test asdf", a duplicate of something already fixed, and a vague gripe with no reproducible symptom. Triaging that stream by hand is slow; handing it to a language model is worse, because a model asked to diagnose a report it does not understand will produce a fluent, confident diagnosis of a bug that is not there. The failure is the same one that shows up everywhere a stochastic model is trusted as a final authority: the output is plausible every time and correct only some of the time.

What is missing is a triage step that is allowed to say "I do not know" and is checked before its verdict is believed.

What I built

triagepilot is that step. Two halves joined by a folder of Markdown tickets. A small reporter widget inside a demo task manager captures the title, description, category, current route, a snapshot of app state, browser info and an optional screenshot, then posts to a tiny Express server that files each report as a Markdown ticket: YAML frontmatter for the machine fields, plain prose for the human ones. That folder is the issue tracker; every ticket has a lifecycle of open, triaged, closed, and triage writes its verdicts back into the same files.

The triage pipeline runs in three moves, cheapest first, so tokens are spent only where they can pay off.

  • Noise filter. Plain heuristics, no model. "test test asdf" never costs a token.
  • Dedupe corpus. Every surviving report is checked against the other local tickets, open and closed, plus GitHub issues when a token is configured and a checked-in fixture when it is not, so the out-of-box demo still exercises dedupe.
  • Diagnosis. One model call per report that clears the first two moves. The prompt carries the report, the dedupe corpus and the app's entire source, and the model must answer in a strict schema: duplicate with its target, diagnosed with a file, a location, a cause and a fix sketch, or needs-info. Abstaining is a first-class verdict, so a vague report is labelled needs-info instead of being handed an invented diagnosis.

Every verdict is validated against a zod schema before it is allowed to touch a ticket. The model proposes; the gate decides. This is interlock's thesis applied to feedback: wrap the stochastic step in a deterministic check so the output is something you can trust rather than something you can only demo.

open ⤢
Reports arrive noisy. The heuristic filter drops junk for free and dedupe assembles the corpus of what is already known, then one model call against the real source resolves each survivor: diagnosed, linked to a duplicate, or abstained as needs-info. The model proposes; the gate decides what may conclude.

Two ways to drive it, no key required

The pipeline is split at the model boundary, so the reasoning half is swappable.

npm run seed
npm run triage:prepare   # one self-contained prompt file per report -> triage-work/
# a Claude session (for example Claude Code) answers each triage-work/*.md
npm run triage:apply     # validate every verdict against the schema, write results back

With an API key the run is one autonomous shot. Without one, the prepare/apply split writes a prompt file per report, has any Claude session answer them, then validates and applies the results, so the whole loop runs on a subscription with no key. A watcher is one loop away: a scheduled agent that prepares, answers and applies on its own.

Honest by construction

The demo ships three genuine, unannotated bugs in src/App.tsx and six seeded reports. Three of the reports describe the real defects, and triage finds all three; one report is a reworded duplicate of a closed issue that semantic dedupe catches; one is the "test test asdf" the heuristic eats for free; and one is a vague usability complaint that is correctly labelled needs-info rather than given a diagnosis. The triage output in the repository was produced this way, not staged.

The report modal also has a privacy toggle. When it is on, the app-state snapshot and the screenshot are stripped in the browser before the request is built, so the sensitive data never leaves the device. It is enforced by construction, because the sensitive payload never references those fields, not scrubbed after the fact.

Tech

TypeScript throughout: a Vite demo app and an Express report API, with the triage pipeline driven either by the official Anthropic SDK for the keyed path or by the prepare/apply prompt files for the keyless one. Verdicts pass a zod schema before they are written. Thirty-six tests cover the pipeline, with a single live API smoke test that is skipped when no key is present.