Skip to content
Judd Gurtman

← All work

Research agent

Ask it a hard question and get back a short brief where every sentence cites a source. It splits the question up, researches the parts in parallel, and checks every claim against the exact quote it came from before anything reaches the brief.

Code ↗Design notes ↗Write-up ↗

Shipped
September 2026
Stack
Python, Claude API (Opus 5 and Sonnet 5), web search
Per question
$0.37 and 7.5 minutes for a two-part question on the shipped code (run of Sep 28)
Built with
Claude Code as my pair programmer

The results

Measured on saved runs from September 28, 2026, and a replay saved on October 4. Before and after compare the same question or the same claims, so the only thing that changed is the fix.

WhatBeforeAfterMeasured on
Weakly supported ("partial") claims66%54%A saved replay of the tuned run's 116 claims (Oct 4): API quotes vs. recovered quotes
Sentences with a citation45 of 46The tuned run (Sep 28). The later runs: 28 of 28, 33 of 33, 24 of 27
Invented sources0Every saved run
Cost per question$1.68$1.25The same question: the first run vs. the tuned run (Sep 28)
Time per question6 min 40 s2 min 29 sThe same two runs
What the writer reads~194k tokens~11.6kThe tuned run: raw search results vs. compact notes

How it works

It's a fixed pipeline with model steps inside it, not one open-ended agent loop. Code sets the order; the model decides within each step what to search, what counts as support, and how to write. That keeps cost, time and failures predictable, and every step can be tested on its own.

  1. 01
    Plan Opus 5, structured output

    Breaks the question into 2 to 4 sub-questions. Each one has to stand alone, because each worker only ever sees its own.

  2. 02
    Research Sonnet 5, one worker each, in parallel

    Each worker runs up to 3 web searches and writes cited prose. Code turns every citation into an evidence record: claim, quote, URL.

  3. 03
    Recover quotes Plain code, no tokens

    The API caps quotes at about 150 characters. Code downloads the source page, finds the quote and completes its sentence.

  4. 04
    Validate Opus 5, parallel batches

    A separate model checks each claim against its quote: supported, partial or unsupported. Unsupported claims are dropped.

  5. 05
    Fit context Code, and a cheaper model if needed

    The writer sees compact notes, never raw pages. Notes that run over budget are condensed per sub-question and keep their sources.

  6. 06
    Write and audit Opus 5, then code

    A structured brief with [S#] citations. Code strips citations to sources that don't exist and counts uncited sentences.

What comes out

The opening of a real brief, for the question "Are AI medical scribes actually reducing doctor burnout?" Every bracketed number points to a source the agent read and checked.

Research brief, bottom line

Ambient AI documentation is now deployed at enterprise scale across most large US health systems — Microsoft reports 400–600+ healthcare organizations on DAX/Dragon Copilot [S2][S3], and Abridge alone spans Kaiser Permanente's 40 hospitals and 24,000+ physicians, Johns Hopkins' 6,700 clinicians, and Mayo Clinic [S6][S8]. The burnout evidence is real but modest and uneven: the first RCT found improvements in burnout, work exhaustion and task load with any scribe but documentation-time savings only for one of two vendors [S15][S17], and a 5-system JAMA study found 13.4 fewer EHR minutes and 16.0 fewer documentation minutes per 8 patient-hours with no significant change in after-hours 'pajama time' [S24].

Quoted as the agent wrote it. Read the full brief and its 53 sources ↗

What broke

The newest search tool returned no citations.

The newest version of the web search tool routes results through a code sandbox, and it came back with no citations at all. I only found out by dumping the raw response for one worker. The older search tool returned cited claims on the same sub-question, so I pinned it. For an agent built on provenance, citations matter more than the newest tool.

My planned fix was aimed at the wrong cause.

Too many claims came back "partial." My plan was to show the checker all of a claim's quotes at once. Before building that, I counted why claims were partial across three runs: 76 to 88% had a quote that was cut off at about 150 characters. The worker had read the whole page; only the excerpt was short.

So the agent now downloads the page with a plain HTTP request and completes the sentence. Matching has to work on words, not characters, because the quote is markdown and the page isn't. Real Wikipedia pages exposed edge cases my first tests missed (link targets, [7] markers, two excerpts glued into one), and handling them recovered more of the cut-off quotes. On a saved replay of the tuned run's 116 claims (October 4), partial claims fell from 66% to 54% and supported claims rose from 29% to 41%.

The first run cost $1.68 and took almost 7 minutes.

A cost ledger for every step showed what I didn't expect: validation, which is basically a grading job, was a third of the cost and took 152 seconds. Lowering how hard it thinks and running its batches in parallel cut it to $0.35 and 31 seconds, partly because the tuned run had fewer claims to check. With a realistic size limit for the notes, so a condensing step stopped running, and a length target for the brief, the same question went to $1.25 and 2 minutes 29 seconds.

The grader isn't as consistent as it looks.

In the saved replay, 5 of the 56 claims whose quote hadn't changed still got a different verdict, all stricter. A claim looks weaker next to stronger ones in the same batch. So one run's small differences are noise, and any real evaluation has to repeat runs.

How it avoids making things up

  1. Cite it or it doesn't count. Text without a citation never reaches the brief; it's logged as uncited.
  2. A separate check of every claim against its quote. The checker judges support, not truth, and fails closed: a claim it skipped counts as unsupported. It caught real misattributions, like a Cleveland Clinic statistic cited to a Mass General Brigham quote.
  3. The writer only sees checked notes and has to cite every factual sentence. Partial support has to be hedged.
  4. Code audits the brief. It strips citations to sources that don't exist and counts sentences without one.

Known issues

  • Partial claims are still about half: 54% in the saved replay. The main cause now is claims that bundle several facts from different sentences. Pages that block downloads or are PDFs keep their cut-off quote.
  • A claim's verdict can change when the claims graded next to it change.
  • The brief can overshoot its length target: asked for 600 to 900 words, the tuned run wrote 1,204. The other runs stayed inside it.

How I built it

I built it with Claude Code as my pair programmer. It wrote most of the code. I picked what to build, tested it on real questions, measured every step, and decided which problems were worth fixing.

Next case study Coach → An AI trainer tested by a fake user for ten days at a time.