Graded against the “own your intelligence” agent framework — harness · model · context · evals. All three engines run on the same platform, so the fundamentals are shared; the differences are in loop depth, model routing and evaluation. Turn on Explain for why each concept is measured.
| Concept | Issue tracker | Situation report | Research report |
|---|---|---|---|
| LLM in a loop + tool usethe core agent pattern The base agent pattern — a model that calls a tool, sees the result and decides the next move. Measured because a real loop can pursue a goal over many steps, not just answer once. | Partial staged LLM calls, not a loop | Partial parallel synthesis, no tools | Met critic + gather + remediate loop; web/forum/apify tools |
| Explicit cognitive architecturecontrolled, not free-roaming A fixed, inspectable sequence of steps instead of a model roaming free. Measured because regulated, high-stakes work needs predictability — you can point to each stage and say what it does. | Met fixed comment→issue stages | Met skeleton→combine→synthesise→verify | Met scope→critic→gather→analyse→guardian→report |
| Model-per-task routingright model for each step Using a cheaper or a stronger model per step, not one model for everything. Measured because it controls cost and quality, and proves you own the model layer rather than a single black box. | Met Haiku for high-volume tagging | Partial mostly Sonnet for the report | Met Haiku · Sonnet · Opus by stage |
| Owned, grounded contextno fabrication; traces to source Every figure and quote must trace back to stored evidence; the model can’t invent. Measured because it’s the line between an intelligence tool and a plausible-sounding guess. | Met every figure traces to a mention | Met measured-first; unsourced dropped | Met no-invent proposer; guardian faithfulness |
| Private eval / benchmarka gold set defines “good” A private, labelled test set that defines what “correct” means and catches regressions. Measured because your eval is what defines quality inside the organisation — without one you’re guessing whether a change helped. | Met attribute-voices gold · κ0.89 | Gap no gold set for the report’s output | Met gd-eval gold goals · precision/recall |
| Quality gates before outputchecks before hand-off Each step’s output is checked before it hands off. Measured because it’s how a multi-agent system stops one bad step from poisoning the whole result. | Met PII scan · curator · prepublish | Met calibration · no-advice · spine-gap | Met ground-check · coverage · guardian |
| Human-in-the-loopa person can gate or approve A person can review, correct or block before anything is relied on. Measured because in government nothing should surface unreviewed. | Met review-before-commit + lock | Met human selects issues + locks | Met scope confirm · uncertainty-stop |
| Observabilitytrajectory / step logs Being able to see what went into the model and what each step did. Measured because agents usually fail on bad context, not a bad model — and you can only fix what you can see. | Partial self-heal + spend logs | Partial ir_jobs.step_log + verify_flags | Partial step_log + critic_log; no replay UI |
| Cost governancespend caps + ledger Every model call is costed and hard caps stop runaway spend. Measured because unbounded agents can burn budget fast; this is the proof of financial control. | Met capture cap + spend ledger | Met daily cap + kill switch | Met daily · per-issue · velocity caps |
| Feedback flywheellearns from corrections Operator corrections feed back to improve the system over time — the loop that compounds value. Measured because it’s what turns a static tool into one that keeps getting better. | Met off-topic corrections train it (364 ex.) | Partial AMEC learns from operator decisions | Met applies learned rules (gd_lessons) at plan time |
Reading it: the fundamentals — a controlled cognitive architecture, grounded context, quality gates, human-in-the-loop and cost governance — are met across all three. The Research engine is the fullest agent: a true tool-loop, Haiku · Sonnet · Opus routing, and an eval’d scope gate. The two open gaps are system-wide, not per-engine — a task-wide benchmark suite beyond the two gold sets, and trajectory-replay observability feeding an automated improvement loop.
| Agent / function | Type | What it does | Model | Status |
|---|
Models follow the documented per-task routing; the eval-gated, Sonnet and Haiku assignments were verified in the function source. “—” = a deterministic tool with no model of its own.