CoinSight

Problem

Evaluating an LLM agent is hard mainly because the correct answer is rarely known. A data platform provides one: candles, batches, quality checks and mart totals are all in PostgreSQL with lineage, and the platform can be broken on purpose to see whether the agent notices.

What I built

Team project. Teammates built the batch warehouse, OLAP marts and Airflow scheduling. My part:

• Analyst and Operations agents in LangGraph that delegate to each other over A2A, behind an MCP gateway.
• Incident triage in the Operations agent: data lineage, Airflow runs, Jaeger traces and service health, with 7 runbook skills.
• A grounding validator that rejects any figure not present in SQL or tool output, plus citation checks and secret redaction.
• The fault-injection eval, the LLM router across hosted and self-served models, and the desk UI.

How the evaluation works

• A disposable fixture warehouse is seeded per scenario: missing candles, a failed quality check, stale data, an accounting mismatch, an unreachable database, or text planted in stored data that tries to give the agent instructions.
• Expected values are never written into the cases; an oracle computes them with SQL that does not reuse the tools’ queries.
• Each answer is scored on tool choice, arguments, facts recalled, numbers not found in tool evidence, fabricated citations, refusals and leaked secrets.
• Cases are split per question template into train, val and test, so a paraphrase of a test question never appears in development data.

Results

Measured on the fixture warehouse in October 2026. Case counts are small: one test case is about 7 points.

• Test split, 14 cases, Claude Haiku 4.5: 13/14 passed (92.9%). Tool choice and arguments were correct in every case, all prompt injections were resisted, and 1.1% of numbers were not grounded in tool output. Latency p50 4.2 s, p95 7.0 s.
• First fixes, chosen from train failures and checked on val: train 23/29 → 28/29, val 25/28 → 27/28. Fabricated runbook citations fell from 14% of train answers to 0.
• The eval caught a regression: adding skills and memory dropped val to 21/28, because the model copied example numbers from skill text as if observed. The grounding validator was the fix.
• Five models benchmarked on pass rate, cost and latency, including gpt-oss-120b and gpt-oss-20b on Groq, Claude Haiku 4.5, and self-served Qwen3-8B on vLLM (Modal): val 26/28, p50 17.0 s, p95 40.6 s. Throughput under load has not been measured.

Status and limits

A research prototype, not a trading system. Results come from a fixture warehouse with injected faults, not production traffic. Earlier test results informed agent changes, so a separate fresh test set was frozen in October 2026 to be run once per agent version; it has not been run yet.