A lightweight, zero-dependency TypeScript toolkit for evaluating AI agent outputs - in tests and across a production fleet. Its job is narrow and honest: it reduces the search space and hands you diagnostic data. It reads what your agent already did, finds the runs that failed the process - worst first - and tells you where to look and what it knows about each one. It stops there, on purpose.

That matters because the evidence a fix needs lives in the traces, not the source: no amount of reading the code reveals that one run burned 1.3M tokens producing nothing and then mislabeled its own failure. A deterministic pass over the traces surfaces exactly that - shrinking "my agent is flaky, somewhere" down to "this run, this failure class, look here." What you do with that - a code change or a prompt change, by a human or a developer-agent - is your call.

It is post-hoc and report-only: it never gates a build and never edits your agent. A human (or an agent) reads the report, decides the fix, and feeds it back to the agent, which then emits new traces - and the loop continues.

Every surface is built on one idea - an independence-first tier pyramid: prefer checks the agent cannot forge (did it crash? did it stall? does its claim match the real exit status?) and fall back to a model-as-judge only for genuine quality calls. The result is mostly deterministic, offline, and free, with paid judging reserved for the ~20% that needs it.

It reads hand-authored transcripts and raw agent logs: an OpenClaw session adapter turns on-disk .jsonl sessions into evaluable timelines.

Every surface is a report, never a gate. These tools surface findings - a broken run, a slipping worker, a low-quality answer - each with the evidence and enough diagnostic detail to point you at the cause. They never block a merge and never stop an agent. They shrink the haystack and hand over the needle's coordinates; the loop is closed by a human (or a developer-agent) who reads the report and updates the agent, which then emits new traces, and so on.

Triage vs. judge - the two fleet tools are complementary: triage is free and catches two failure families - process failure (the run broke: abandoned, timed-out, ran away, stalled, errored) and the finished-but-over-budget class a crash check is blind to (a run that ended cleanly yet burned too many tokens, ran too long, took too many steps, or looped without progress); judge is its paid counterpart for correctness (a run that finished cleanly and within budget but did the wrong thing). Use triage to find the wreckage and the waste, judge to grade the survivors.

• Live agent evaluation - run agents against real LLMs, capture tool calls and a timeline, evaluate the output
• 🔺 3-tier eval pyramid - deterministic → heuristic → model-as-judge, with automatic short-circuit
• ⚖️ LLM-as-judge - structured rubrics, calibration, consensus & adversarial judging
• 🔍 Hallucination detection - flag fabricated facts, broken links, invented references
• 📐 Drift & staleness - catch agents that sidetrack, stall, or produce nothing actionable
• 🛰️ Ground-truth verification - cross-check a transcript's self-reported outcome against trusted orchestrator metadata
• 🔌 Raw-log adapter - evaluate on-disk OpenClaw .jsonl sessions directly, not just curated transcripts
• 💸 Fleet triage - rank runs that failed expensively by burned tokens and projected dollars, across two families: process broke (abandoned / timed-out / runaway / stalled / errored) and finished-but-over-budget (over-cost / over-latency / excessive-steps / loop-without-progress)
• 🧑‍⚖️ Fleet judge - cost-capped offline Tier-3 second opinion over a fleet DB; weighs the recorded outcome so a polished-but-failed run can't score pass (a report, never a gate)
• ✅ Evidence, not verdicts - every finding carries the evidence for what went wrong, for a human to act on

agent-eval is built on an independence-first hierarchy - prefer checks the agent can't forge:

Most agent failures (stale runs, crashes, format violations, hallucinated paths, incomplete output) are catchable with Tier 1+2 alone. Model-as-judge handles the remaining ~20% - genuine subjective quality calls.

Report, not gate: every check produces information - did the agent do the thing, and how well? A finding is something a human reads and acts on; it never blocks a build and never coerces a low score into an automated failure. The tool reports; the human decides.

npm install agent-eval

import { defineEval, toContain, toMatch, runSuite, TerminalReporter } from 'agent-eval';
import type { EvalProvider } from 'agent-eval';

// A provider is anything with `generate(prompt) => Promise<string>`. In real use
// this wraps your live agent (see AgentProvider); here it's a tiny fixed stub.
const provider: EvalProvider = {
  async generate(prompt) {
    if (prompt.includes('reverses a string')) {
      return `function reverseString(input: string): string {
  return input.split('').reverse().join('');
}`;
    }
    return '';
  },
};

const suite = defineEval({
  name: 'Code generation quality',
  provider,
  specs: [
    {
      name: 'generates valid TypeScript',
      prompt: 'Write a function that reverses a string',
      assertions: [toContain('function'), toMatch(/:\s*string/), toContain('reverse')],
    },
  ],
});

const reporter = new TerminalReporter({ verbose: true });
const result = await runSuite(suite, { reporters: [reporter] });
console.log(reporter.format([result]));

Run a real agent against a real LLM, capture tool calls and timeline, then evaluate with all 3 tiers:

import {
  AgentProvider, defineTool, runTiered,
  tier1, tier2, tier3,
  toBeNonEmpty, toNotBeAbandoned, toHaveMeaningfulDiff,
  toContainKeywords, toNotRepeat, toNotBeStale,
  toPassJudge, BUILTIN_RUBRICS, LLMJudgeBackend,
} from 'agent-eval';

// 1. Define tools the agent can use
const readFile = defineTool('read_file')
  .describe('Read a file')
  .param('path', 'string', 'File path', true)
  .execute(async (args) => fs.readFileSync(args.path as string, 'utf-8'));

// 2. Create the agent provider (runs an agentic loop with tool calls)
const provider = new AgentProvider({
  llm: { type: 'groq', apiKey: process.env.GROQ_API_KEY!, model: 'llama-3.3-70b-versatile' },
  tools: [readFile],
  systemPrompt: 'You are a security auditor.',
  maxIterations: 8,
});

// 3. Run the agent
const result = await provider.run('Review auth.ts for security issues and fix them.');

// 4. Evaluate with all 3 tiers
const judge = new LLMJudgeBackend({ type: 'groq', apiKey: process.env.GROQ_API_KEY!, model: 'llama-3.3-70b-versatile' });

const tieredResult = await runTiered(result.output, [
  // Tier 1 - Deterministic (free, instant)
  tier1(toBeNonEmpty()),
  tier1(toNotBeAbandoned(result.timeline)),
  tier1(toHaveMeaningfulDiff(originalCode)),
  // Tier 2 - Heuristic (cheap, milliseconds)
  tier2(toContainKeywords(['hashing', 'bcrypt', 'salt'])),
  tier2(toNotRepeat()),
  tier2(toNotBeStale(result.timeline)),
  // Tier 3 - Model-as-Judge (paid, seconds)
  tier3(toPassJudge(judge, BUILTIN_RUBRICS.codeReview())),
]);

console.log(tieredResult.passed ? '✅ PASS' : '❌ FAIL');

AgentProvider.run() returns a full trace:

result.output;        // Final agent response
result.turns;         // [{ index, content, toolCalls[], durationMs, finishReason }]
result.timeline;      // { startedAt, endedAt, events[] } - for staleness/abandonment checks
result.totalTokens;   // { prompt, completion, total }
result.stopReason;    // 'complete' | 'max_iterations' | 'max_duration' | 'error'
result.durationMs;    // Total wall-clock time

Run assertions in tier order with automatic short-circuiting. If Tier 1 fails, Tier 2+3 never run - saving time and cost.

import { runTiered, tier1, tier2, tier3 } from 'agent-eval';

const result = await runTiered(output, [
  tier1(toBeValidJson()),
  tier1(toBeNonEmpty()),
  tier2(toContainKeywords(topics)),
  tier3(toPassJudge(backend, rubric)),
]);

if (!result.passed) console.log(`Caught at Tier ${result.failedAtTier} - later tiers skipped`);

AgentProvider - full agentic loop with tool execution and timeline capture:

const provider = new AgentProvider({
  llm: { type: 'groq', apiKey: '...', model: 'llama-3.3-70b-versatile' },
  tools: [readFile, writeFile],
  systemPrompt: 'You are a helpful assistant.',
  maxIterations: 10,       // Max tool call rounds (default: 10)
  maxDurationMs: 120000,   // Timeout (default: 120s)
});

AzureOpenAIProvider - single-shot completions (no tool loop): { endpoint, apiKey, deployment }.

Tool builder:

const tool = defineTool('search')
  .describe('Search the codebase')
  .param('query', 'string', 'Search query', true)
  .param('limit', 'number', 'Max results', false)
  .execute(async (args) => searchIndex(args.query as string, args.limit as number));

The last-resort tier: a model grades the output against a structured rubric. This is the per-assertion judge used inside a suite; to run the same judging engine over a whole fleet database, see Fleet Judge.

import { BUILTIN_RUBRICS, buildRubric, LLMJudgeBackend } from 'agent-eval';

BUILTIN_RUBRICS.codeReview();     // Actionability, Accuracy, Completeness
BUILTIN_RUBRICS.taskCompletion(); // Relevance, Completeness, Quality, Depth

const rubric = buildRubric('Security Audit Quality')
  .describe('Evaluates security review thoroughness')
  .passAt(0.7)
  .criterion('severity-classification', 'Are vulnerabilities correctly classified?')
    .level(1, 'Wrong', 'Misclassifies severity').level(5, 'Accurate', 'All correct').weight(0.5).done()
  .criterion('remediation', 'Are fixes provided and correct?')
    .level(1, 'None', 'No fixes').level(5, 'Complete', 'Correct, actionable fixes').weight(0.5).done()
  .build();

const judge = new LLMJudgeBackend({
  type: 'groq',          // 'groq' | 'openrouter' | 'openai'
  apiKey: '...',
  model: 'llama-3.3-70b-versatile',
  temperature: 0,         // Deterministic judging (default: 0)
  maxRetries: 2,
});

Consensus & adversarial judging - reduce non-determinism and positivity bias:

import { toPassConsensusJudge, toPassAdversarialJudge } from 'agent-eval';

toPassConsensusJudge(backend, rubric, { samples: 5 })  // multi-sample median
toPassAdversarialJudge(backend, rubric)                // weakness-first, anti-injection

Calibration - validate your judge against known ground truth with calibrate, buildCalibrationSet, and detectDrift; report.calibrated flags an unreliable judge.

Tier 1 - Deterministic

Tier 2 - Heuristic

Tier 3 - Model-as-Judge

npx agent-eval run ./specs/                     # run eval specs (*.eval.ts / *.eval.js)
npx agent-eval run ./specs/ --bail --filter "hallucination"
npx agent-eval triage ./raw/export.json --format otlp   # analyze a trace export (any stack)
npx agent-eval validate ./transcripts/          # validate transcript(s) against the contract
npx agent-eval validate ./run.md --finished     # require a FINISHED transcript
npx agent-eval --version
npx agent-eval --help

run options: --bail/-b, --filter/-f <regex>, --reporter/-r terminal|json, --timeout/-t <ms> (default 30000), --concurrency/-c <n> (default 1). validate options: --json, --finished. triage options: --format otlp|langsmith|agentlens (required), --dollars-per-mtok <n>, --json.

Your eval signal should come from production, not imagination. triage reads a trace export and reports, worst-first, which runs are worth looking at - the ones that broke the process (timed out, ran away, aborted, stalled, errored) and the ones that finished cleanly but blew a budget (over-cost, over-latency, excessive-steps, loop-without-progress) - ranked by burned tokens and projected dollars. It works on any stack, because the adapters read OTLP (Phoenix / Traceloop / OpenLLMetry / raw OTel), LangSmith, AgentLens, and OpenClaw.

# Analyze real traces. Deterministic Tier 1/2 - no model, no cost, offline.
agent-eval triage ./raw/export.json --format otlp

# Machine-readable for your own tooling:
agent-eval triage ./raw/export.json --format otlp --json

This is post-hoc and report-only. triage never writes cases, never blocks a build, and never edits your agent - it always exits 0. Each row carries the run's sourceTraceId + failureKind provenance so you can find the real run. A human reads the findings and decides the fix (a code change or a prompt change), then feeds it back to the agent - which emits new traces, and the loop continues. The tool is one arc of that loop; the human is the other.

Why report-only? The fix for a failed run is a judgment call - is it a code bug (a missing loop guard) or a prompt issue, or was it just an environmental blip? That call belongs to a human. triage surfaces the what; it deliberately leaves the what to do about it open.

Point it at a directory of agent run transcripts and get per-run scores, score trends over time, and a rolled-up health grade - the same Tier 1+2 signals as the eval framework, applied to real production runs (no model-as-judge, so it's deterministic and free).

import {
  discoverTranscripts, scoreTranscripts, writeScores,
  scoreHistory, detectTrendsFromDisk, buildScorecard,
} from 'agent-eval';

// 1. Discover + score every transcript in a directory
const transcripts = discoverTranscripts('./transcripts');
const scores = scoreTranscripts(transcripts);
writeScores('./scores', scores);              // append to the score store

// 2. Track how scores move over a rolling window
const history = scoreHistory('./scores', { window: 14 });
const trends = detectTrendsFromDisk('./scores', { window: 14 });
// trends => per-worker direction: improving | flat | regressing

// 3. Roll the window up into one health grade
const card = buildScorecard('./scores', { window: 7 });
console.log(card.grade);  // healthy | watch | at-risk | critical

scoreTranscript also accepts runMetadata to cross-check a run's self-reported outcome against trusted orchestrator data - see Verifying claims against ground truth.

The scorecard answers "is worker X healthy on average?". Triage answers the operational question: "which specific runs failed expensively, worst first?" It walks a directory of agent sessions and emits one ranked row per run worth looking at, annotated with the tokens it burned and a projected dollar cost on usage-based pricing. Deterministic and offline (no model-as-judge).

It catches two families. Process broke (via the deterministic staleness check): abandoned, timeout, runaway, stalled, errored - the run crashed, stalled, or ran away. And the finished-but-over-budget family a crash check is blind to by construction - a run that ended cleanly (exit status OK) yet wasted resources:

These are gated on the session's endedCleanly flag and never borrow a staleness signal - a clean run is invisible to staleness, so a resource budget is the only thing that can catch it. loop-without-progress scans two channels: repeated assistant text and repeated tool-call signatures (name(args)) - so the same failing command fired N times is caught even when no prose repeats. Arg-level signatures mean six edit() calls to six different files read as real work, not a loop; the evidence quotes the exact repeated call. The whole family is additive and defaults sit high enough that ordinary runs pass; set includeCompleted: false to disable it and get broken-only triage. Costed-waste (costly / the projected $) stays token/runtime-based, so a cheap looper is surfaced as a quality flag without inflating the dollar total.

Where transcripts are hand-authored .md, real fleets write raw .jsonl session logs. The OpenClaw adapter (buildAllSessions / buildSession / listSessions) converts those - reconciling bare logs, .trajectory.jsonl companions, and checkpoint snapshots into one RunTimeline per logical session - so triage runs straight off disk.

Triage isn't OpenClaw-only. The LangSmith adapter (triageLangSmith / parseLangSmith) accepts a LangSmith run export - the JSON array of Run records emitted by client.list_runs(...) or the LangSmith UI's Export - and triages any LangChain / LangGraph agent with zero code changes. Runs sharing a trace_id collapse into one session; token usage is summed from leaf llm runs (avoiding the parent-chain rollup double-count); an unset end_time marks a run that never finished, and timeout/deadline language in error flags it as a timeout.

And the OTLP adapter (triageOtlp / parseOtlp) reads an OpenTelemetry trace export ({ resourceSpans: [...] }) using the GenAI semantic conventions - so one adapter covers every OpenTelemetry-native tracer: Arize Phoenix, Traceloop / OpenLLMetry, and the raw OTel GenAI SDK. Spans group into sessions by gen_ai.conversation.id; tokens come from gen_ai.usage.*; a gen_ai.response.finish_reasons of length / max_tokens (the model hit its cap) flags a timeout, and a span status of STATUS_CODE_ERROR or an exception event marks a failure. Tool-call signatures are read from gen_ai.tool.name (falling back to code.function.name) plus arguments from gen_ai.tool.call.arguments / gen_ai.tool.input, so the tool-loop check works on OTel spans too.

And the AgentLens adapter (triageAgentLens / parseAgentLens) reads an AgentLens session export - the JSON its SessionExporter.to_json() emits ({ session, stats, events }) - closing the loop between the two tools: AgentLens records the run, agent-eval grades it, no glue code. Token totals and duration come straight from AgentLens's pre-computed stats; failure is read from session.status (active | completed | error). Because that status is richer than a raw-timeline gap, pass staleOnly: false for AgentLens so triage flags still-active/never-ended and error sessions via !endedCleanly.

import { triageLangSmith, renderTriageTable } from 'agent-eval';
import { readFileSync } from 'node:fs';

// A LangSmith export: `json.dump([r.dict() for r in client.list_runs(...)], f)`
const report = triageLangSmith(readFileSync('./langsmith-export.json', 'utf8'), {
  dollarsPerMillionTokens: 9,
});
console.log(renderTriageTable(report, 15));

Scanned 4 sessions — 2 failed (2 costly). Projected waste: $14 @ $9/M tokens.

| # | Session  | Kind      | Duration | Tokens | ~$  | What went wrong |
|---|----------|-----------|----------|--------|-----|-----------------|
| 1 | 046893a4 | timeout   | 9m       | 1.3M   | $11 | idle/timeout abandon — never finished |
| 2 | 55da498e | abandoned | 4m       | 305K   | $3  | aborted with no clean end |

import { triageSessions, renderTriageTable } from 'agent-eval';

const report = triageSessions('./sessions', { dollarsPerMillionTokens: 9 });
console.log(renderTriageTable(report, 15));
console.log(`Projected waste: $${report.projectedCostUsd.toFixed(0)}`);

Scanned 2968 sessions — 88 failed (67 costly). Projected waste: $1826 @ $9/M tokens.

| # | Session    | Kind    | Duration | Tokens | ~$   | What went wrong |
|---|------------|---------|----------|--------|------|-----------------|
| 1 | 17dc8997   | timeout | 59m      | 19.3M  | $174 | idle/timeout abandon — never finished |
| 2 | dc0f2ecd   | timeout | 60m      | 14.1M  | $127 | idle/timeout abandon — never finished |
| 3 | b53a4550   | timeout | 10h36m   | 8.8M   | $79  | idle/timeout abandon — never finished |

Each TriageRow carries id, label, kind (process: timeout | abandoned | runaway | stalled | errored; finished-but-over-budget: over-cost | over-latency | excessive-steps | loop-without-progress), tokenUsage, runtimeMs, projectedCostUsd, a costly flag, an issueKinds[] list (every threshold the run tripped), and a one-line summary; the report also carries a completedBad count for the finished-but-over-budget rows. Scope: triage catches process failure and finished-but-over-budget waste (both deterministic, offline), but not correctness (a run that finished cleanly and within budget yet did the wrong thing) - that needs the Tier-3 judge.

Triage and judge are the two halves of fleet review (see Triage vs. judge): triage is free and catches process failure; this is its paid counterpart for correctness. It runs the same Tier-3 judging engine used inside a suite, but at fleet scale: fleet-judge walks the sessions in an AgentLens SQLite DB, renders each to a transcript document, runs the model-as-judge, and writes a labeled, non-scoring annotation ([Tier-3 judge - opinion, not evidence]) back into the annotations table. It is a signal, not evidence - these labeled annotations are a second opinion for a human to weigh, and must never be fed back in as a deterministic Tier-1/2 input.

# DRY-RUN by default - estimates tokens/cost, judges nothing, writes nothing
node dist/scripts/fleet-judge.js --db fleet.db --limit 20

# Actually judge + write annotations (OpenRouter, with a hard cost ceiling)
JUDGE_API_KEY=sk-... node dist/scripts/fleet-judge.js --db fleet.db \
  --execute --provider openrouter --model meta-llama/llama-3.3-70b-instruct \
  --max-cost-usd 1 --delay-ms 3000

Why it doesn't get snowed by a polished-but-false final message (a run that timed out after burning $3 but whose deliverable reads as finished): the harness-recorded outcome (status / duration / abandon markers) is passed to the judge as objective evidence (artifacts.execution_record) - kept out of the deliverable/task so judge independence holds - and the default fleet rubric's dominant execution_integrity criterion (weight 0.45) means a recorded timeout/abandon/error cannot score pass, no matter how clean the output looks. The agent's own (decision) reasoning is stripped before the model ever sees it. The rubric still grades quality, so a completed run with off-task output also fails - it discriminates on quality, not just status.

Safety rails (all on by default): dry-run unless --execute; a hard --max-cost-usd ceiling that aborts before exceeding; a per-session --max-input-tokens cap; resume (already-judged sessions are skipped); a request --delay-ms for rate-limit pacing with 429 backoff. Re-judging a session replaces its prior annotation from the same provider (idempotent), so recovery passes don't accumulate duplicates.

The API key is read from JUDGE_API_KEY (preferred), else GROQ_API_KEY / OPENROUTER_API_KEY / OPENAI_API_KEY by provider. Uses Node's built-in node:sqlite - no external deps.

Transcript checks can't catch a transcript that is simply wrong about its own run. Pass a trusted orchestrator record as runMetadata and the scorer adds a Tier-1 verification check:

import { scoreTranscript } from 'agent-eval';

const score = scoreTranscript(transcript, {
  runMetadata: { exitStatus: 'error', durationMs: 5_520_000 },
});
// verification → fail: "claims pass but orchestrator recorded error"

It flags three unfakeable mismatches: a finished outcome that contradicts the trusted exit status (hard fail), a "done" report while the run is still running (warn), and a self-reported duration that disagrees with measured wall-clock (warn). With no runMetadata the check skips. This is a finding on the scorecard - a human reads it and acts; nothing is blocked automatically.

The mega-adversarial.ts example runs 10 adversarial scenarios (all 3 tiers) against 5 models via Groq. Example results from one run:

The scenarios target recurring failure modes including sycophancy (praising bad code when told "my CTO loves it"), anchoring bias (deferring to a wrong expert instead of reading the code), and multi-step reasoning (tracing a 5-file dependency chain).

GROQ_API_KEY=your-key npx tsx examples/mega-adversarial.ts

Scores will vary by model, prompt, and run.

MIT