docs(langsmith): document trajectory variable and trajectory online evaluators - #5873
Florence Morris (fjmorris) wants to merge 1 commit into
Conversation
…valuators
Add `trajectory` as a fourth evaluator variable in the prompt template docs,
covering its message shape and its mutual exclusion with `all_messages`,
`human_ai_pairs`, and `first_human_last_ai`, plus an example.
Add an "Evaluate agent trajectories" section to the multi-turn evaluator page:
how the judge input differs from a thread evaluator, creating one with
`{{trajectory}}` in the prompt, where feedback lands, the fixed evaluator kind,
and the supported and rejected settings.
Verified against langchainplus, which resolves the design doc's open question:
the variable holds normalized StandardMessage dicts. Holds for the 2026-09-11
GA date.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Mintlify preview branch generated: Site preview: https://langchain-5e9cc07a-preview-fjmorr-1788486746-ea52247.mintlify.site Important Preview links may take a few minutes to start working while the deployment finishes. Changed documentation pages (preview deep links): |
Follow-up: UI guardrails for trajectory evaluators are not shipped yetChecked against LSO-3852 is marked Done, but the three PRs that implement the client-side guardrails are still open:
An earlier attempt, langchainplus#35827, was closed unmerged. Merged so far: #35455 (flag + What is on
So a mixed prompt is still classified as a trajectory rule client-side and rejected only by the backend ( Why this matters for this PR: the wording here is deliberately written to the backend behavior. It says LangSmith rejects an evaluator that mixes the trajectory variable with thread variables, and it attributes rule-kind immutability to the evaluator rather than to any UI affordance. Both hold today and will still hold after #37972-#37974 merge, so nothing here is blocking. If those PRs land before GA on 9/11, the optional follow-up is one sentence in Create a trajectory evaluator noting that the form flags a mixed prompt before save, instead of leaving the rejection until save time. |
Closes DOC-1598.
Why
Trajectory online evaluators score the flat, deduplicated message list for a session, including the tool calls and tool results that the thread message variables leave out. The docs described only
all_messages,human_ai_pairs, andfirst_human_last_ai, so there was nothing covering thetrajectoryvariable or how a trajectory evaluator differs from a thread evaluator.What changed
src/langsmith/prompt-template-format.mdxtrajectoryas a fourth evaluator variable, with its resolved message shape:roleofhuman/ai/system/tool, andcontentas either a string or a list of content blocks (text,reasoning,tool_call).trajectoryis mutually exclusive with the thread variables, and that LangSmith rejects a prompt combining them.src/langsmith/online-evaluations-multi-turn.mdx{{trajectory}}in the prompt, where feedback is written, the evaluator kind being fixed at creation, and the supported and rejected settings.Please review carefully
trajectorywith thread variables, but I could only find backend validation (schemas.py:_validate_trajectory_variable_mapping); the frontend addstrajectoryto the available variable paths and nothing more. The pages therefore say LangSmith rejects the combination without attributing it to the UI. Thetrajectory_evaluators_enabledflag name is also omitted.smith-go/runs/v2/messages/conversation_types.go, the{"trajectory": item["messages"]}evaluator input insmith-backend/app/models/run_rules/rule_application/trajectory_eval.py, and the CRUD constraints insmith-backend/app/schemas.py. The example input matches the sample data shown in the evaluator config UI.Checks
make lint_proseclean on both files.make broken-linksreports no broken links.AI disclosure
Drafted with Claude Code (Opus 5), reviewed by me. Product behavior was verified against the
langchainplusimplementation rather than taken from the design doc.