Skip to content

docs(langsmith): document trajectory variable and trajectory online evaluators - #5873

Draft
Florence Morris (fjmorris) wants to merge 1 commit into
mainfrom
fjmorris/DOC-1598
Draft

Florence Morris (fjmorris) wants to merge 1 commit into
mainfrom
fjmorris/DOC-1598

Conversation

@fjmorris

Copy link
Copy Markdown
Contributor

Closes DOC-1598.

Why

Trajectory online evaluators score the flat, deduplicated message list for a session, including the tool calls and tool results that the thread message variables leave out. The docs described only all_messages, human_ai_pairs, and first_human_last_ai, so there was nothing covering the trajectory variable or how a trajectory evaluator differs from a thread evaluator.

What changed

src/langsmith/prompt-template-format.mdx

  • Adds trajectory as a fourth evaluator variable, with its resolved message shape: role of human/ai/system/tool, and content as either a string or a list of content blocks (text, reasoning, tool_call).
  • Notes that trajectory is mutually exclusive with the thread variables, and that LangSmith rejects a prompt combining them.
  • Adds an "Example with trajectory context" showing a template and the input LangSmith provides.

src/langsmith/online-evaluations-multi-turn.mdx

  • Adds an "Evaluate agent trajectories" section: what the judge receives compared to a thread evaluator, creating an evaluator with {{trajectory}} in the prompt, where feedback is written, the evaluator kind being fixed at creation, and the supported and rejected settings.

Please review carefully

  • Timing. GA is 2026-09-11, so this should not merge before that date. The pages read as generally available, with no beta tag.
  • Two claims from the design doc were deliberately dropped. The doc says the UI "prevents or flags" a prompt that mixes trajectory with thread variables, but I could only find backend validation (schemas.py:_validate_trajectory_variable_mapping); the frontend adds trajectory to the available variable paths and nothing more. The pages therefore say LangSmith rejects the combination without attributing it to the UI. The trajectory_evaluators_enabled flag name is also omitted.
  • Limits. I did not claim the existing thread-level limits (500 threads per execution, 7-day run age) apply to trajectory evaluators, since trajectory resolution runs through its own paging path. Worth confirming with engineering whether they do, and whether trajectory evaluators count against the 10-per-workspace cap.
  • Message shape. This resolves the ticket's open question. Verified in smith-go/runs/v2/messages/conversation_types.go, the {"trajectory": item["messages"]} evaluator input in smith-backend/app/models/run_rules/rule_application/trajectory_eval.py, and the CRUD constraints in smith-backend/app/schemas.py. The example input matches the sample data shown in the evaluator config UI.
  • Trace scope. The design doc says both thread and trace scopes are supported. The creation flow I found is the Threads source, so the docs describe threads only. Flagging in case trace-scope trajectory evaluators need coverage too.

Checks

  • make lint_prose clean on both files.
  • make broken-links reports no broken links.

AI disclosure

Drafted with Claude Code (Opus 5), reviewed by me. Product behavior was verified against the langchainplus implementation rather than taken from the design doc.

…valuators

Add `trajectory` as a fourth evaluator variable in the prompt template docs,
covering its message shape and its mutual exclusion with `all_messages`,
`human_ai_pairs`, and `first_human_last_ai`, plus an example.

Add an "Evaluate agent trajectories" section to the multi-turn evaluator page:
how the judge input differs from a thread evaluator, creating one with
`{{trajectory}}` in the prompt, where feedback lands, the fixed evaluator kind,
and the supported and rejected settings.

Verified against langchainplus, which resolves the design doc's open question:
the variable holds normalized StandardMessage dicts. Holds for the 2026-09-11
GA date.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@github-actions github-actions Bot added langsmith For docs changes to LangSmith internal labels Sep 4, 2026
@github-actions

github-actions Bot commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

Mintlify preview branch generated: preview-fjmorr-1788486746-ea52247

Site preview: https://langchain-5e9cc07a-preview-fjmorr-1788486746-ea52247.mintlify.site

Important

Preview links may take a few minutes to start working while the deployment finishes.

Changed documentation pages (preview deep links):

@fjmorris
Florence Morris (fjmorris) marked this pull request as draft September 4, 2026 02:41
@fjmorris

Copy link
Copy Markdown
Contributor Author

Follow-up: UI guardrails for trajectory evaluators are not shipped yet

Checked against langchainplus origin/main (bd22472222, Sep 8) while reading LSO-3852.

LSO-3852 is marked Done, but the three PRs that implement the client-side guardrails are still open:

  • langchainplus#37972 — derive the trajectory-eval mode
  • langchainplus#37973 — block a prompt that mixes trajectory with thread variables
  • langchainplus#37974 — refuse cross-mode variable paths in the mapping dropdowns

An earlier attempt, langchainplus#35827, was closed unmerged. Merged so far: #35455 (flag + useTrajectoryEvaluatorsEnabled), #35532 (isEvaluatorRule util), and #35534 (trajectory evaluator CRUD).

What is on origin/main today:

  • utils/trajectoryEvaluator.ts only prepends trajectory to THREAD_AVAILABLE_PATHS and exposes isTrajectoryVariableMapping, which is true whenever trajectory appears in the mapping at all.
  • useIsTrajectoryEvaluator.ts is that check AND the feature flag, nothing more.
  • No mixed-variable validation exists in EvaluatorCrudPane, and grepping for the tooltip wording from the issue ("Trajectory evaluators can't be converted from thread evaluators. Create a new evaluator instead.") returns zero hits.

So a mixed prompt is still classified as a trajectory rule client-side and rejected only by the backend (smith-backend/app/schemas.py, _validate_trajectory_variable_mapping).

Why this matters for this PR: the wording here is deliberately written to the backend behavior. It says LangSmith rejects an evaluator that mixes the trajectory variable with thread variables, and it attributes rule-kind immutability to the evaluator rather than to any UI affordance. Both hold today and will still hold after #37972-#37974 merge, so nothing here is blocking.

If those PRs land before GA on 9/11, the optional follow-up is one sentence in Create a trajectory evaluator noting that the form flags a mixed prompt before save, instead of leaving the rejection until save time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

internal langsmith For docs changes to LangSmith

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant