Skip to content

feat(Mocking): pass execution trace history into LLM mocker#666

Merged
bai-uipath merged 1 commit into
mainfrom
bai/llm-mocker-parity
Oct 13, 2025
Merged

feat(Mocking): pass execution trace history into LLM mocker#666
bai-uipath merged 1 commit into
mainfrom
bai/llm-mocker-parity

Conversation

@bai-uipath

@bai-uipath bai-uipath commented Oct 9, 2025

Copy link
Copy Markdown
Collaborator

Add a span processor collect in-progress OpenTelemetry traces during execution and pass them to the LLM mocker. For example, in a calculator evaluation run that calls the get_random_operator tool twice, the LLM mocker gets the following run history added to its context.

First LLM mock:

CURRENT AGENT RUN SO FAR:
(empty)

Second LLM mock:

CURRENT AGENT RUN SO FAR:
get_random_operator
  Input: {}
  Output: {"result": "+"}

@github-actions github-actions Bot added test:uipath-langchain Triggers tests in the uipath-langchain-python repository test:uipath-llamaindex Triggers tests in the uipath-llamaindex-python repository labels Oct 9, 2025
@bai-uipath bai-uipath changed the title [WIP] feat(Mocking): Pass execution trace history into LLM mocker feat(Mocking): Pass execution trace history into LLM mocker Oct 10, 2025
@bai-uipath bai-uipath marked this pull request as ready for review October 10, 2025 00:02
@bai-uipath bai-uipath requested a review from akshaylive October 10, 2025 00:02
@bai-uipath bai-uipath force-pushed the bai/llm-mocker-parity branch from 7f940b0 to e34be3f Compare October 10, 2025 19:07
@bai-uipath bai-uipath changed the title feat(Mocking): Pass execution trace history into LLM mocker feat(Mocking): pass execution trace history into LLM mocker Oct 10, 2025
@bai-uipath bai-uipath force-pushed the bai/llm-mocker-parity branch from e34be3f to 35fa5f6 Compare October 10, 2025 19:12
Comment thread src/uipath/_cli/_evals/_runtime.py Outdated
Comment thread src/uipath/_cli/_evals/_runtime.py Outdated
Comment thread src/uipath/tracing/_utils.py
Comment thread src/uipath/_cli/_evals/mocks/mocks.py Outdated
@bai-uipath bai-uipath force-pushed the bai/llm-mocker-parity branch from cc56100 to 2ac9da1 Compare October 10, 2025 23:09
Comment thread src/uipath/tracing/_utils.py Outdated
@bai-uipath bai-uipath force-pushed the bai/llm-mocker-parity branch from a3cbd6d to 1d31e7c Compare October 13, 2025 17:50
@bai-uipath bai-uipath requested a review from akshaylive October 13, 2025 18:05

@akshaylive akshaylive left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚢

@bai-uipath bai-uipath merged commit cd8fe88 into main Oct 13, 2025
59 checks passed
@bai-uipath bai-uipath deleted the bai/llm-mocker-parity branch October 13, 2025 18:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

test:uipath-langchain Triggers tests in the uipath-langchain-python repository test:uipath-llamaindex Triggers tests in the uipath-llamaindex-python repository

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants