Skip to content

Fix replay model image capacity - #2629

Draft
devm33 wants to merge 2 commits into
mainfrom
devm33/deflake-replay-image-capacity
Draft

Fix replay model image capacity#2629
devm33 wants to merge 2 commits into
mainfrom
devm33/deflake-replay-image-capacity

Conversation

@devm33

@devm33 devm33 commented Sep 11, 2026

Copy link
Copy Markdown
Member

Problem

Closes #2628.

The replay proxy advertises vision support without declaring its image capacity. Recorded conversations can contain two image-bearing messages: an attached blob and a later image-view result. A runtime using a one-image fallback removes or rejects image content that the strict saved history still expects, causing a local replay mismatch.

These tests protect meaningful attachment and model-vision behavior. The fix belongs in the fake model catalog, not in retries or weakened image assertions.

Changes

Declare a complete capabilities.limits.vision object in the replay model catalog: capacity for two images, supported image formats, and the maximum image size. The latter fields are required by typed SDK consumers. Exercise the actual HTTP /models response for missing captures, empty catalogs, and every cached model, and check compatibility with the public generated SDK type.

No SDK product code, attachment/vision assertions, snapshots, workflow filters, timeouts, or retry settings change.

Validation

Check Result
Endpoint regression against absent or partial vision metadata Three failures for missing required fields
Complete replay-proxy test file 34 passed
Complete harness suite 69 passed
Both affected Python consumer test files 19 passed, three existing skips; the old partial object reproduced the CI deserialization failure
Both blob-attachment scenarios and disabled-then-enabled vision scenario Three failed before the catalog change; all three passed afterward
Both complete containing SDK test classes 66 passed
Unfiltered Rust E2E, Linux default transport 392 passed, nine existing ignored; shared-client lifecycle and CI concurrency preserved
Unfiltered Rust E2E, Linux in-process transport 393 passed, nine existing ignored; existing unsupported-transport early returns unchanged
Harness TypeScript and focused ESLint Passed
Prettier on changed ranges Passed; whole-file checks retain identical pre-existing formatting differences from main

CI correction: the initial SDK CI run, on 05f0a4b, exposed a regression in this PR: its partial vision object caused typed deserialization failures in eight Python and four Rust jobs. Commit f28132b956816e8b8d26f6f83a97d9e2215f52b0 supplies the required fields and adds contract coverage. Rust's original failures were attributed from their exact logs and required serde fields; subsequent local Rust E2E coverage is listed above.

Local reproduction did not reproduce the macOS timeouts. On unchanged f28132b, using pinned CLI 1.0.84-4, both affected groups passed in both transports on Linux. Because filtered runs restart the shared client after every test, the complete unfiltered E2E suites were also run with the CI feature, concurrency, and timeout settings. Both affected cases executed successfully in those suites. These local passes do not establish the macOS failure mechanism or count toward independent CI proof.

Latest candidate CI is not green. In first-attempt run 34651337391, all Python, Go, and .NET legs passed. The Rust model-listing cases now pass on both macOS transports, and the other Rust test legs passed. Two different macOS Rust cases failed: the default transport timed out collecting events through session.idle in should_emit_pending_messages_modified_event_when_message_queue_changes; the in-process transport timed out after 60 seconds in send_and_wait in empty_mode_system_message_replace_llm_follows_caller_content_verbatim. Neither reached its final behavior assertion. These failures remain under investigation, without retries or relaxed timeouts.

Earlier failures are preserved. The previous Go macOS in-process permission-denial test did not observe permission.completed on client1; client2 observed the denial and the protected file remained unchanged. The previous macOS .NET shard-1 job exceeded its 20-minute job timeout, but its log retrieval failed and no diagnostic artifact exists, so build, execution, and teardown cannot be distinguished. Their new-head passes do not establish what caused the earlier failures. Neither is claimed fixed by the catalog correction.

Pinned C# integration proof complete: 10/10. All ten independent, consecutive workflow_dispatch runs below passed on their first attempts with SDK f28132b956816e8b8d26f6f83a97d9e2215f52b0 and runtime 73442c32d5311e27ff26880074a0e2e4c416b5f0. Each preparation log confirms the requested and actually checked-out SDK SHA. No test/job reruns or results from another revision are included.

Every run executed all four backend test steps successfully. CAPI passed 893 tests with four existing skips, including both original blob-attachment cases and the disabled-then-enabled vision case. Anthropic Messages, OpenAI Completions, and OpenAI Responses each passed 466 tests with four existing skips in every run. Full-workflow results, test-step outcomes, success markers, populations, and all three original cases were verified from the job records and logs.

These ten runs were dispatched by a separate process using devm33 on the runtime helper PR's then-current branch. That helper later advanced to another runtime revision; these results are not proof for its newer head. They do establish this unchanged SDK candidate's compatibility with the single runtime revision above. This PR remains draft because the broader SDK macOS Rust CI failures described above are unresolved. Completing this C# proof does not erase them or prove other SDK languages/platforms stable. Integration links require runtime-repository access.

A newer runtime pair also has unresolved failures. With this same verified SDK commit and runtime b507360811e60d1d01e992d6d562af6f2802aa34, a separate ten-run wave reported eight successes and two failures: OpenAI Responses run 34664067194 aborted because its test host crashed after 210 passing tests, while OpenAI Completions run 34664076296 exceeded the 15-minute test-step deadline after partial progress. Neither identifies a failing test or established cause, and neither retained crash/hang diagnostics. They are not rerun, dismissed as image-catalog failures, or combined with the fixed-runtime proof below. This is another reason not to claim broad CI stability or mark the PR ready.

Later same-pair observations remain unresolved. A September 12 snapshot of ten later external dispatches on the same newer runtime/SDK pair reported nine successful workflows and one failure; all ten requested and actually prepared the unchanged SDK candidate. Those nine API successes are not a full-population proof or evidence of a causal repair.

The additional CAPI failure was SessionE2ETests.Should_Create_A_Session_With_Replaced_SystemMessage_Config: the existing 120-second assistant/idle helper timed out after create and send returned, before the final assertions. This case uses the shared TCP client. The helper also backfills event history; which required observation was missing remains unknown. The job completed with 897 total, 892 passed, one failed, four skipped, and all three original image/vision cases passed. The earlier host crash and 15-minute hang remain unresolved. No newer-runtime proof is credited (0/10), and this PR remains draft with merge readiness unresolved. These observations do not alter the historical ten-run proof or separate runtime-31d confirmations below.

Workflow / platform Run number and link SDK commit Runtime commit Outcome
C# / Linux, four backends 58133 / 34662650224 f28132b956 73442c32d5 Passed attempt 1
C# / Linux, four backends 58134 / 34662652697 f28132b956 73442c32d5 Passed attempt 1
C# / Linux, four backends 58135 / 34662655718 f28132b956 73442c32d5 Passed attempt 1
C# / Linux, four backends 58136 / 34662659445 f28132b956 73442c32d5 Passed attempt 1
C# / Linux, four backends 58137 / 34662663297 f28132b956 73442c32d5 Passed attempt 1
C# / Linux, four backends 58138 / 34662667171 f28132b956 73442c32d5 Passed attempt 1
C# / Linux, four backends 58139 / 34662670744 f28132b956 73442c32d5 Passed attempt 1
C# / Linux, four backends 58140 / 34662674633 f28132b956 73442c32d5 Passed attempt 1
C# / Linux, four backends 58141 / 34662678497 f28132b956 73442c32d5 Passed attempt 1
C# / Linux, four backends 58142 / 34662682355 f28132b956 73442c32d5 Passed attempt 1

Separate confirmations on runtime 31d3910d62c056099d6867034f5c63359cabc7c6 also passed with this SDK: 34661817072, 34662939810, and 34662940100. They are not combined with the ten-run pair above. Earlier 05f0a4b SDK results remain excluded.

Advertise room for the two image-bearing messages in replay fixtures and verify the HTTP models response for missing, empty, and cached model catalogs.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 7e3dcf84-04c4-4bb7-8a42-7ca5904f3cf7
@devm33 devm33 added sdk/dotnet testing post-to-slack Post PR to reviewers Slack channel deep-review Requested detailed code review skip-changelog No user-facing changelog entry required labels Sep 11, 2026
Return complete vision limits compatible with generated SDK model parsers. Validate the HTTP catalog fields and check the fixture against the shared SDK vision type.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 7e3dcf84-04c4-4bb7-8a42-7ca5904f3cf7
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

deep-review Requested detailed code review post-to-slack Post PR to reviewers Slack channel sdk/dotnet skip-changelog No user-facing changelog entry required testing

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Replay catalog omits the image capacity required by saved histories

2 participants