Fix replay model image capacity - #2629
Draft
devm33 wants to merge 2 commits into
Draft
Conversation
Advertise room for the two image-bearing messages in replay fixtures and verify the HTTP models response for missing, empty, and cached model catalogs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 7e3dcf84-04c4-4bb7-8a42-7ca5904f3cf7
Return complete vision limits compatible with generated SDK model parsers. Validate the HTTP catalog fields and check the fixture against the shared SDK vision type. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 7e3dcf84-04c4-4bb7-8a42-7ca5904f3cf7
stephentoub
approved these changes
Sep 12, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Closes #2628.
The replay proxy advertises vision support without declaring its image capacity. Recorded conversations can contain two image-bearing messages: an attached blob and a later image-view result. A runtime using a one-image fallback removes or rejects image content that the strict saved history still expects, causing a local replay mismatch.
These tests protect meaningful attachment and model-vision behavior. The fix belongs in the fake model catalog, not in retries or weakened image assertions.
Changes
Declare a complete
capabilities.limits.visionobject in the replay model catalog: capacity for two images, supported image formats, and the maximum image size. The latter fields are required by typed SDK consumers. Exercise the actual HTTP/modelsresponse for missing captures, empty catalogs, and every cached model, and check compatibility with the public generated SDK type.No SDK product code, attachment/vision assertions, snapshots, workflow filters, timeouts, or retry settings change.
Validation
CI correction: the initial SDK CI run, on
05f0a4b, exposed a regression in this PR: its partial vision object caused typed deserialization failures in eight Python and four Rust jobs. Commitf28132b956816e8b8d26f6f83a97d9e2215f52b0supplies the required fields and adds contract coverage. Rust's original failures were attributed from their exact logs and required serde fields; subsequent local Rust E2E coverage is listed above.Local reproduction did not reproduce the macOS timeouts. On unchanged
f28132b, using pinned CLI1.0.84-4, both affected groups passed in both transports on Linux. Because filtered runs restart the shared client after every test, the complete unfiltered E2E suites were also run with the CI feature, concurrency, and timeout settings. Both affected cases executed successfully in those suites. These local passes do not establish the macOS failure mechanism or count toward independent CI proof.Latest candidate CI is not green. In first-attempt run 34651337391, all Python, Go, and .NET legs passed. The Rust model-listing cases now pass on both macOS transports, and the other Rust test legs passed. Two different macOS Rust cases failed: the default transport timed out collecting events through
session.idleinshould_emit_pending_messages_modified_event_when_message_queue_changes; the in-process transport timed out after 60 seconds insend_and_waitinempty_mode_system_message_replace_llm_follows_caller_content_verbatim. Neither reached its final behavior assertion. These failures remain under investigation, without retries or relaxed timeouts.Earlier failures are preserved. The previous Go macOS in-process permission-denial test did not observe
permission.completedon client1; client2 observed the denial and the protected file remained unchanged. The previous macOS .NET shard-1 job exceeded its 20-minute job timeout, but its log retrieval failed and no diagnostic artifact exists, so build, execution, and teardown cannot be distinguished. Their new-head passes do not establish what caused the earlier failures. Neither is claimed fixed by the catalog correction.Pinned C# integration proof complete: 10/10. All ten independent, consecutive
workflow_dispatchruns below passed on their first attempts with SDKf28132b956816e8b8d26f6f83a97d9e2215f52b0and runtime73442c32d5311e27ff26880074a0e2e4c416b5f0. Each preparation log confirms the requested and actually checked-out SDK SHA. No test/job reruns or results from another revision are included.Every run executed all four backend test steps successfully. CAPI passed 893 tests with four existing skips, including both original blob-attachment cases and the disabled-then-enabled vision case. Anthropic Messages, OpenAI Completions, and OpenAI Responses each passed 466 tests with four existing skips in every run. Full-workflow results, test-step outcomes, success markers, populations, and all three original cases were verified from the job records and logs.
These ten runs were dispatched by a separate process using
devm33on the runtime helper PR's then-current branch. That helper later advanced to another runtime revision; these results are not proof for its newer head. They do establish this unchanged SDK candidate's compatibility with the single runtime revision above. This PR remains draft because the broader SDK macOS Rust CI failures described above are unresolved. Completing this C# proof does not erase them or prove other SDK languages/platforms stable. Integration links require runtime-repository access.A newer runtime pair also has unresolved failures. With this same verified SDK commit and runtime
b507360811e60d1d01e992d6d562af6f2802aa34, a separate ten-run wave reported eight successes and two failures: OpenAI Responses run 34664067194 aborted because its test host crashed after 210 passing tests, while OpenAI Completions run 34664076296 exceeded the 15-minute test-step deadline after partial progress. Neither identifies a failing test or established cause, and neither retained crash/hang diagnostics. They are not rerun, dismissed as image-catalog failures, or combined with the fixed-runtime proof below. This is another reason not to claim broad CI stability or mark the PR ready.Later same-pair observations remain unresolved. A September 12 snapshot of ten later external dispatches on the same newer runtime/SDK pair reported nine successful workflows and one failure; all ten requested and actually prepared the unchanged SDK candidate. Those nine API successes are not a full-population proof or evidence of a causal repair.
The additional CAPI failure was
SessionE2ETests.Should_Create_A_Session_With_Replaced_SystemMessage_Config: the existing 120-second assistant/idle helper timed out after create and send returned, before the final assertions. This case uses the shared TCP client. The helper also backfills event history; which required observation was missing remains unknown. The job completed with 897 total, 892 passed, one failed, four skipped, and all three original image/vision cases passed. The earlier host crash and 15-minute hang remain unresolved. No newer-runtime proof is credited (0/10), and this PR remains draft with merge readiness unresolved. These observations do not alter the historical ten-run proof or separate runtime-31d confirmations below.f28132b95673442c32d5f28132b95673442c32d5f28132b95673442c32d5f28132b95673442c32d5f28132b95673442c32d5f28132b95673442c32d5f28132b95673442c32d5f28132b95673442c32d5f28132b95673442c32d5f28132b95673442c32d5Separate confirmations on runtime
31d3910d62c056099d6867034f5c63359cabc7c6also passed with this SDK: 34661817072, 34662939810, and 34662940100. They are not combined with the ten-run pair above. Earlier05f0a4bSDK results remain excluded.