Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
WalkthroughReplaces fixed delays and manual mailbox polling in e2e email tests with a new Mailbox.waitForMessagesWithSubject(subject) helper; implements polling + caching in Mailbox.fetchMessages and updates tests to assert the returned MailboxMessage snapshots. All edits are test-only. Changes
Sequence Diagram(s)sequenceDiagram
autonumber
actor Test
participant Mailbox as Mailbox (tests/helpers)
participant Server as Email API / Server
Note over Test,Mailbox: Test requests email delivery (triggered by app flow)
Test->>Mailbox: waitForMessagesWithSubject(subject)
activate Mailbox
Mailbox->>Server: fetchMessages() [full list]
Server-->>Mailbox: messages (headers + maybe bodies)
Mailbox->>Mailbox: cache full bodies by message.id
alt matches found
Mailbox-->>Test: [MailboxMessage[] matching subject]
else no match
Mailbox->>Mailbox: wait(200) then retry (up to 20)
Mailbox-->>Test: throw "Message with subject ... not found" (after retries)
end
deactivate Mailbox
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes Possibly related PRs
Suggested reviewers
Poem
Pre-merge checks (2 passed, 1 inconclusive)❌ Failed checks (1 inconclusive)
✅ Passed checks (2 passed)
✨ Finishing touches
🧪 Generate unit tests
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Greptile Summary
This PR refactors email testing infrastructure to improve test reliability and reduce flakiness. The changes introduce a new waitForMessagesWithSubject helper method that implements proper polling instead of using fixed delays when waiting for emails.
Key Changes:
- New utility method: Adds
waitForMessagesWithSubjectto the Mailbox class inhelpers.tsthat polls for messages with a specific subject line (up to 20 retries with 200ms intervals) - Test refactoring: Updates email-related tests in
send-email.test.tsandunsubscribe-link.test.tsto replace manualwait(2000)/wait(3000)calls followed by message filtering with the new helper method - Improved reliability: Eliminates race conditions where tests might fail if emails don't arrive within arbitrary timeout periods
The polling mechanism waits up to 4 seconds total (20 × 200ms) for expected emails to arrive, making tests more deterministic while still being reasonably fast when emails arrive quickly. This pattern follows testing best practices by using condition-based waiting instead of fixed delays, which is particularly important for CI/CD environments where timing can be unpredictable. The changes maintain identical test behavior but with better reliability guarantees, bringing consistency to how email testing is handled across the codebase.
Confidence score: 5/5
- This PR is safe to merge with minimal risk as it only improves test infrastructure without changing production code
- Score reflects that these are well-contained test improvements using established patterns (polling with reasonable timeouts)
- No files require special attention as all changes are straightforward test reliability improvements
3 files reviewed, no comments
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (3)
apps/e2e/tests/backend/endpoints/api/v1/send-email.test.ts (1)
595-597: Minor: await mailbox waits in parallel to speed up test- await userA.mailbox.waitForMessagesWithSubject(subject); - await userB.mailbox.waitForMessagesWithSubject(subject); - await userC.mailbox.waitForMessagesWithSubject(subject); + await Promise.all([ + userA.mailbox.waitForMessagesWithSubject(subject), + userB.mailbox.waitForMessagesWithSubject(subject), + userC.mailbox.waitForMessagesWithSubject(subject), + ]);apps/e2e/tests/helpers.ts (2)
200-203: Consider options on waitForMessagesWithSubject (includeBody, retries, interval)Proposed signature change (backward compatible defaults keep current behavior):
- public readonly waitForMessagesWithSubject: (subject: string) => Promise<MailboxMessage[]>; + public readonly waitForMessagesWithSubject: ( + subject: string, + options?: { includeBody?: boolean; maxRetries?: number; intervalMs?: number } + ) => Promise<MailboxMessage[]>;
227-238: Optional: allow returning messages with bodies (unblocks body assertions without a second fetch)This keeps default noBody behavior for existing snapshots, but enables consumers to request bodies when needed.
- this.waitForMessagesWithSubject = async (subject: string) => { - const maxRetries = 20; - for (let i = 0; i < maxRetries; i++) { - const messages = await this.fetchMessages({ noBody: true }); - const withSubject = messages.filter(m => m.subject === subject); - if (withSubject.length > 0) { - return withSubject; - } - await wait(200); - } - throw new Error(`Message with subject ${subject} not found`); - }; + this.waitForMessagesWithSubject = async ( + subject: string, + options?: { includeBody?: boolean; maxRetries?: number; intervalMs?: number } + ) => { + const maxRetries = options?.maxRetries ?? 20; + const intervalMs = options?.intervalMs ?? 200; + for (let i = 0; i < maxRetries; i++) { + const messages = await this.fetchMessages({ noBody: true }); + const withSubject = messages.filter(m => m.subject === subject); + if (withSubject.length > 0) { + if (options?.includeBody) { + const full = await this.fetchMessages(); // uses cache + return full.filter(m => m.subject === subject); + } + return withSubject; + } + await wait(intervalMs); + } + throw new Error(`Message with subject ${subject} not found after ${maxRetries} retries`); + };
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (3)
apps/e2e/tests/backend/endpoints/api/v1/send-email.test.ts(5 hunks)apps/e2e/tests/backend/endpoints/api/v1/unsubscribe-link.test.ts(2 hunks)apps/e2e/tests/helpers.ts(4 hunks)
🧰 Additional context used
📓 Path-based instructions (2)
**/*.{ts,tsx,js,jsx}
📄 CodeRabbit inference engine (AGENTS.md)
Prefer ES6 Map over Record when representing key–value collections
Files:
apps/e2e/tests/helpers.tsapps/e2e/tests/backend/endpoints/api/v1/unsubscribe-link.test.tsapps/e2e/tests/backend/endpoints/api/v1/send-email.test.ts
**/*.test.{ts,tsx,js}
📄 CodeRabbit inference engine (AGENTS.md)
In tests, prefer .toMatchInlineSnapshot where possible; refer to snapshot-serializer.ts for snapshot formatting and handling of non-deterministic values
Files:
apps/e2e/tests/backend/endpoints/api/v1/unsubscribe-link.test.tsapps/e2e/tests/backend/endpoints/api/v1/send-email.test.ts
🧬 Code graph analysis (1)
apps/e2e/tests/helpers.ts (1)
packages/stack-shared/src/utils/promises.tsx (1)
wait(260-268)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (9)
- GitHub Check: Security Check
- GitHub Check: build (22.x)
- GitHub Check: lint_and_build (latest)
- GitHub Check: restart-dev-and-test
- GitHub Check: all-good
- GitHub Check: setup-tests
- GitHub Check: build (22.x)
- GitHub Check: docker
- GitHub Check: docker
🔇 Additional comments (7)
apps/e2e/tests/backend/endpoints/api/v1/send-email.test.ts (4)
297-307: LGTM: subject-based wait with stable inline snapshotGood switch to mailbox.waitForMessagesWithSubject and snapshotting metadata (avoids brittle HTML assertions).
412-412: LGTM: waiting on subject is sufficientThis ensures the draft send completed before asserting draft state.
741-741: LGTM: transactional subject waitConsistent with the new helper; avoids HTML coupling.
780-780: LGTM: default category subject waitConsistent usage across tests.
apps/e2e/tests/helpers.ts (3)
4-4: LGTM: import wait for polling helper
133-133: LGTM: cosmetic NiceResponse constructor change
211-226: Nice: full-message cache avoids redundant network callsCaching the full message while optionally omitting body for snapshots keeps polling cheap and snapshots stable.
| const messages = await user.mailbox.waitForMessagesWithSubject("Custom Test Email Subject"); | ||
| const sentEmail = messages[0]; | ||
| expect(sentEmail!.body?.html).toMatchInlineSnapshot(`"http://localhost:8102/api/v1/emails/unsubscribe-link?code=%3Cstripped+query+param%3E"`); |
There was a problem hiding this comment.
Fix: waitForMessagesWithSubject returns messages without bodies; fetch full message before asserting body
waitForMessagesWithSubject uses fetchMessages({ noBody: true }), so sentEmail.body is undefined and the snapshot/assertions will fail. Fetch the full message after waiting.
- const messages = await user.mailbox.waitForMessagesWithSubject("Custom Test Email Subject");
- const sentEmail = messages[0];
+ await user.mailbox.waitForMessagesWithSubject("Custom Test Email Subject");
+ const sentEmail = (await user.mailbox.fetchMessages()).find(m => m.subject === "Custom Test Email Subject")!;Optionally make the link extraction robust (handles either plain-URL body or an link):
- const unsubscribeLinkMatch = sentEmail!.body?.html.match(/href="([^"]+)"/);
- expect(unsubscribeLinkMatch).toBeDefined();
- const unsubscribeUrl = unsubscribeLinkMatch![1];
+ const html = sentEmail!.body?.html ?? "";
+ const unsubscribeUrl = html.includes('href="')
+ ? (html.match(/href="([^"]+)"/)![1])
+ : html.trim();
+ expect(unsubscribeUrl).toMatch(/^https?:\/\//);📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| const messages = await user.mailbox.waitForMessagesWithSubject("Custom Test Email Subject"); | |
| const sentEmail = messages[0]; | |
| expect(sentEmail!.body?.html).toMatchInlineSnapshot(`"http://localhost:8102/api/v1/emails/unsubscribe-link?code=%3Cstripped+query+param%3E"`); | |
| await user.mailbox.waitForMessagesWithSubject("Custom Test Email Subject"); | |
| const sentEmail = (await user.mailbox.fetchMessages()).find(m => m.subject === "Custom Test Email Subject")!; | |
| expect(sentEmail!.body?.html).toMatchInlineSnapshot(`"http://localhost:8102/api/v1/emails/unsubscribe-link?code=%3Cstripped+query+param%3E"`); |
| const messages = await user.mailbox.waitForMessagesWithSubject("Custom Test Email Subject"); | ||
| const sentEmail = messages[0]; | ||
| expect(sentEmail).toBeDefined(); |
There was a problem hiding this comment.
Same issue here: fetch full message before using body
This block also reads body from a noBody result.
- const messages = await user.mailbox.waitForMessagesWithSubject("Custom Test Email Subject");
- const sentEmail = messages[0];
+ await user.mailbox.waitForMessagesWithSubject("Custom Test Email Subject");
+ const sentEmail = (await user.mailbox.fetchMessages()).find(m => m.subject === "Custom Test Email Subject")!;📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| const messages = await user.mailbox.waitForMessagesWithSubject("Custom Test Email Subject"); | |
| const sentEmail = messages[0]; | |
| expect(sentEmail).toBeDefined(); | |
| await user.mailbox.waitForMessagesWithSubject("Custom Test Email Subject"); | |
| const sentEmail = (await user.mailbox.fetchMessages()).find(m => m.subject === "Custom Test Email Subject")!; | |
| expect(sentEmail).toBeDefined(); |
🤖 Prompt for AI Agents
In apps/e2e/tests/backend/endpoints/api/v1/unsubscribe-link.test.ts around lines
143 to 145, the test accesses the message body without first fetching the full
message; modify the code to retrieve the complete message (e.g., call the
mailbox API to fetch the full message by id or use a provided method like
getMessage or fetchFullMessage) before reading its body, then assert on the
body—ensure you replace usages of messages[0].body with the
fetchedFullMessage.body (or equivalent) so you never read from a noBody result.
There was a problem hiding this comment.
Actionable comments posted: 0
♻️ Duplicate comments (1)
apps/e2e/tests/helpers.ts (1)
235-235: Re: automated comment about runAsynchronously — this is already awaited.
The pattern in Line 235 usesawait wait(200), not fire-and-forget; the bot warning looks like a false positive.
🧹 Nitpick comments (2)
apps/e2e/tests/helpers.ts (2)
210-226: Avoid N+1 full-message fetches when noBody is true.
When callers only need headers/subject (e.g., polling), short-circuit using the list payload to skip per-message detail fetches. Keeps Map usage (preferred over Record) and reduces network/CPU.Apply this diff:
this.fetchMessages = async ({ noBody } = {}) => { - const res = await niceFetch(new URL(`/api/v1/mailbox/${encodeURIComponent(mailboxName)}`, INBUCKET_API_URL)); - return await Promise.all((res.body as any[]).map(async (message) => { - let fullMessage: any; - if (fullMessageCache.has(message.id)) { - fullMessage = fullMessageCache.get(message.id); - } else { - const fullMessageRes = await niceFetch(new URL(`/api/v1/mailbox/${encodeURIComponent(mailboxName)}/${message.id}`, INBUCKET_API_URL)); - fullMessage = fullMessageRes.body; - fullMessageCache.set(message.id, fullMessage); - } - const messagePart = noBody ? omit(fullMessage, ["body", "attachments"]) : fullMessage; - return new MailboxMessage(messagePart); - })); + const res = await niceFetch(new URL(`/api/v1/mailbox/${encodeURIComponent(mailboxName)}`, INBUCKET_API_URL)); + const list = res.body as any[]; + if (noBody) { + // Build from list payload to avoid N extra network calls + return list.map((message) => new MailboxMessage(omit(message, ["body", "attachments"]))); + } + return await Promise.all(list.map(async (message) => { + let fullMessage: any = fullMessageCache.get(message.id); + if (!fullMessage) { + const fullMessageRes = await niceFetch(new URL(`/api/v1/mailbox/${encodeURIComponent(mailboxName)}/${message.id}`, INBUCKET_API_URL)); + fullMessage = fullMessageRes.body; + fullMessageCache.set(message.id, fullMessage); + } + return new MailboxMessage(fullMessage); + })); };If the list payload lacks some fields needed by MailboxMessage in your snapshots, keep the current path for those cases and only short-circuit when safe.
227-238: Make polling cheaper and error messages more actionable.
Use noBody during polling and include mailbox/elapsed time + last subjects to ease debugging timeouts.this.waitForMessagesWithSubject = async (subject: string) => { - const maxRetries = 20; - for (let i = 0; i < maxRetries; i++) { - const messages = await this.fetchMessages(); - const withSubject = messages.filter(m => m.subject === subject); + const maxRetries = 20; + const intervalMs = 200; + let lastSubjects: string[] = []; + for (let i = 0; i < maxRetries; i++) { + const messages = await this.fetchMessages({ noBody: true }); + lastSubjects = messages.map(m => m.subject); + const withSubject = messages.filter(m => m.subject === subject); if (withSubject.length > 0) { return withSubject; } - await wait(200); + await wait(intervalMs); } - throw new Error(`Message with subject ${subject} not found`); + throw new Error(`Message with subject "${subject}" not found for ${this.emailAddress} after ${maxRetries} retries (~${maxRetries * intervalMs}ms). Last subjects seen: ${lastSubjects.slice(0, 10).join(", ")}`); };
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (1)
apps/e2e/tests/helpers.ts(4 hunks)
🧰 Additional context used
📓 Path-based instructions (1)
**/*.{ts,tsx,js,jsx}
📄 CodeRabbit inference engine (AGENTS.md)
Prefer ES6 Map over Record when representing key–value collections
Files:
apps/e2e/tests/helpers.ts
🧬 Code graph analysis (1)
apps/e2e/tests/helpers.ts (1)
packages/stack-shared/src/utils/promises.tsx (1)
wait(260-268)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (9)
- GitHub Check: lint_and_build (latest)
- GitHub Check: setup-tests
- GitHub Check: docker
- GitHub Check: docker
- GitHub Check: all-good
- GitHub Check: build (22.x)
- GitHub Check: restart-dev-and-test
- GitHub Check: build (22.x)
- GitHub Check: Security Check
🔇 Additional comments (3)
apps/e2e/tests/helpers.ts (3)
4-4: Import for wait is appropriate and used correctly.
No issues; aligns with the new polling helper.
133-133: Constructor formatting change is a no-op.
Safe cosmetic tweak; no semantic impact.
202-202: Good addition: typed polling helper on Mailbox.
Clear API; returning an array is flexible for multi-send scenarios.
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (5)
apps/e2e/tests/backend/endpoints/api/v1/send-email.test.ts (5)
297-312: Assert cardinality and consider snapshotting only stable fields to reduce flakinessAdd an explicit length assertion so the test fails clearly if multiple emails with the same subject arrive. Optionally, snapshot a stable projection instead of the full message (body can churn).
Minimal addition:
const messages = await user.mailbox.waitForMessagesWithSubject("Custom Test Email Subject"); + expect(messages).toHaveLength(1); expect(messages).toMatchInlineSnapshot(`Optional alternative (snapshot stable fields):
- const messages = await user.mailbox.waitForMessagesWithSubject("Custom Test Email Subject"); - expect(messages).toMatchInlineSnapshot(` - [ - MailboxMessage { - "attachments": [], - "body": { - "html": "http://localhost:8102/api/v1/emails/unsubscribe-link?code=%3Cstripped+query+param%3E", - "text": "http://localhost:8102/api/v1/emails/unsubscribe-link?code=%3Cstripped+query+param%3E", - }, - "from": "Test Project <test@example.com>", - "subject": "Custom Test Email Subject", - "to": ["<unindexed-mailbox--<stripped UUID>@stack-generated.example.com>"], - <some fields may have been hidden>, - }, - ] - `); + const messages = await user.mailbox.waitForMessagesWithSubject("Custom Test Email Subject"); + expect(messages).toHaveLength(1); + expect(messages.map(m => ({ + from: m.from, + to: m.to, + subject: m.subject, + attachments: m.attachments.length, + }))).toMatchInlineSnapshot(` + [ + { + "from": "Test Project <test@example.com>", + "to": ["<unindexed-mailbox--<stripped UUID>@stack-generated.example.com>"], + "subject": "Custom Test Email Subject", + "attachments": 0, + }, + ] + `);
417-417: Validate the awaited result, not just completionCapture the returned array and assert at least one (ideally exactly one) message to tighten the check.
- await user.mailbox.waitForMessagesWithSubject("Overridden Subject"); + const msgs = await user.mailbox.waitForMessagesWithSubject("Overridden Subject"); + expect(msgs).toHaveLength(1);
600-602: Parallelize mailbox waits to speed up the testWaiting sequentially adds unnecessary latency; use Promise.all.
- await userA.mailbox.waitForMessagesWithSubject(subject); - await userB.mailbox.waitForMessagesWithSubject(subject); - await userC.mailbox.waitForMessagesWithSubject(subject); + await Promise.all( + [userA, userB, userC].map(u => u.mailbox.waitForMessagesWithSubject(subject)) + );
746-746: Assert cardinality for the transactional emailEnsure exactly one message arrived with the subject.
- await user.mailbox.waitForMessagesWithSubject("Transactional Test Subject"); + const msgs = await user.mailbox.waitForMessagesWithSubject("Transactional Test Subject"); + expect(msgs).toHaveLength(1);
785-785: Assert cardinality for the default category emailSame rationale as above: make the expectation explicit.
- await user.mailbox.waitForMessagesWithSubject("Default Category Test Subject"); + const msgs = await user.mailbox.waitForMessagesWithSubject("Default Category Test Subject"); + expect(msgs).toHaveLength(1);
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (1)
apps/e2e/tests/backend/endpoints/api/v1/send-email.test.ts(5 hunks)
🧰 Additional context used
📓 Path-based instructions (2)
**/*.test.{ts,tsx,js}
📄 CodeRabbit inference engine (AGENTS.md)
In tests, prefer .toMatchInlineSnapshot where possible; refer to snapshot-serializer.ts for snapshot formatting and handling of non-deterministic values
Files:
apps/e2e/tests/backend/endpoints/api/v1/send-email.test.ts
**/*.{ts,tsx,js,jsx}
📄 CodeRabbit inference engine (AGENTS.md)
Prefer ES6 Map over Record when representing key–value collections
Files:
apps/e2e/tests/backend/endpoints/api/v1/send-email.test.ts
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (8)
- GitHub Check: docker
- GitHub Check: build (22.x)
- GitHub Check: lint_and_build (latest)
- GitHub Check: build (22.x)
- GitHub Check: docker
- GitHub Check: all-good
- GitHub Check: setup-tests
- GitHub Check: Security Check
High-level PR Summary
This PR improves email testing in the end-to-end test suite by adding a new
waitForMessagesWithSubjecthelper method to theMailboxclass. This method replaces the previous pattern of using arbitrary wait times (e.g.,wait(2000)) followed by fetching and finding messages, which could lead to flaky tests. The new approach implements a polling mechanism that waits until messages with the specified subject appear, with a reasonable timeout. The PR updates all email-related tests to use this new helper method, making the tests more reliable and less dependent on timing assumptions.⏱️ Estimated Review Time: 0h 15m
💡 Review Order Suggestion
apps/e2e/tests/helpers.tsapps/e2e/tests/backend/endpoints/api/v1/send-email.test.tsapps/e2e/tests/backend/endpoints/api/v1/unsubscribe-link.test.tsImportant
Improves email test reliability by adding
waitForMessagesWithSubjectinMailboxand updating tests to use it.waitForMessagesWithSubjectinMailboxclass inhelpers.tsto replace arbitrary wait times with a polling mechanism.send-email.test.tsandunsubscribe-link.test.tsto usewaitForMessagesWithSubjectfor more reliable email verification.wait()calls withwaitForMessagesWithSubject()insend-email.test.tsandunsubscribe-link.test.ts.helpers.tsto support new functionality.This description was created by
for 90b7bda. You can customize this summary. It will automatically update as commits are pushed.
Review by RecurseML
🔍 Review performed on d14317c..f42dfc5
✅ Files analyzed, no issues (2)
•
apps/e2e/tests/backend/endpoints/api/v1/send-email.test.ts•
apps/e2e/tests/backend/endpoints/api/v1/unsubscribe-link.test.tsSummary by CodeRabbit
Tests
Chores