feat(harmony-client): live PTT voice recognition in Practice (ZEB-152 slice 3) - #48

PR author is in the excluded authors list.

CodeAnt AI is reviewing your PR.

No actionable comments were generated in the recent review. 🎉

sequenceDiagram
    actor User
    participant FlashcardView
    participant AudioCapture
    participant stq8Service

    User->>FlashcardView: Press PTT
    activate FlashcardView
    FlashcardView->>FlashcardView: clear pcmBuffer, set pttActive/segmentStart
    FlashcardView->>AudioCapture: ensureCapture()/start()
    activate AudioCapture
    AudioCapture-->>FlashcardView: streaming PCM frames
    deactivate AudioCapture

    Note over User,AudioCapture: PCM buffered while PTT held

    User->>FlashcardView: Release PTT
    activate FlashcardView
    FlashcardView->>FlashcardView: flush pcmBuffer -> pcm
    FlashcardView->>stq8Service: processPcm(pcm)
    activate stq8Service
    stq8Service-->>FlashcardView: heardNibbles (or empty / error)
    deactivate stq8Service

    alt heardNibbles non-empty
        FlashcardView->>FlashcardView: handleRowComplete(heardNibbles)
    else empty or error
        FlashcardView->>FlashcardView: cancel/reset combo (fallback)
    end
    deactivate FlashcardView

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

Review Summary by Qodo

Integrate live PTT voice recognition into Practice mode

✨ Enhancement

Walkthroughs

• Integrates live voice capture into Practice mode PTT button
• Flushes captured PCM through classifier on button release
• Disables PTT with calibration hint when classifier uncalibrated
• Adds lazy-async AudioCapture initialization and cleanup on unmount

flowchart LR
  PTTPress["PTT Button Press"] -->|ensureCapture| AudioCapture["AudioCapture Start"]
  AudioCapture -->|onPcmFrame| PCMBuffer["PCM Buffer"]
  PTTRelease["PTT Button Release"] -->|flushAndClassify| Classifier["processPcm"]
  Classifier -->|syllables| RowEval["handleRowComplete"]
  RowEval -->|pass/fail| RowState["Update Row State"]
  Unmount["Component Unmount"] -->|cleanup| StopCapture["AudioCapture Stop"]
  CalibCheck["isCalibrated Check"] -->|false| DisablePTT["Disable PTT + Show Hint"]

File Changes

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 📎 Requirement gaps (0)

ⓘ The new review experience is currently in Beta. Learn more

User description

Summary

• Wires AudioCapture → WasmPipeline.process() into FlashcardView's PTT handlers. Holding the button captures PCM, releasing flushes it through the classifier, and the resulting syllable nibbles feed the existing handleRowComplete() evaluation path. This is the ZEB-152 payoff slice: Calibrate-then-Practice now works end-to-end for the first time.
• Lazy-async capture start on first press, one AudioCapture instance reused across holds for the component's lifetime, $effect cleanup on unmount releases the mic when you leave the Practice tab.
• PTT disabled with a "Calibrate your voice..." hint when !stq8Service.isCalibrated(), so an uncalibrated classifier can't emit garbage nibbles at the row-progression logic. SpellbookMode already remounts FlashcardView on tab switch, so the gate re-evaluates after a fresh calibration.

Design decisions worth flagging

PTT release as the row-attempt commit point. If PCM was captured and processPcm returned non-empty syllables, handleRowComplete(nibbles) runs and its existing pass/fail/combo logic takes over untouched. Empty buffer, empty syllables, or a thrown processPcm all fall through to the pre-Slice-3 "cancel row, break combo" path — one fallback covers "didn't hold long enough," "mic permission still resolving," "classifier heard nothing," and "WASM error" consistently instead of four bespoke paths.

Lazy capture vs prefetch on mount. Considered starting capture when the Practice tab mounts, but that keeps the mic indicator on even when the user is just reading cards. Lazy-on-first-press means first hold may catch less PCM than expected (getUserMedia latency), but the fallback path handles that cleanly and subsequent holds are tight.

Prop type widened to Stq8ServiceLike. FlashcardView previously had an inline 3-method shape for stq8Service; with Slice 3 needing isCalibrated + processPcm the inline shape became duplication. Switched to the exported interface to keep the contract in one place.

Out of scope (Slice 4 polish)

• Mismatch display from flashcard-design.md:70-78 (expected vs heard with caret under first-differing byte)
• 2-second momentum timeout during held PTT (spec: progress resets if no syllable advances for 2s)
• "Couldn't hear that clearly" UX for the empty-syllable release case (currently just the silent cancel path)
• Permission-denial recovery flow (currently surfaces error message text below the button)

Verification

• 1121 tests pass (+2 new calibration-gate UI tests in FlashcardView.test.ts)
• svelte-check clean on all touched files (two pre-existing initial-value warnings on isCalibrated/stq8Service unchanged)
• vite build clean — main bundle +2.8 KB gzipped (capture wiring)
• Manual end-to-end verification pending: calibrate on Ildwyn, switch to Practice, hold PTT, speak a Novice-level byte, confirm row-progression feels right

Test plan

• Pull, npm run dev, calibrate once on the Calibrate tab
• Switch to Practice tab, verify PTT button is enabled (no "Calibrate first" hint)
• Hold PTT, say a single Q8 syllable matching the active row, release → row should evaluate (green on match, red flash + reset on mismatch)
• Hold PTT without speaking, release → row cancels, combo breaks (pre-Slice-3 behavior preserved)
• Switch tabs mid-session, verify mic indicator clears (capture stops on unmount)
• Recalibrate mid-session, return to Practice, verify the new profile is in effect

Related: ZEB-152, closes part of ZEB-152 (Slice 1 already landed as #47; Slice 4 is follow-up polish).

🤖 Generated with Claude Code

CodeAnt-AI Description

Use push-to-talk voice input in Practice, with calibration required first

What Changed

• Practice now sends held push-to-talk audio through voice recognition on release, so spoken input can complete the current row instead of always canceling it.
• Push-to-talk is disabled until voice calibration is finished, with a clear hint telling users to calibrate first.
• If the microphone cannot start, the screen shows a short microphone error message instead of failing silently.
• Leaving Practice now releases the microphone, and the new behavior is covered by tests for both calibrated and uncalibrated states.

Impact

✅ Voice input works in Practice
✅ Fewer incorrect attempts from uncalibrated users
✅ Clearer microphone setup feedback

🔄 Retrigger CodeAnt AI Review

@codeant-ai ask: Your question here

@codeant-ai ask: Can you suggest a safer alternative to storing this secret?

@codeant-ai: Your feedback here

@codeant-ai: Do not flag unused imports.

@codeant-ai: review

PR Code Suggestions ✨

+  // Serialize concurrent ensureCapture() calls. Without this, rapid
+  // press/release cycles before the first getUserMedia resolves can
+  // start multiple AudioCapture instances — the second overwrites
+  // audioCapture and orphans the first with an open mic that cleanup
+  // can't reach. captureStart pins the in-flight promise so subsequent
+  // callers await the same result instead of kicking off another start.
+  let captureStart: Promise<void> | null = null;
+  // destroyed lets a pending start know the component has unmounted. If
+  // start() resolves after cleanup ran, we stop the capture immediately
+  // rather than assigning it to the now-unmounted component — otherwise
+  // the mic stays live forever because cleanup saw audioCapture=null.
+  let destroyed = false;
+
   function onPcmFrame(pcm: Float32Array): void {
     if (pttActive) pcmBuffer.push(pcm);
   }
 
-  async function ensureCapture(): Promise<void> {
-    if (audioCapture) return;
+  function ensureCapture(): Promise<void> {
+    if (audioCapture) return Promise.resolve();
+    if (captureStart) return captureStart;
     const capture = new AudioCapture();
-    try {
-      await capture.start(onPcmFrame);
-      audioCapture = capture;
-      captureError = '';
-    } catch (err) {
-      // Permission denied, no mic, AudioContext construction failure, etc.
-      // Leave `audioCapture` null so the next PTT press will retry; surface
-      // a short message in the UI so the user knows why nothing's landing.
-      captureError = err instanceof Error ? err.message : String(err);
-      console.warn('[harmony-client] flashcard PTT: capture start failed:', err);
-    }
+    captureStart = (async () => {
+      try {
+        await capture.start(onPcmFrame);
+        if (destroyed) {
+          // Unmounted while start was pending — release the mic now.
+          // cleanup couldn't reach it because audioCapture was still null.
+          await capture.stop();
+          return;
+        }
+        audioCapture = capture;
+        captureError = '';
+      } catch (err) {
+        // Permission denied, no mic, AudioContext construction failure, etc.
+        // Leave `audioCapture` null so the next PTT press will retry; surface
+        // a short message in the UI so the user knows why nothing's landing.
+        if (destroyed) return;
+        captureError = err instanceof Error ? err.message : String(err);
+        console.warn('[harmony-client] flashcard PTT: capture start failed:', err);
+      } finally {
+        captureStart = null;
+      }
+    })();
+    return captureStart;
   }
 
   // Generate challenge when level changes (or on mount, since previousLevel starts null).
@@ -231,10 +257,14 @@
   }
 
   // Release the mic when this component unmounts (tab switch out of
-  // Practice, or navigation away from Spellbook). AudioCapture.stop()
-  // is safe to call on a never-started instance and internally handles
-  // already-stopped state.
+  // Practice, or navigation away from Spellbook). If a capture start is
+  // still in flight when we unmount, the `destroyed` flag tells the
+  // pending IIFE inside ensureCapture to stop the capture itself once
+  // start() resolves — cleanup here can only reach already-assigned
+  // captures, so the destroyed-flag path is the only way to guarantee
+  // no orphaned mic.
   $effect(() => () => {
+    destroyed = true;
     const c = audioCapture;
     audioCapture = null;
     pttActive = false;

 let audioCapture: AudioCapture | null = null;
 let pcmBuffer: Float32Array[] = [];
 let captureError = $state('');
+let captureStartPromise: Promise<void> | null = null;
 
 function onPcmFrame(pcm: Float32Array): void {
   if (pttActive) pcmBuffer.push(pcm);
 }
 
 async function ensureCapture(): Promise<void> {
   if (audioCapture) return;
-  const capture = new AudioCapture();
-  try {
-    await capture.start(onPcmFrame);
-    audioCapture = capture;
-    captureError = '';
-  } catch (err) {
-    // Permission denied, no mic, AudioContext construction failure, etc.
-    // Leave `audioCapture` null so the next PTT press will retry; surface
-    // a short message in the UI so the user knows why nothing's landing.
-    captureError = err instanceof Error ? err.message : String(err);
-    console.warn('[harmony-client] flashcard PTT: capture start failed:', err);
-  }
+  if (captureStartPromise) return captureStartPromise;
+
+  captureStartPromise = (async () => {
+    const capture = new AudioCapture();
+    try {
+      await capture.start(onPcmFrame);
+      audioCapture = capture;
+      captureError = '';
+    } catch (err) {
+      // Permission denied, no mic, AudioContext construction failure, etc.
+      // Leave `audioCapture` null so the next PTT press will retry; surface
+      // a short message in the UI so the user knows why nothing's landing.
+      captureError = err instanceof Error ? err.message : String(err);
+      console.warn('[harmony-client] flashcard PTT: capture start failed:', err);
+    } finally {
+      captureStartPromise = null;
+    }
+  })();
+
+  return captureStartPromise;
 }
 

1. Open the Harmony client and switch to Spellbook mode using the "Spellbook" button wired in `src/lib/components/NavPanel.svelte:185-186` (found via Grep), which sets the app into Spellbook mode so that `spellbookContent()` in `src/App.svelte:10-15,878-14` renders `<SpellbookMode {stq8Service}>` (confirmed in `src/App.svelte:868-18`).

2. In `src/lib/components/SpellbookMode.svelte:86-109`, select the "Practice" tab so the `{:else if activeTab === 'practice'}` branch renders `<FlashcardView {level} {expressMode} {stq8Service} initialStats={lastStats} onStatsUpdate={handleStatsUpdate} />` at lines 125-131, wiring the live `stq8Service` instance down into `FlashcardView`.

3. With the PR's updated `FlashcardView` implementation (the diff hunk around lines 44-70), ensure `stq8Service.isReady()` is true so the `{#if !stq8Service.isReady()}` guard allows the main Practice UI to render, and complete the calibration flow in `CalibrationView.svelte` (referenced from `SpellbookMode.svelte:133-137`) so that `stq8Service.isCalibrated()` returns true and the PTT button is enabled in the diff's `<PttButton active={pttActive} disabled={!stq8Service.isCalibrated()} ... />` block.

4. On the Practice tab, press and hold the PTT button rendered by `FlashcardView` (see the `<PttButton ... onPttStart={handlePttStart} onPttStop={handlePttStop} />` markup in the PR diff around lines 295-300). The mouse-down event triggers `onmousedown={() => activate('mouse')}` in `src/lib/components/PttButton.svelte:71-83`, which calls `activate('mouse')` (lines 27-32). Because `activeInputs` is initially empty, `activate` calls `onPttStart?.()`, invoking `handlePttStart()` in `FlashcardView` (PR diff around lines 93-101). In the PR version, `handlePttStart` calls `await ensureCapture()`, which enters the existing `ensureCapture()` snippet at lines 48-70: `audioCapture` is null, so it constructs `const capture = new AudioCapture();` and awaits `capture.start(onPcmFrame)`, which internally calls `navigator.mediaDevices.getUserMedia(...)` in `src/lib/voice/audio-capture.ts:21-36`.

5. Before the first `capture.start(onPcmFrame)` resolves (for example, while the browser microphone permission prompt is still showing or `getUserMedia` is slow), release the PTT and then quickly press it again. The release path runs `onmouseup={() => deactivate('mouse')}` in `PttButton.svelte:78-80`; `deactivate('mouse')` at lines 34-39 sees `activeInputs` go back to empty and calls `onPttStop?.()`, which triggers `handlePttStop()` in `FlashcardView` (PR diff around lines 104-131) while the original `ensureCapture()` call is still awaiting `capture.start(...)`.

6. While that first `ensureCapture()` is still pending and `audioCapture` has not yet been assigned, the rapid second press again calls `activate('mouse')` → `onPttStart?.()` → `handlePttStart()` → `ensureCapture()`. Because `audioCapture` is still null, the second `ensureCapture()` invocation also constructs `const capture = new AudioCapture();` and calls `await capture.start(onPcmFrame)`. When both `start()` calls eventually succeed, each `try` block assigns `audioCapture = capture` in turn, so the later-resolving `start()` overwrites the reference to the earlier `AudioCapture` instance. That earlier instance never has `stop()` called (the unmount `$effect` in the PR diff only calls `c.stop()` on the instance currently stored in `audioCapture`), meaning that for some period there are two active `AudioCapture` graphs and one of them is effectively leaked until garbage collection, contrary to the intended "one AudioCapture instance for the lifetime of this component" design.

This is a comment left during a code review.

**Path:** src/lib/components/FlashcardView.svelte
**Line:** 48:70
**Comment:**
	*Race Condition: `ensureCapture()` is vulnerable to concurrent calls: two rapid press cycles before the first `start()` resolves can create two separate `AudioCapture` instances, both opening the mic, with the later assignment overwriting the earlier reference and leaking resources. Serialize initialization with a shared in-flight promise so only one start attempt can run at a time.

Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix

CodeAnt AI finished reviewing your PR.

Addressed as `aae96eb`. Both findings had real correctness bugs under the framing — took them in order.

Finding 1: Mic start race + unmount leak

Verified the trace. `ensureCapture()`'s only guard was `if (audioCapture) return`, but `audioCapture` gets assigned after `await capture.start(...)` — which internally chains `getUserMedia` + `audioWorklet.addModule` + `createMediaStreamSource`. Two rapid press/release cycles before the first start resolves both pass the null guard, both call `capture.start()`, and the second assignment overwrites the first — orphaning a live AudioCapture with an open MediaStream and AudioContext that cleanup can't ever reach.

The unmount case is the same shape from the other side: `$effect` cleanup sees `audioCapture=null` (start hasn't resolved yet), does nothing. Then start() resolves on a dead component, assigns a live capture to a variable nobody holds.

Fix: serialize on a shared `captureStart: Promise | null`. First caller creates the promise and kicks off the IIFE; concurrent callers get handed the same promise to await instead of racing a second start. When the IIFE's `await capture.start()` resolves, it checks a `destroyed` flag (set by cleanup) — if true, it stops the capture itself and returns without assigning. The `captureStart = null` runs in `finally` so a failed start clears the slot and the next PTT press retries fresh.

`destroyed` and `captureStart` are plain lets, not `$state` — they're internal plumbing that doesn't drive any reactivity.

Finding 2: disabled blocks PTT stop

Verified the trace. `PttButton.deactivate()` opened with `if (disabled) return`, which ran before removing the input from `activeInputs` and calling `onPttStop`. If a parent flipped `disabled` true mid-hold — FlashcardView now does exactly this via `!stq8Service.isCalibrated()`, and voice-comms could do similar on permission revoke or rate-limit trip — the release would be silently swallowed, and the parent's `pttActive` would stick true forever because onPttStop is the only signal that ever clears it.

Same bug in `handleKeyUp` via the `|| disabled` in its early return — if Space was pressed while enabled and `disabled` flipped true before keyup, the release silently vanished.

Fix: dropped the `disabled` guards from `deactivate()` and `handleKeyUp()`. The existing `activeInputs.has(source)` check already rejects releases for inputs that were never activated, so no spurious stops fire. `activate()` keeps its `disabled` guard intact — rejecting new interactions on a disabled control is still correct; the asymmetry is the point. This is a shared component (used by voice-comms too via `CodecToggle`), so the fix pays dividends beyond Practice.

Added two regression tests covering the mid-hold transition (mouse + keyboard) so this can't silently regress later.

Verification

• 1125 tests pass (+4: 2 PttButton regression tests for mid-hold disabled, plus the 2 calibration-gate tests from the earlier commit)
• `svelte-check` clean on touched files
• `vite build` clean (main bundle unchanged materially, +0.08 KB gz)

Still awaiting manual end-to-end verification from @jenglund on the audio loop feel — the test plan in the PR description walks through it. The race fix won't be observable in normal use (would require rapid-fire press cycles in <300ms to trigger), but the disabled-mid-hold fix shores up an invariant for any future flow that might toggle the gate during a hold.

Addressed as `feafe90`. CodeRabbit caught something I'd genuinely missed — and my previous regression test was a false positive.

Finding 1: mouse release lost on disabled mid-hold (Critical)

Verified the claim. Modern browsers (Chrome M120+, Safari 17+, Firefox 124+) filter `mouseup`/`click` on disabled form controls per the HTML spec. My earlier `deactivate()` fix dropped the `disabled` guard so the handler body would run if reached — but on modern browsers the button-local `onmouseup` never fires on a disabled target, so the handler body wasn't reached in the real production scenario.

The reason the regression test passed anyway: jsdom doesn't replicate browser-level disabled-button event filtering. `fireEvent.mouseUp(btn)` on a disabled button in jsdom happily dispatches to the target's onmouseup. So the test was exercising jsdom's permissive event model, not Chrome's actual behavior. Test green, production bug still there.

Fix: added a window-level `onmouseup` handler that calls `deactivate('mouse')`, mirroring the existing keyboard pattern. Window listeners bypass the disabled-target filter since they're bound to `window`, not the button. The `activeInputs.has('mouse')` check in `deactivate()` keeps it a no-op when the user wasn't holding PTT (releasing mouse elsewhere on the page).

Declined `setPointerCapture` as an alternative — it'd be a bigger refactor from mouse events to pointer events in a component shared with voice-comms, and the minimal window-mouseup fix closes the same gap.

Also added a genuine real-browser-simulation test that dispatches `mouseup` only on window (not on the button), exercising the disabled-filter workaround directly. That test would have caught the original bug.

Finding 2: touch regression coverage (Nitpick)

Touch events aren't subject to the same disabled-button filtering (they aren't click-chain), so the existing `deactivate()` fix already worked for touch — no production fix needed. But the test coverage was mouse+keyboard only, and the shared `deactivate()` path deserves explicit touch coverage too. Added `touchend` and `touchcancel` mid-hold regression tests.

Verification

• 1128 tests pass (+3: touchend-disabled, touchcancel-disabled, window-mouseup-disabled)
• `svelte-check` clean on touched files
• `vite build` clean (+0.04 KB gz for the new handler)

Three inputs (mouse, keyboard, touch), one consistent rule now: release listeners live on `window` so they can't be intercepted by the target. Touch is the exception where target-local listeners still work, but putting all three through window-level paths (keyboard + mouse already there) would be the cleaner long-term direction if we ever refactor to pointer events for voice-comms.

Good catch — fixed in 5c94cf0.

CodeRabbit is right: with the window-level onmouseup handler from the previous round catching release anywhere in the document, onmouseleave stopped being a safety net and became a UX bug. Dragging off the button should never commit — only a real mouse release should. Removed the handler and added a regression test that mousedowns, mouseleaves, asserts PTT is still held, then mouseups on window to confirm the actual release path works.

The broader principle worth noting for the next review round: when you add a broader safety net (global window listener), the local safety net it displaces often flips from defense to bug. Worth auditing narrower handlers whenever a global one goes in.

CodeAnt AI is running the review.

User description

Summary

• Wires AudioCapture → WasmPipeline.process() into FlashcardView's PTT handlers. Holding the button captures PCM, releasing flushes it through the classifier, and the resulting syllable nibbles feed the existing handleRowComplete() evaluation path. This is the ZEB-152 payoff slice: Calibrate-then-Practice now works end-to-end for the first time.
• Lazy-async capture start on first press, one AudioCapture instance reused across holds for the component's lifetime, $effect cleanup on unmount releases the mic when you leave the Practice tab.
• PTT disabled with a "Calibrate your voice..." hint when !stq8Service.isCalibrated(), so an uncalibrated classifier can't emit garbage nibbles at the row-progression logic. SpellbookMode already remounts FlashcardView on tab switch, so the gate re-evaluates after a fresh calibration.

Design decisions worth flagging

PTT release as the row-attempt commit point. If PCM was captured and processPcm returned non-empty syllables, handleRowComplete(nibbles) runs and its existing pass/fail/combo logic takes over untouched. Empty buffer, empty syllables, or a thrown processPcm all fall through to the pre-Slice-3 "cancel row, break combo" path — one fallback covers "didn't hold long enough," "mic permission still resolving," "classifier heard nothing," and "WASM error" consistently instead of four bespoke paths.

Lazy capture vs prefetch on mount. Considered starting capture when the Practice tab mounts, but that keeps the mic indicator on even when the user is just reading cards. Lazy-on-first-press means first hold may catch less PCM than expected (getUserMedia latency), but the fallback path handles that cleanly and subsequent holds are tight.

Prop type widened to Stq8ServiceLike. FlashcardView previously had an inline 3-method shape for stq8Service; with Slice 3 needing isCalibrated + processPcm the inline shape became duplication. Switched to the exported interface to keep the contract in one place.

Out of scope (Slice 4 polish)

• Mismatch display from flashcard-design.md:70-78 (expected vs heard with caret under first-differing byte)
• 2-second momentum timeout during held PTT (spec: progress resets if no syllable advances for 2s)
• "Couldn't hear that clearly" UX for the empty-syllable release case (currently just the silent cancel path)
• Permission-denial recovery flow (currently surfaces error message text below the button)

Verification

• 1121 tests pass (+2 new calibration-gate UI tests in FlashcardView.test.ts)
• svelte-check clean on all touched files (two pre-existing initial-value warnings on isCalibrated/stq8Service unchanged)
• vite build clean — main bundle +2.8 KB gzipped (capture wiring)
• Manual end-to-end verification pending: calibrate on Ildwyn, switch to Practice, hold PTT, speak a Novice-level byte, confirm row-progression feels right

Test plan

• Pull, npm run dev, calibrate once on the Calibrate tab
• Switch to Practice tab, verify PTT button is enabled (no "Calibrate first" hint)
• Hold PTT, say a single Q8 syllable matching the active row, release → row should evaluate (green on match, red flash + reset on mismatch)
• Hold PTT without speaking, release → row cancels, combo breaks (pre-Slice-3 behavior preserved)
• Switch tabs mid-session, verify mic indicator clears (capture stops on unmount)
• Recalibrate mid-session, return to Practice, verify the new profile is in effect

Related: ZEB-152, closes part of ZEB-152 (Slice 1 already landed as #47; Slice 4 is follow-up polish).

🤖 Generated with Claude Code

Summary by CodeRabbit

• New Features

  • Push-to-talk voice input for flashcards with lazy mic capture and audio classification.

• Improvements

  • PTT disabled until microphone calibration; shows calibration or error hints and updated layout/hint styling.
  • Improved error handling, serialized capture start to avoid conflicts, ensures active capture is stopped on exit.
  • Service prop contract standardized for clearer integration.
  • PTT button now reliably unwinds/releases if disabled mid-hold so stops fire once.

• Tests

  • Added tests covering calibration UI and PTT behavior when disabled flips mid-hold.

CodeAnt-AI Description

Add calibrated push-to-talk voice practice with mic feedback

What Changed

• Practice mode now requires voice calibration before push-to-talk can be used, and shows a prompt to calibrate first when it is not ready.
• Holding push-to-talk now captures speech and uses the heard syllables to complete the practice row; releasing with no usable audio still cancels the row and breaks the combo as before.
• If microphone access fails, the screen shows a short microphone error message so users know why voice input is not working.
• Releasing push-to-talk now always ends the hold, even if the control becomes disabled during the press, preventing the button from getting stuck.

Impact

✅ Fewer wrong practice attempts from uncalibrated voice input
✅ Clearer guidance before starting voice practice
✅ Fewer stuck push-to-talk sessions after calibration or permission changes

🔄 Retrigger CodeAnt AI Review

@codeant-ai ask: Your question here

@codeant-ai ask: Can you suggest a safer alternative to storing this secret?

@codeant-ai: Your feedback here

@codeant-ai: Do not flag unused imports.

@codeant-ai: review

Sequence Diagram

This PR wires the Practice flashcard view's push to talk button into live microphone capture and the STQ8 classifier, gated by calibration, so releasing the button commits a spoken row attempt into the existing evaluation and progression logic.

sequenceDiagram
    participant User
    participant PracticeView
    participant AudioCapture
    participant Stq8Service

    User->>PracticeView: Open Practice tab
    PracticeView->>Stq8Service: Check calibrated status
    Stq8Service-->>PracticeView: Return calibrated state and enable or disable PTT

    User->>PracticeView: Hold push to talk
    PracticeView->>AudioCapture: Start capture and buffer PCM while held
    AudioCapture-->>PracticeView: Deliver PCM frames
    User->>PracticeView: Release push to talk
    PracticeView->>Stq8Service: Flush buffered PCM for classification
    Stq8Service-->>PracticeView: Return syllable nibbles
    PracticeView->>PracticeView: Evaluate row, update stats, advance card

Generated by CodeAnt AI

CodeAnt AI finished running the review.

CodeAnt AI is running the review.

User description

Summary

• Wires AudioCapture → WasmPipeline.process() into FlashcardView's PTT handlers. Holding the button captures PCM, releasing flushes it through the classifier, and the resulting syllable nibbles feed the existing handleRowComplete() evaluation path. This is the ZEB-152 payoff slice: Calibrate-then-Practice now works end-to-end for the first time.
• Lazy-async capture start on first press, one AudioCapture instance reused across holds for the component's lifetime, $effect cleanup on unmount releases the mic when you leave the Practice tab.
• PTT disabled with a "Calibrate your voice..." hint when !stq8Service.isCalibrated(), so an uncalibrated classifier can't emit garbage nibbles at the row-progression logic. SpellbookMode already remounts FlashcardView on tab switch, so the gate re-evaluates after a fresh calibration.

Design decisions worth flagging

PTT release as the row-attempt commit point. If PCM was captured and processPcm returned non-empty syllables, handleRowComplete(nibbles) runs and its existing pass/fail/combo logic takes over untouched. Empty buffer, empty syllables, or a thrown processPcm all fall through to the pre-Slice-3 "cancel row, break combo" path — one fallback covers "didn't hold long enough," "mic permission still resolving," "classifier heard nothing," and "WASM error" consistently instead of four bespoke paths.

Lazy capture vs prefetch on mount. Considered starting capture when the Practice tab mounts, but that keeps the mic indicator on even when the user is just reading cards. Lazy-on-first-press means first hold may catch less PCM than expected (getUserMedia latency), but the fallback path handles that cleanly and subsequent holds are tight.

Prop type widened to Stq8ServiceLike. FlashcardView previously had an inline 3-method shape for stq8Service; with Slice 3 needing isCalibrated + processPcm the inline shape became duplication. Switched to the exported interface to keep the contract in one place.

Out of scope (Slice 4 polish)

• Mismatch display from flashcard-design.md:70-78 (expected vs heard with caret under first-differing byte)
• 2-second momentum timeout during held PTT (spec: progress resets if no syllable advances for 2s)
• "Couldn't hear that clearly" UX for the empty-syllable release case (currently just the silent cancel path)
• Permission-denial recovery flow (currently surfaces error message text below the button)

Verification

• 1121 tests pass (+2 new calibration-gate UI tests in FlashcardView.test.ts)
• svelte-check clean on all touched files (two pre-existing initial-value warnings on isCalibrated/stq8Service unchanged)
• vite build clean — main bundle +2.8 KB gzipped (capture wiring)
• Manual end-to-end verification pending: calibrate on Ildwyn, switch to Practice, hold PTT, speak a Novice-level byte, confirm row-progression feels right

Test plan

• Pull, npm run dev, calibrate once on the Calibrate tab
• Switch to Practice tab, verify PTT button is enabled (no "Calibrate first" hint)
• Hold PTT, say a single Q8 syllable matching the active row, release → row should evaluate (green on match, red flash + reset on mismatch)
• Hold PTT without speaking, release → row cancels, combo breaks (pre-Slice-3 behavior preserved)
• Switch tabs mid-session, verify mic indicator clears (capture stops on unmount)
• Recalibrate mid-session, return to Practice, verify the new profile is in effect

Related: ZEB-152, closes part of ZEB-152 (Slice 1 already landed as #47; Slice 4 is follow-up polish).

🤖 Generated with Claude Code

Summary by CodeRabbit

• New Features

  • Push-to-talk voice input for flashcards with lazy mic capture and audio classification.

• Improvements

  • PTT disabled until microphone calibration; shows calibration or error hints and updated layout/hint styling.
  • Improved error handling, serialized capture start to avoid conflicts, ensures active capture is stopped on exit.
  • Service prop contract standardized for clearer integration.
  • PTT button now reliably unwinds/releases if disabled mid-hold so stops fire once.

• Tests

  • Added tests covering calibration UI and PTT behavior when disabled flips mid-hold.

CodeAnt-AI Description

Require voice calibration before Practice PTT, and keep push-to-talk release working even if the control becomes disabled mid-hold

What Changed

• Practice now blocks push-to-talk until voice calibration is complete and shows a clear prompt to calibrate first.
• Releasing push-to-talk now still ends the hold even if calibration is lost or another disable state appears while the button is pressed, preventing the control from getting stuck active.
• Cursor drift off the button no longer cancels a hold early; the release is only committed when the press actually ends.
• Added coverage for mouse, keyboard, and touch releases, plus the calibration gate in Practice.

Impact

✅ Fewer stuck push-to-talk states
✅ Clearer Practice setup guidance
✅ More reliable voice input on desktop and mobile

🔄 Retrigger CodeAnt AI Review

@codeant-ai ask: Your question here

@codeant-ai ask: Can you suggest a safer alternative to storing this secret?

@codeant-ai: Your feedback here

@codeant-ai: Do not flag unused imports.

@codeant-ai: review

Sequence Diagram

This PR wires the Practice push-to-talk button through a shared AudioCapture instance and the STQ8 classifier so that releasing PTT commits a row attempt based on recognized syllable nibbles, or falls back to canceling the row and breaking the combo when nothing usable is heard. PTT is gated by classifier calibration, but the diagram focuses on the main calibrated success and fallback paths.

sequenceDiagram
    participant User
    participant PttButton
    participant FlashcardView
    participant AudioCapture
    participant Stq8Service

    User->>PttButton: Hold push to talk
    PttButton->>FlashcardView: onPttStart
    FlashcardView->>AudioCapture: Ensure capture and start streaming
    AudioCapture-->>FlashcardView: Buffer PCM frames while held
    User->>PttButton: Release push to talk
    PttButton->>FlashcardView: onPttStop via window mouse and key handlers
    FlashcardView->>Stq8Service: Flush buffered PCM and call processPcm

    alt Syllables detected
        FlashcardView->>FlashcardView: Run handleRowComplete and advance card
    else No syllables or error
        FlashcardView->>FlashcardView: Cancel current row and reset combo
    end

Generated by CodeAnt AI

PR Code Suggestions ✨

1. Open Practice view in the Harmony client so `FlashcardView.svelte` is mounted and renders `<PttButton>` (see `src/lib/components/FlashcardView.svelte:25-31` where `<PttButton>` is wired with `onPttStart={handlePttStart}` and `onPttStop={handlePttStop}`).

2. Press and hold the primary (left) mouse button on the PTT button, which triggers `onmousedown={() => activate('mouse')}` in `src/lib/components/PttButton.svelte:96-97`. This adds `'mouse'` to `activeInputs` and calls `onPttStart`, which in `FlashcardView.handlePttStart` at `FlashcardView.svelte:119-127` sets `pttActive = true` and starts audio capture.

3. While still holding the primary button down on the PTT control, click and then release a secondary or middle mouse button anywhere on the page (for example, middle-click to scroll), causing a `mouseup` event on `window`. The `<svelte:window onmouseup={handleMouseUp} />` binding at `PttButton.svelte:87` invokes `handleMouseUp` at `PttButton.svelte:76-83`, which currently deactivates the `'mouse'` source for every mouseup without checking which button changed.

4. `deactivate('mouse')` at `PttButton.svelte:34-44` removes `'mouse'` from `activeInputs` and, because the set becomes empty, calls `onPttStop`, which runs `FlashcardView.handlePttStop` at `FlashcardView.svelte:12-39`. This ends the PTT segment and commits or cancels the row attempt even though the primary button is still physically held, prematurely terminating voice capture and row evaluation.

This is a comment left during a code review.

**Path:** src/lib/components/PttButton.svelte
**Line:** 76:83
**Comment:**
	*Logic Error: The window-level mouseup handler deactivates PTT for every mouse button release, not just the primary button that started the hold. Releasing a secondary/middle button anywhere on the page can therefore terminate an active hold early and trigger an unintended commit/cancel. Accept the mouse event and ignore non-primary releases when unwinding the `mouse` source.

Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix

CodeAnt AI finished running the review.

CodeAnt AI is running the review.

Sequence Diagram

This PR wires the Practice flashcard PTT button into audio capture and the STQ8 classifier, gating use on calibration and driving existing row evaluation from recognized syllables.

sequenceDiagram
    participant User
    participant FlashcardView
    participant PttButton
    participant AudioCapture
    participant Stq8Service

    User->>FlashcardView: View practice card
    FlashcardView->>Stq8Service: Read calibration state
    FlashcardView-->>User: Configure PTT enabled or disabled with hint
    User->>PttButton: Hold and release push to talk
    PttButton->>FlashcardView: Notify PTT start and stop
    FlashcardView->>AudioCapture: Capture PCM during hold
    FlashcardView->>Stq8Service: Send buffered PCM for processing
    Stq8Service-->>FlashcardView: Return syllable nibbles and match info
    FlashcardView->>FlashcardView: Evaluate row and update stats
    FlashcardView-->>User: Show updated grid and streak progress

Generated by CodeAnt AI

PR Code Suggestions ✨

1. Run the harmony-client UI and navigate to the Spellbook Practice tab, which (per `docs/plans/2026-03-11-flashcard-ui-plan.md:7`) renders `FlashcardView.svelte` for voice-driven flashcards. Ensure calibration is completed so `stq8Service.isCalibrated()` is true and the PTT button is enabled (`FlashcardView.svelte:325–333`).

2. Start a practice attempt by holding the PTT button rendered by `PttButton.svelte` (`src/lib/components/PttButton.svelte:89–103`). Holding the button triggers `activate('mouse' | 'touch' | 'keyboard')` (`PttButton.svelte:27–44`), which calls the `onPttStart` prop and thereby `handlePttStart()` in `FlashcardView.svelte` (`lines 119–128`), setting `pttActive = true` and starting to buffer audio frames in `onPcmFrame()` (`FlashcardView.svelte:65–67`).

3. While still holding PTT, speak a syllable or short phrase. The `AudioCapture` instance (`src/lib/voice/audio-capture.ts`) is active after `ensureCapture()` resolves (`FlashcardView.svelte:69–96`), and its `AudioWorkletNode.port.onmessage` handler (`audio-capture.ts:50–52`) asynchronously calls `onFrame(pcm)`, which is wired to `onPcmFrame(pcm)` in `FlashcardView.svelte`. As long as `pttActive` is true, `onPcmFrame` pushes each `Float32Array` into `pcmBuffer` (`FlashcardView.svelte:65–67`).

4. Release the PTT button. This causes `deactivate(...)` in `PttButton.svelte` to call the `onPttStop` prop once all active inputs are released (`PttButton.svelte:34–45`), which invokes `handlePttStop()` in `FlashcardView.svelte` (`lines 130–157`). Inside `handlePttStop`, `pttActive` is immediately set to `false` (`line 131`), then `flushAndClassify()` is called (`line 138`), which copies and clears `pcmBuffer` before passing it to `stq8Service.processPcm` (`lines 159–176). Because `AudioWorkletNode.port.onmessage` delivery is asynchronous, any audio frames that were already produced by the worklet but whose `onmessage` events fire after `pttActive` was set to `false` (and after `flushAndClassify` ran) will be dropped by `onPcmFrame` (the `if (pttActive)` guard at `line 65`). In practice you can confirm this by instrumenting `onPcmFrame` and `handlePttStop` with logging: you will observe `onPcmFrame` events occurring after `handlePttStop` completes where `pttActive` is already `false`, and those frames never make it into the `pcmBuffer` that was just flushed. This systematically truncates the tail of each utterance by up to a frame (or more, depending on scheduling), meaning the classifier at `stq8Service.processPcm` sometimes runs on incomplete audio and can yield fewer or mismatched `syllables`, causing occasional false negatives in row evaluation via `handleRowComplete()` (`FlashcardView.svelte:182–219`).

This is a comment left during a code review.

**Path:** src/lib/components/FlashcardView.svelte
**Line:** 130:138
**Comment:**
	*Race Condition: `handlePttStop` flushes and classifies the buffer immediately after release, but AudioWorklet `postMessage` delivery is asynchronous, so frames captured right before release can arrive just after this flush and get dropped. This truncates utterances at the tail and causes false negatives/mismatches. Delay flush until pending frame messages are drained (or add an explicit end-of-utterance/frame-boundary mechanism) before calling `processPcm`.

Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix

CodeAnt AI finished running the review.

CodeAnt AI is running the review.

Sequence Diagram

This PR wires the Practice push to talk button through shared audio capture and the STQ8 classifier so that releasing PTT commits a spoken row attempt, while uncalibrated or empty captures fall back to the existing cancel and combo reset behavior.

sequenceDiagram
    participant User
    participant FlashcardView
    participant PttButton
    participant AudioCapture
    participant Stq8Service

    User->>FlashcardView: Open Practice card
    FlashcardView->>Stq8Service: Check calibration and prepare challenge

    User->>PttButton: Press and hold push to talk
    PttButton-->>FlashcardView: onPttStart
    FlashcardView->>AudioCapture: Start capture and buffer PCM while held

    User->>PttButton: Release push to talk
    PttButton-->>FlashcardView: onPttStop
    FlashcardView->>Stq8Service: Process buffered PCM to syllable nibbles

    alt Nibbles detected
        FlashcardView->>FlashcardView: Evaluate row, update stats, load next card
    else No audio or error
        FlashcardView->>FlashcardView: Cancel row and reset combo
    end

Generated by CodeAnt AI

PR Code Suggestions ✨

1. Render the main app (`src/App.svelte:878-881`), which mounts `SpellbookMode` (`src/lib/components/SpellbookMode.svelte:48-83`) and, on the Practice tab, `FlashcardView` (`SpellbookMode.svelte:25-32`) with a `stq8Service` whose `isCalibrated()` initially returns true.

2. Within a test or dev harness, use a `Stq8ServiceLike` stub where `isCalibrated()` reads from an internal flag and `processPcm()` returns some non-empty syllable list; start a PTT hold by dispatching `mousedown` on the `PttButton` in `FlashcardView` (`FlashcardView.svelte:325-330`), which calls `PttButton`'s `activate('mouse')` (`PttButton.svelte:27-32`) and then `handlePttStart()` in `FlashcardView` (`FlashcardView.svelte:119-128`), setting `pttActive=true` and buffering PCM via `onPcmFrame()` (`FlashcardView.svelte:65-67`).

3. While the hold is still active (mouse button or spacebar still down), flip the stub's internal calibrated flag to false so that subsequent calls to `stq8Service.isCalibrated()` return false; Svelte reactivity then re-renders `FlashcardView` so that `PttButton` receives `disabled={!stq8Service.isCalibrated()}` as true (`FlashcardView.svelte:325-327`) and the "Calibrate your voice…" hint appears (`FlashcardView.svelte:331-333`), but the hold remains active because `PttButton`'s `deactivate()` is intentionally not guarded by `disabled` (`PttButton.svelte:34-45`).

4. Release the PTT input (e.g., dispatch `mouseup` on window), which triggers `PttButton`'s `handleMouseUp()` and `deactivate('mouse')` (`PttButton.svelte:76-85`), causing `onPttStop` to fire and call `handlePttStop()` in `FlashcardView` (`FlashcardView.svelte:130-157); inside `handlePttStop`, the code unconditionally invokes `flushAndClassify()` (`FlashcardView.svelte:138-139`), which calls `stq8Service.processPcm(pcm)` (`FlashcardView.svelte:173-175`) and, if any nibbles are returned, feeds them into `handleRowComplete(nibbles)` (`FlashcardView.svelte:139-145, 182-219`) even though `stq8Service.isCalibrated()` is now false and the UI has disabled PTT, demonstrating that uncalibrated/stale classification still drives row progression.

This is a comment left during a code review.

**Path:** src/lib/components/FlashcardView.svelte
**Line:** 138:145
**Comment:**
	*Logic Error: `handlePttStop` always runs classification even if calibration became invalid during an active hold (a case this PR explicitly supports by allowing release while `disabled` flips). That can feed uncalibrated/stale output into row progression and incorrectly mark attempts as pass/fail. Re-check calibration at release and skip `flushAndClassify()` when not calibrated so release falls through the cancel-row path.

Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix

CodeAnt AI finished running the review.