⚡ Clutch.ai

• Core Capabilities
• System Architecture
• Getting Started
• Audio Configuration
• Data & Model Preparation
• Execution
• Overlay Modes
• Fallback Mechanisms
• Evaluation
• Contributing
• License

Clutch.ai operates through a highly optimized, low-latency online pipeline:

1. Audio Capture: Streams audio from a selected input device (microphone or loopback).
2. Voice Activity Detection: Detects speech segments using RMS VAD (with an optional GRU VAD model).
3. Transcription: Converts speech to text using Faster-Whisper (base.en, CPU int8).
4. Intent Classification: Classifies the transcript as technical_question, personal_behavioral, or noise.
5. Context Retrieval: Retrieves relevant supporting context from ChromaDB (with an optional reranker).
6. Insight Generation: Generates concise, interview-ready guidance via Groq (default 70B) with local Ollama fallback.
7. Live Overlay: Streams the generated output directly to a floating, unobtrusive PyQt5 overlay.

• pipeline.py — Application entry point and orchestrator
• audio/capture.py — Audio capture, VAD processing, and transcription
• audio/devices.py — Input device resolution (--list-devices support)
• classifier/predict.py — BiLSTM/MLP classifier inference module
• rag/retriever.py — ChromaDB retrieval engine with optional reranking hooks
• llm/hint_gen.py — Prompt engineering and LLM generation (Groq/Ollama)
• ui/overlay.py — Stealth and demonstration overlay implementations

git clone https://github.com/AhsanRiaz786/clutch-ai.git
cd clutch-ai
pip install -r requirements.txt

Duplicate the example environment file:

cp .env.example .env

Update .env with your specific configuration values:

• GROQ_API_KEY — Required for Groq LLM generation
• LLM_MODEL — Defaults to llama-3.3-70b-versatile
• CLUTCH_INPUT_DEVICE — Optional audio input selector (e.g., blackhole)
• OVERLAY_DEMO_MODE — Toggle overlay visibility
  • 0 = Stealth Mode (Hidden from screen capture on macOS)
  • 1 = Demo Mode (Visible in screen capture)
• MIN_CLASSIFIER_CONFIDENCE — Default 65

Leave CLUTCH_INPUT_DEVICE unset, or set it to default or mic.

List available audio devices:

python audio/capture.py --list-devices

Then, set CLUTCH_INPUT_DEVICE to the matching name fragment or index.

1. Install BlackHole 2ch.
2. Open Audio MIDI Setup and create a Multi-Output Device comprising:
   • BlackHole 2ch
   • Your primary speakers/headphones
3. In System Settings → Sound → Output, select the newly created Multi-Output device.
4. Update your .env file:
   • CLUTCH_INPUT_DEVICE=blackhole

Utilize Stereo Mix (or equivalent loopback input), and configure CLUTCH_INPUT_DEVICE accordingly.

Incorporate your personal context documents into the vector database:

• data/notes/ — General notes and documentation
• data/code/ — Code snippets and files
• data/resume/ — Resume and experiential context

Construct the vector database and (optionally) retrain the classification models:

python ingest/ingest.py
python classifier/train.py
python classifier/lstm_classifier.py
python classifier/finetune_embeddings.py

Launch the Clutch.ai pipeline:

python pipeline.py

Expected startup sequence logs:

• [UI] Overlay mode: DEMO (capture-visible) or STEALTH (capture-hidden)
• [PIPELINE] Prerequisites OK
• [AUDIO] VAD capture ready — listening for speech ...

• Demonstration Mode: Set OVERLAY_DEMO_MODE=1. Ideal for showcases. Share your entire screen if the meeting application excludes floating overlays during window-only sharing.
• Privacy / Stealth Mode: Set OVERLAY_DEMO_MODE=0. Designed for live scenarios (macOS natively applies NSWindowSharingNone to hide the overlay).

• If the Groq API key is missing or invalid, the application automatically falls back to a local Ollama instance.
• If Ollama is not running, the system will return a safe, generic fallback response to ensure uninterrupted operation.

To initialize the local Ollama fallback:

ollama pull llama3.2:3b
ollama serve

Optional scripts are provided to evaluate the retrieval and latency performance:

python eval/eval_retrieval.py
python eval/eval_latency.py

Contributions, issues, and feature requests are welcome! Feel free to check out the issues page.

Please read our Contributing Guidelines and Code of Conduct before submitting a Pull Request.

This project is MIT licensed.