[Epic] Analytics, conversation intelligence, evaluations, and experiments #80

Parent roadmap: #76

Problem

QuickVoice has useful KPI, volume, performance, call-detail, transcript, extracted-data, and evaluation surfaces, while several open PRs propose Langfuse tracing. It does not yet provide one stable metric model, end-to-end operational and business reporting, searchable conversation intelligence, systematic regression evaluation, or version-aware production experimentation.

Outcome

Operators can understand what happened and why; builders can convert failures into repeatable tests; reviewers can combine automated and human quality signals; and owners can promote a better agent version using statistically and operationally defensible evidence.

Required capabilities

• Canonical organization-scoped events for calls, sessions, workflow nodes, turns, model/provider spans, tools, transfers, campaign actions, evaluations, and business outcomes.
• Clear metric definitions and freshness, timezone, currency, filter, and aggregation semantics.
• Dashboards for outcome, funnel, cohort, cost, latency, reliability, workflow paths, campaigns, tools, providers, models, and agents.
• Keyword and semantic transcript search, topics, summaries, sentiment, intent, objections, customer timeline, and saved views.
• Scenario datasets, tests generated from calls, next-response/tool/full-conversation assertions, controlled mocks, and simulated users.
• Repeated probabilistic runs, failure clustering, flakiness, baseline comparison, human QA queues, reviewer calibration, and release gates.
• Immutable configuration versions, branches, deterministic traffic allocation, canaries, promotion, rollback, and experiment analysis.

Child issues

• [Analytics] Define the canonical event, metric, cost, latency, quality, and outcome model
• [Analytics] Build advanced dashboards, funnels, cohorts, breakdowns, and custom reports
• [Intelligence] Add transcript search, topic discovery, sentiment, summaries, and customer timelines
• [Evals] Add datasets, test-from-call, assertions, tool mocks, and multi-turn user simulation
• [Evals] Add probabilistic batch runs, human QA, calibration, regression gates, and CI execution
• [Experiments] Add agent branches, traffic splits, canary releases, promotion, and rollback

Dependencies

• Workflow/version contracts.
• Customer/campaign identity model.
• Public API/event and privacy contracts.
• Consolidated observability implementation from the overlapping Langfuse PRs.

Acceptance criteria

• A documented metric produces the same result in dashboard, API, export, and experiment analysis for the same filters.
• Every trace and stored artifact is organization scoped and applies zero-PII/retention policy before export to optional observability providers.
• Users can drill from an aggregate anomaly to relevant calls, spans, workflow nodes, transcripts, and evaluation evidence where policy permits.
• A production failure can become a sanitized regression test without copying protected data by default.
• Release gates can block promotion but require an authorized, reasoned, audited override path.
• Experiment assignment is stable, mutually exclusive, observable, and reversible.
• Metric correctness, authorization, timezone/currency, redaction, evaluation determinism boundaries, statistical summaries, and rollback are tested.

Boundary

Automated sentiment, summaries, and LLM judgments are assistive signals with confidence and provenance, not unquestionable facts.