sequent

An A/B testing engine whose confidence intervals stay valid no matter how often you look at them, so a result can be acted on the moment it appears.

A fixed-horizon t-test assumes you look once, at a sample size committed to before the experiment started. Nobody does this. Dashboards refresh, stakeholders ask on Thursday, and the test gets stopped the first time it goes green.

That habit is not a small sin. Each look is another chance to cross the threshold by luck, and the chances compound. In this repository's own 1000-simulation A/A study — two identical variants, no real effect anywhere — a t-test peeked continuously fires at 32.7% against a nominal α of 5%. Roughly one in three "wins" from a peeked fixed-horizon test is nothing at all.

The usual answers are unsatisfying. Committing to a horizon and refusing to look is organisationally impossible. Bonferroni-style corrections over a handful of planned interim analyses work, but they require knowing the number of looks in advance and they waste power. Neither addresses the actual behaviour, which is continuous monitoring by people who will act on what they see.

The fix has to make continuous peeking correct, rather than forbidding it.

sequent computes an always-valid confidence sequence using the mixture sequential probability ratio test (mSPRT).

Under H₀ the mixture likelihood ratio is a non-negative martingale, so by Ville's inequality P(∃n : reject) ≤ α — the guarantee holds uniformly over time rather than at one pre-committed n. You may look as often as you like and stop the moment 0 leaves the interval. Same 1000-simulation A/A study, same continuous peeking: 2.6%, comfortably under α.

The interval half-width is

r = √( V(V+τ²)/τ² · (2·ln(1/α) + ln((V+τ²)/V)) )

with V the running variance of the difference in means and τ² the mixture variance — the prior scale of effects worth detecting. Sufficient statistics are accumulated with Welford's online algorithm, so a decision costs no scan over history. Full derivation in backend/app/stats/sequential.py; the study artifact is docs/aa_study.txt.

τ² encodes what size of effect you care about. Set it too small and the sequence is slow to reject a large true effect; too large and it never tightens on a small one. It is the one genuinely subjective input, so it lives per-experiment in the database rather than as a global constant.

CUPED adjusts each observation by a pre-assignment covariate:

Y_adj = Y − θ(X − E[X]),    θ = Cov(Y,X)/Var(X)

Because X is measured before assignment it cannot correlate with the treatment, so the adjustment removes variance without moving the expected effect. The reduction is approximately ρ². On the seeded checkout-redesign experiment the console reports −54.8% variance, which tightens the interval from [0.042, 0.211] to [0.073, 0.190] on identical data.

Variant assignment is a SHA-256 hash of (key:unit:salt). The rollout gate and the variant bucket are drawn independently, so widening a rollout from 30% to 50% admits new units without reshuffling the ones already enrolled. A chi-square goodness-of-fit test on observed versus intended split flags sample-ratio mismatch before anyone reads the result.

checkout-redesign with 3,002 control and 2,998 treatment units. The confidence sequence narrows as events accumulate and the decision boundary is crossed at n≈2,350 — a point reached during the experiment, not at a horizon fixed before it. SRM reads balanced at p=0.959.

The same experiment with variance reduction enabled: −54.8% variance, and the interval tightens from [0.042, 0.211] to [0.073, 0.190]. The point estimate barely moves, which is the property that makes the adjustment legitimate.

promo-banner lifts its primary metric while degrading latency by 0.266 in the increase_bad direction. The guardrail is flagged REGRESSION, so the primary win does not stand on its own.

null-test runs two identical variants. It reads inconclusive and stays there — the outcome a peeked fixed-horizon test fails to deliver roughly a third of the time.

clients / traffic simulator
   │  POST /api/assign    deterministic hash → variant (+ % rollout gate)
   │  POST /api/events    metric value (+ pre-period covariate for CUPED)
   ▼
FastAPI ── SQLAlchemy ──► Postgres  (experiments · assignments · events)
   │
   └─ metrics engine
        ├─ sequential.py   mSPRT confidence sequence
        ├─ cuped.py        pre-period covariate adjustment
        ├─ guardrails.py   same test on a protected metric
        ├─ srm.py          chi-square on the split
        └─ assignment.py   SHA-256 bucketing
   ▼
React console  ·  confidence sequence · CUPED toggle · guardrails · segments

The statistics live in pure functions under backend/app/stats/, taking arrays and returning intervals with no database or HTTP awareness. That boundary is what makes the A/A study possible: scripts/aa_experiment.py calls the same code path the API calls, 1000 times, with no server running.

• Python 3.11 or later
• Node.js 18 or later
• Docker (for Postgres) or any Postgres 16 reachable on port 5434

git clone https://github.com/varadharajanv0310/sequent.git
cd sequent
cp .env.example .env
docker compose up -d db
python -m venv backend/.venv
backend/.venv/Scripts/python -m pip install -r backend/requirements.txt   # Windows
# backend/.venv/bin/pip install -r backend/requirements.txt               # macOS / Linux
cd frontend && npm install && cd ..

make setup does the same on a machine with make.

cd backend && ../backend/.venv/Scripts/python -m app.seed      # 3 demo experiments
../backend/.venv/Scripts/python -m uvicorn app.api:app --port 8002
cd ../frontend && npm run dev                                  # console on :5175

Seeding creates checkout-redesign (a real effect), promo-banner (a real effect with a latency regression) and null-test (A/A), each with simulated traffic.

make test        # 13 tests
make verify      # end-to-end gate against a running API
make aa-study    # regenerates docs/aa_study.txt

curl "http://localhost:8002/api/experiments/checkout-redesign/results?metric=revenue&cuped=true"

{
  "experiment": "checkout-redesign",
  "metric": "revenue",
  "use_cuped": true,
  "pooled": {
    "n_control": 3002,
    "n_treatment": 2998,
    "effect": 0.13170462899376767,
    "ci_lower": 0.07299133594995998,
    "ci_upper": 0.19041792203757535,
    "significant": true,
    "decision": "significant",
    "cuped_variance_reduction": 0.5480921115783428
  }
}

sequent/
├── backend/app/stats/     # sequential (mSPRT), cuped, assignment, srm, guardrails
├── backend/app/           # api.py (FastAPI), models.py, seed.py, simulator.py
├── backend/scripts/       # aa_experiment.py (1000-sim study), verify.py (live gate)
├── backend/tests/         # A/A false-positive gate, power, CUPED, assignment, SRM
├── frontend/src/          # React console — App.tsx, SeqChart.tsx
└── docs/                  # aa_study.txt artifact, screenshots

All numbers below come from docs/aa_study.txt, regenerated by make aa-study — 1000 simulations per arm, continuous peeking, nominal α = 0.05.

• Normal approximation. The mSPRT here is built for a difference in means. Binary metrics are handled through the normal approximation rather than an exact Bernoulli mixture, which is poor at very low conversion rates and small n.
• No multiple-testing correction across metrics. Each metric is always-valid on its own; testing ten metrics on one experiment reintroduces a family-wise problem the platform does not currently account for.
• Segment slices are unshrunk. Small segments produce wide intervals and invite over-reading. There is no hierarchical shrinkage.
• Single-node. Assignment is stateless and would scale, but event ingest is a plain synchronous insert with no batching or queue.
• Not tested against production traffic. All validation is against the built-in simulator, whose event stream is better behaved than a real one.

• Exact Bernoulli mSPRT for proportion metrics, removing the normal approximation
• Hierarchical shrinkage on segment slices to control over-reading of small cells
• SRM diagnosis — attribute a mismatch to a specific assignment or logging stage
• Batched event ingest behind a queue, so a traffic spike cannot block the API
• A create-experiment form in the console; today experiments are created over the API

MIT