Fleet Ensemble

An agentic performance system where MIDI tracks become living instruments that coalesce under a director's feel.

Like fine paint on liquid white base paint — each color finds its edge in relation to the others, not by command, but by resonance.

Every MIDI track can be rendered as an agentic instrument — a player with its own perception, its own reflexes, its own voice. An agentic director performs the rendering synoptically, shaping the feel of the whole ensemble. The instruments align themselves to the director's feel the way a jazz quartet locks into a pocket — not by following a click track, but by listening.

This is the agentic compiler pattern applied to music. Instead of compiling source code to machine code, we compile a musical score to a live performance. Each instrument is an agent. The director is the orchestrator. The canvas is the output.

The result is not a playback. It is a happening — a performance that is different every time, because the agents respond to each other and to the director's feel in real time. Like Wittgenstein's language games, the meaning is in the playing, not in the score.

• Architecture
• The Canvas Metaphor
• The Director as Weather System
• The Agentic Compiler → Agentic Performer Analogy
• Relation to Fleet JEPA-MIDI
• The Director is a JEPA
• The Performer is Any Model
• The Instruments
• Design Documents
• Status
• License

┌─────────────────────────────────────────────────────────────┐
│                   AGENTIC DIRECTOR                           │
│   Hears the whole ensemble. Shapes the feel.                 │
│   "Lay back. Darker. More space. Push the bridge."           │
│   Operates on the ENSEMBLE level — not individual notes.     │
│                                                              │
│   ┌──────────────────────────────────────────────┐           │
│   │  TRI-CHAMBER ARCHITECTURE                    │           │
│   │  ┌─────────┐  ┌─────────┐  ┌─────────┐      │           │
│   │  │ ORACLE  │  │ MAESTRO │  │  PULSE  │      │           │
│   │  │ (LLM)   │  │ (Trained│  │ (Algo)  │      │           │
│   │  │ Phrase  │  │ Pulse   │  │ Sub-ms  │      │           │
│   │  │ level   │  │ level   │  │ level   │      │           │
│   │  └─────────┘  └─────────┘  └─────────┘      │           │
│   └──────────────────────────────────────────────┘           │
└──────────────────┬──────────────────────────────────────────┘
                   │
                   │ FEEL_TILT packets
                   │ (7 parameters + per-instrument offsets)
                   │  ρ  epsilon  σ  τ  γ  λ  Φ
                   ▼
┌──────────────────────────────────────────────────────────────┐
│              AGENTIC INSTRUMENTS (one per track)              │
│                                                              │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐  ┌──────────┐    │
│  │  Piano   │  │   Bass   │  │  Drums   │  │  Guitar  │    │
│  │  Agent    │  │  Agent   │  │  Agent   │  │  Agent   │    │
│  │           │  │           │  │           │  │           │    │
│  │ Listens   │  │ Listens   │  │ Listens   │  │ Listens   │    │
│  │ to others │  │ to others │  │ to others │  │ to others │    │
│  │ Aligns to │  │ Aligns to │  │ Aligns to │  │ Aligns to │    │
│  │ director  │  │ director  │  │ director  │  │ director  │    │
│  └─────┬─────┘  └─────┬─────┘  └─────┬─────┘  └─────┬─────┘    │
│        │              │              │              │          │
│        └──────────────┴──────┬───────┴──────────────┘          │
│                              │                                 │
│              CNS Protocol Bus                                 │
│              (embeddings, intents, drift, roles)              │
└──────────────────────────────┬─────────────────────────────────┘
                               │ MIDI events
                               ▼
                    ┌───────────────────┐
                    │   OUTPUT CANVAS    │
                    │                   │
                    │  The rendered      │
                    │  performance —     │
                    │  not a playback,   │
                    │  a happening       │
                    └───────────────────┘

The director operates across five nested timescales: arc (minutes to hours), section (15-60 seconds), phrase (2-10 seconds), pulse (~125ms), and emergence detection (event-driven). Each timescale has its own cognitive chamber, from the LLM Oracle that thinks in musical language to the algorithmic Pulse that runs physics at sub-millisecond resolution.

Liquid white base paint is the ground. Each instrument is a color poured onto the surface. The colors don't mix mechanically — they find their own edges through surface tension, viscosity, density. The director doesn't tell each color where to go. The director tilts the canvas.

• The base paint is the JEPA embedding space — the shared musical ground
• Each color is an instrument agent with its own timbre, register, and behavior
• The tilt is the director's feel parameters — the seven-dimensional feel space: pulse density, energy flux, harmonic tilt, temporal asymmetry, coupling pressure, risk appetite, articulation
• The painting is what emerges — unpredictable, alive, coalesced

The physics of paint on a tilted surface maps precisely to the math. The Marangoni effect — flow driven by surface tension gradients — is the harmonic rotation R_σ in the SDE. Brownian motion of particles in the paint is the stochastic exploration λ · dW. The drying time is the exponential smoothing constant that prevents stepped transitions. This is not a loose analogy — it is the same mathematics.

The director does not conduct with a baton. It does not cue entrances. It does not correct pitch. It is the air. Two hundred years of orchestral tradition rested on a fatal hubris: that you could produce coherent collective beauty by prescribing trajectory for every discrete actor. This director does not command molecules. It sets the atmospheric conditions under which music condenses.

This architecture derives from chaos theory, the original Lorenz attractor model of atmospheric convection, and the Navier-Stokes equations of fluid dynamics. Just as Navier-Stokes defines the rules of the medium — not the path of any single molecule — the director maintains only a global state vector of atmospheric feel parameters. No agent is told what to play. They feel the conditions of the space they play inside.

The feel parameters are atmospheric variables:

There is no master timeline. There is only a forecast. The director cannot tell you what will be played 17 bars from now. It can only tell you what the air will feel like then. You do not ask a storm to keep time. You stand inside it, and listen.

Full deep-dive: Director Design §The Director as Weather System and §The Director as Spacetime Curvature.

Critical insight: Traditional MIDI sequencers are ahead-of-time compilers — they produce the same binary every time. Fleet Ensemble agents are JIT compilers that recompile every millisecond while running. The "binary" (rendered performance) is different every time, because the optimization passes respond to live ensemble conditions.

Fleet Ensemble uses Fleet JEPA-MIDI's embedding space as its shared language. Each instrument agent has its own JEPA reader. The director operates on the ensemble-level embedding — the sum of all instruments' current states.

The director is not an LLM. The director is not a conductor with a baton. The director is a JEPA — a Joint Embedding Predictive Architecture (LeCun, 2022) that perceives the whole ensemble's feel and outputs the tilt.

The JEPA director:

• Perceives every instrument's current state via their embeddings
• Predicts where the ensemble is heading
• Outputs the tilt — feel parameters that shape how instruments render:
  • Tempo curve — micro-adjustments to pulse (living tempo, not static BPM)
  • Dynamic shape — intensity envelope across the ensemble
  • Color — bright/dark, dense/sparse, active/still
  • Weight — which instrument carries the moment
  • Space — how much silence between events
• Learns what tilts produce what emergent behaviors — trained on real ensemble performances

This is perception-to-direction, not perception-to-language-to-direction. The JEPA doesn't describe the feel in words. It perceives the feel in its latent space and outputs directorial parameters directly.

The performer is modular and pluggable. Any model that can take MIDI and render it as precise instructions for a music rendering system. Could be an LLM that thinks in phrasing. Could be a rules engine. Could be a trained transformer. The point: the performer renders MIDI with intelligence and musicality — not just notes on a page, but a real performance.

The performer receives:

• The MIDI score (what to play)
• The JEPA director's tilt (how to play it)
• Its own instrument's current state

The performer outputs:

• Rendered MIDI events with musicality — micro-timing, velocity shaping, phrasing, breath, space
• More than the score. The score made alive.

Each instrument agent has five modules and a shared internal clock running at 1 kHz:

1. A voice — its MIDI program, register, timbral preferences, polyphony limit
2. A JEPA reader — perceives the ensemble state at pulse rate (62.5 Hz)
3. A reflex engine — fast algorithmic responses (<10ms), like spinal reflexes
4. An alignment module — adjusts timing, dynamics, articulation, and note choice based on director feel + ensemble state
5. A listening module — hears other instruments via their embeddings + MIDI output

Instruments communicate through the CNS protocol bus — the same packet-based agent communication system the fleet uses. A MIDI note is a packet. A phrase is a pulse. A performance is a session.

Each instrument has a distinct personality fingerprint — a set of behavioral parameters that determines how it responds to the ensemble:

• The Piano — The Accompanist-Poet (alignment_gain: 0.25). Listens more than it speaks. Drops notes to make space. Think Herbie Hancock in Miles's Second Quintet, Bill Evans in his trio.
• The Bass — The Anchor (alignment_gain: 0.7). Steady, foundational, almost never drops a root. Think Ron Carter, Paul Chambers.
• The Drums — The Grid Incarnate (alignment_gain: 0.9). Defines the time. Everyone else adjusts to the drums. Think Tony Williams at 19 with Miles, Elvin Jones with Coltrane.

• Director Design — The agentic director's perception, feel space, tri-chamber architecture, emergence detection, mathematical formalism, and operational modes. Start here.
• Instrument Agent Design — The engineering spec for a single instrument agent: internal architecture, perception pipeline, communication protocol, alignment mechanics, concrete instrument designs, and training pipeline.

• Rust 1.75+ (2021 edition)
• fleet-gateway for LLM oracle calls
• A trained JEPA encoder from fleet-jepa-midi
• MIDI input/output via virtual MIDI port or file

git clone https://github.com/SuperInstance/fleet-ensemble.git
cd fleet-ensemble
cargo build --release

# Start the ensemble with a MIDI score
./target/release/fleet-ensemble --score path/to/score.mid --bpm 120

# With specific instruments
./target/release/fleet-ensemble \
  --score path/to/score.mid \
  --instruments piano,bass,drums \
  --bpm 120

The director communicates with instruments through a seven-dimensional feel space. Each parameter is a continuous value that the director adjusts in real-time:

These parameters are sent as FEEL_TILT packets on the CNS protocol bus. Each instrument receives the global tilt plus per-instrument offsets.

The director doesn't use a single brain. It uses three cognitive chambers, each operating at a different timescale:

1. Oracle (LLM) — Thinks in musical language ("build tension," "quote the bridge"). Called every 1-4 bars. This is where musical knowledge lives.

2. Maestro (Trained Model) — A trained neural network that perceives the ensemble's current state at pulse rate. This is where musical feel lives — the JEPA.

3. Pulse (Algorithmic) — Pure math running at sub-millisecond resolution. Physics-based models of timing, dynamics, and articulation. This is where precision lives.

Each instrument has an alignment engine that adjusts its playing based on the director's feel parameters and the ensemble's current state. The alignment gain determines how strongly an instrument follows the director vs. plays its own thing:

• Low alignment gain (0.25) — Independent, conversational. Like Herbie Hancock comping behind a soloist — drops notes, leaves space, responds.
• High alignment gain (0.9) — Tight, foundational. Like Tony Williams driving the band — defines the time, everyone else adjusts.

The alignment engine uses stochastic differential equations (SDEs) to smoothly interpolate between director-guided and self-directed behavior:

$$dX_t = \gamma (\mu_{director} - X_t),dt + \lambda,dW_t$$

where $\gamma$ is the coupling pressure, $\mu_{director}$ is the director's target, and $\lambda \cdot dW_t$ is Brownian motion exploration.

use fleet_ensemble::Director;

let mut director = Director::new()
    .jepa_encoder("weights/jepa_encoder.pt")?
    .gateway("http://127.0.0.1:8787/v1")
    .feel_space(FeelSpace::default());

// Run the director loop
director.run(&score, &mut midi_output).await?;

use fleet_ensemble::InstrumentAgent;

let piano = InstrumentAgent::new("piano")
    .voice(Voice::piano())
    .alignment_gain(0.25)      // conversational
    .reflex_latency(Duration::from_millis(10));

piano.run(&mut ensemble_bus).await?;

Instruments and the director communicate via the CNS protocol bus:

// Director broadcasts feel parameters
bus.broadcast(FeelTilt {
    rho: 0.6, epsilon: 0.7, sigma: 0.3,
    tau: -0.1, gamma: 0.5, lambda: 0.3, phi: 0.6,
    per_instrument: HashMap::from([
        ("piano", Offset { rho: -0.1, ..Default::default() }),
        ("drums", Offset { tau: 0.05, ..Default::default() }),
    ]),
}).await?;

// Instrument responds
let tilt = bus.receive().await?;
let adjusted_notes = instrument.align(notes, &tilt);

[director]
tempo = 120
pulse_rate_hz = 8          # 16th notes at 120 BPM
phrase_bars = 4            # LLM called every 4 bars

[director.oracle]
model = "deepseek-chat"
gateway = "http://127.0.0.1:8787/v1"
max_directives = 5

[director.maestro]
encoder_path = "weights/jepa_encoder.pt"
smoothing = 0.8            # EMA smoothing factor

[director.pulse]
tick_rate_hz = 1000        # 1 kHz internal clock

[instruments.piano]
program = 0                # GM Acoustic Grand Piano
alignment_gain = 0.25
polyphony = 8
register_range = [21, 108] # A0 to C8

[instruments.bass]
program = 33               # GM Electric Bass
alignment_gain = 0.7
polyphony = 1
register_range = [28, 60]

[instruments.drums]
program = 0                # drums use channel 10
alignment_gain = 0.9
is_drum = true

cargo test        # Unit tests: alignment math, CNS protocol, voice config
cargo test --test integration  # Multi-agent integration tests
cargo bench      # Performance: alignment latency, pulse tick timing

Tests verify:

• Alignment engine produces valid musical output under all feel parameter combinations
• CNS protocol packets are correctly serialized and deserialized
• Instrument agents respect polyphony limits and register ranges
• Director's tri-chamber switches correctly between Oracle/Maestro/Pulse
• 1 kHz tick loop maintains timing under load

Fleet Ensemble runs locally — the 1 kHz tick loop and sub-millisecond algorithmic engine demand it. The LLM Oracle can be remote (via fleet-gateway), but everything else must be local.

[Unit]
Description=Fleet Ensemble — Agentic Performance System
After=fleet-gateway.service

[Service]
Type=simple
WorkingDirectory=%h/projects/fleet-ensemble
ExecStart=%h/projects/fleet-ensemble/target/release/fleet-ensemble --score %h/scores/current.mid
Restart=always
MemoryMax=1G

[Install]
WantedBy=default.target

Connect Fleet Ensemble to your DAW via virtual MIDI ports:

1. Create a virtual MIDI input (e.g., ensemble_in)
2. Create a virtual MIDI output (e.g., ensemble_out)
3. Point Fleet Ensemble at those ports
4. Route MIDI tracks in your DAW to/from those ports

• Director Design — the full director architecture specification
• Instrument Agent Design — the instrument agent engineering spec
• CNS Protocol (cns-bridge) — the communication bus
• fleet-jepa-midi Design Docs — the JEPA encoder this system uses
• MIDI 1.0 Specification
• General MIDI Standard — instrument program numbers
• Stochastic Differential Equations in Rust — RNG for SDE solving

• Miles Davis Second Quintet (Wikipedia) — the musical reference for ensemble interaction
• Herbie Hancock — conversational comping
• Bill Evans Trio — democratic interplay
• Tony Williams — push and pull of time
• Elvin Jones — polyrhythmic foundation
• Ron Carter — the anchor
• The Jazz Process by Adrian Cho — collaboration lessons from jazz

• Stochastic Differential Equations (Wikipedia) — the math of the alignment engine
• Brownian Motion (Wikipedia) — the stochastic exploration term
• Lorenz System (Wikipedia) — atmospheric chaos theory inspiring the director
• Navier-Stokes Equations (Wikipedia) — fluid dynamics as ensemble metaphor
• Marangoni Effect (Wikipedia) — surface tension gradients = harmonic rotation
• Exponential Moving Average (Wikipedia) — smoothing and decoupling
• Game Theory (Wikipedia) — multi-agent alignment as cooperative game

• SDE Numerical Methods (Euler-Maruyama) — solving SDEs in discrete time
• Real-Time Audio Programming — why the 1 kHz tick matters
• MIDI Clock vs. MIDI Time Code — timing synchronization
• Jitter Buffer (Wikipedia) — handling timing variability
• Systemd Services — production deployment

• Agentic AI (Wikipedia) — the paradigm this system exemplifies
• Compiler Theory (Wikipedia) — the compiler → performer analogy
• Emergence (Wikipedia) — how complex behavior arises from simple agents
• Language Games (Wittgenstein) — meaning is in the playing
• Bob Ross Wet-on-Wet Technique — the canvas metaphor

Concept phase. Repo created Aug 13, 2026. Design in progress. Architecture is fully specified; implementation begins next.

MIT