feat: add the K3 XTML chat core and REPL, without the bundled tokenizer - #55
feat: add the K3 XTML chat core and REPL, without the bundled tokenizer#55Avicennasis wants to merge 7 commits into
Conversation
Exact rendering and tokenisation of the checkpoint's XTML chat format: message envelopes, the assistant generation prompt, reasoning_content on prior assistant turns, JSONL history load/save, and a parser for one assistant completion. Untrusted user text is encoded with added tokens disallowed, so a user typing "<|end_of_msg|>" cannot forge a control id. Cherry-picked from BlakeEvans22's PR FareedKhan-dev#20 with one change: the bundled copies of tiktoken.model, tokenizer_config.json, encoding_k3.py and the Kimi license are NOT carried over. That bundle is what closed FareedKhan-dev#20. The chat gate now takes TOK_FILES and prints NOT RUN when the vocabulary is absent, exactly like the tokenizer gate; under CMake it goes through cmake/run_chat_test.cmake, a copy of run_tok_test.cmake, so ctest keeps counting it and reports a skip rather than dropping it from the list.
Wire --chat, transcript restart, system-message validation, REPL commands, and deterministic sampling into the streamed CPU inference CLI. Keep batch completion and its memory-budget behavior unchanged. (cherry picked from commit ae91df7)
Document the official text-only chat command, transcript privacy and restart behavior, sampling/reproducibility, inherited context limits, CPU/disk memory model, test coverage, and remaining scope. (cherry picked from commit 8214583)
…/--seed The REPL as landed sampled by default (temperature 1.0, top-p 0.95) and offered --greedy to switch it off. That inverts docs/ROADMAP.md item 5, which says sampling must be opt-in and off by default, and it would have made --chat the one path in the engine whose output cannot be reproduced. Greedy is now the default in chat as in batch; any sampling flag turns sampling on, and --greedy still forces argmax over them. Also: docs/TESTING.md described a tests/fixtures/chat bundle that this branch does not carry (see the chat-core commit), and the REPL commit's ROADMAP edit had dropped the Sampling item entirely. Both put back.
…<|end_of_msg|> The released checkpoint declares two end-of-sequence ids that disagree: config.json / generation_config.json say <|end_of_msg|> (163586), while tokenizer_config.json says [EOS] (163585). Run against the real weights, the model closes its turn with <|close|>message<|sep|> followed by [EOS] -- 163585 -- and never emits 163586, which the template inserts between messages. The REPL stopped only on 163586 and the parser accepted only a 163586 closure, so against the real model a --chat turn would have run to --gen (4096 by default in chat: about four days at this engine's speed) and then been rejected as malformed. K3ChatTemplate gains eos_id ([EOS], checked to be 163585 like eom_id is checked to be 163586); the REPL breaks on either id and prints which one ended the turn; the parser accepts a closure followed by either. Two test_chat cases cover the new behaviour: a turn ended by [EOS] parses, and a closure followed by any other id is still rejected. Verified after the change: test_chat passes against the released tokenizer; make test green.
|
Pushed one more commit, 8bf2d04, after reading the REPL against what the released model actually emits. The finding. The checkpoint declares two end-of-sequence ids that disagree: The fix. Pending. A real-checkpoint REPL smoke (one greedy turn, |
At the speeds a streamed trunk runs at, a minute or two per token, the REPL printed nothing between reading the user line and printing the finished turn. A six-hour run that hit its timeout left no record of how far it had got, and a turn cut short by --gen looked identical to a hung process. One unbuffered line per token: count against --gen, the id, and seconds since the turn began. Not exercised on the tiny checkpoint, whose 256-id vocabulary cannot host the template's control ids (the REPL refuses it, correctly); the scheduled real-checkpoint run is the first exercise of this line.
Real-model REPL smoke (dev, 2026-09-06)
Result: structurally complete turn, correct answer, cap reached one token before the end id. Decoded output (150 ids, greedy; think block abbreviated): Token 150 was the Timing at Log: dev |
|
Heads up before the workflow approval goes through — In It's the only finding in The fix is to rename - enc, cfg, special = load()
+ enc, _cfg, special = load()
Separately: the description mentions crediting me in CONTRIBUTORS.md, but the diff doesn't touch that file. Not a blocker, just flagging it since it'd be easy to lose at merge time. Thanks for picking this up and dropping the tokenizer bundle — that was the right call. |
…int encoder
The checkpoint's own encoder (encoding_k3.py) takes thinking=True/False and
thinking_effort in {low, high, max}; the C template hard-coded thinking on at
max. Every real-model smoke so far spent most of its budget in the think
channel (119 of 150 tokens in the last one, at ~140 s each), so the option
that skips it is the largest speedup available without touching the engine.
- K3ChatOptions {thinking, thinking_effort}; k3_chat_render_opts and
k3_chat_parse_assistant_opts take it, the old entry points wrap the
defaults (thinking on, max) so existing callers and bytes are unchanged.
- thinking=False: no thinking-effort system message, stored assistant turns
render without their think channel, the generation prompt opens
<|open|>response<|sep|> directly, and the parser expects response-only
output with reasoning_content NULL. All as the encoder does.
- thinking_effort: the effort message is one XTML segment (BPE merges cross
"=max", so it cannot be split); "medium" is refused because the encoder's
_VALID_THINKING_EFFORTS asserts on it even though its prompt text lists it.
- CLI: --no-think, --thinking-effort E; both refused outside --chat, together,
or with a bad effort (rc 2), gated in make test with a fake model dir so
the exit code cannot come from the loader.
- tests: no-think (37 ids), effort=low (105 ids) and two-turn no-think
(64 ids) renders checked byte- and id-exact against the official tokenizer;
parser cases for response-only turns, missing end id, and cross-mode
rejection.
Reported by the real-model chat smoke on PR FareedKhan-dev#55.
|
Pushed Why. The smoke above spent 119 of its 150 tokens in the think channel at ~140 s each. The encoder ( What changed.
Verification.
Next. Smoke #4 is queued on the same box:
|
|
Smoke #4 result: with Same box, same checkpoint, same message ("Reply with exactly five words."), binary
Generated ids: Two things worth noting for the review:
Nothing else on this PR is pending from my side. |
What this changes
--chat: a terminal REPL that renders the checkpoint's own XTML chat format exactly -- message envelopes, thethinking_effortsystem preamble, the assistant generation prompt,reasoning_contenton prior assistant turns -- with JSONL history load/save and a parser for one assistant completion. ROADMAP item 6 ("Chat template"). Greedy by default;--temperature/--top-p/--seedopt in to sampling.Why
Without the template the engine completes the prompt instead of answering it. I hit this on the released checkpoint: a hand-built XTML prompt fed through
--idsgave a correct, well-formed answer,tok.py chat(which looks for achat_templatekey the checkpoint does not ship) silently gave a document continuation. The format is not guessable from ChatML; it is defined byencoding_k3.pyin the checkpoint, and this C core reproduces it.Provenance, and the change from #20
The chat core and REPL are @BlakeEvans22's, from #20, which was closed for one stated reason: it bundled
tiktoken.model,tokenizer_config.json,encoding_k3.pyand the Kimi license as a test fixture. This branch carries his three commits with authorship preserved (-xtrailers), cherry-picked via @ScriptedAlchemy's fork where they lived on, and drops the bundle entirely -- no commit in this PR ever adds those files.test_chatnow takesTOK_FILES, exactly liketest_tok, and prints NOT RUN when the vocabulary is absent; under CMake it goes throughcmake/run_chat_test.cmake, a copy ofrun_tok_test.cmake, so ctest keeps counting it and reports a skip rather than dropping it.My two additions on top: the fixture removal and its wiring, and a commit making chat greedy by default -- the REPL as written sampled by default with
--greedyto opt out, which inverts ROADMAP item 5 and would have made--chatthe only path whose output cannot be reproduced. That commit also restores the Sampling item his ROADMAP edit had removed. Blake should be the CONTRIBUTORS.md name for the feature.Verification
make testpasses (all weightless gates); the chat gate reports NOT RUN without a vocabularymake test TOK_FILES=/path/to/Kimi-K3passes with the released tokenizer:roundtrip: 94402 bytes -> 28264 ids -> 94402 bytes : PASS,CHAT TESTS PASSEDmake portablebuilds with no new warningsmake tok(the roundtrip above) passes;third_party/tok.hgainstok_encode_mode()so untrusted user text is encoded with added tokens disallowed -- a user typing<|end_of_msg|>cannot forge a control id -- and fixes a signed-overflow UB in the base64 rank decoder on long lines--chatchangesDifferential parity against Moonshot's own encoder, which is the check this thread could never run: rendering
[system "You are helpful.", user "Hello"]withadd_generation_prompt=True, thinking=True, thinking_effort="max"through the checkpoint'sencoding_k3.pyand encoding with itstiktoken.modelgives 104 ids;test_chat --emiton the C core gives the same 104 ids, identical at every position.Numbers, if this is a performance change
Not a performance change.
Risk
The template always emits the
thinking_effort=maxpreamble and opens a<think>channel, as the official encoder does by default; there is no flag yet to renderthinking=False, so a chat turn pays for the model's reasoning tokens before its answer. On a streamed-trunk desktop preset that is real money per turn (this engine is ~95 s/token there) and is the obvious follow-up. The REPL rebuilds the full prefill from the JSONL history on every restart; retained in-process state across turns is listed on the ROADMAP behind an equivalence gate, not done here.