Co-authored-by: Simon Brandeis <33657802+SBrandeis@users.noreply.github.com>

* golden end-to-end tests through transformers v5

Same suite, two builds: `make golden-release` runs it on the tokenizers
wheel transformers resolves (always green, defines the target behavior);
`make golden` runs it on the in-tree build, where the failures enumerate
what the 1.0 rewrite is still missing. Four layers: the tokenizers API
driven directly, the transformers tokenizer API, inference (generate/
pipeline), and training (Trainer/collators).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* deepen golden tests with the transformers example recipes

Scenarios lifted from the maintained examples/pytorch/ scripts at
v5.14.1: run_qa's overflow/stride/sequence_ids/offsets span labeling,
run_ner's word_ids label alignment, run_mlm's special_tokens_mask +
masking collator, run_translation's text_target and seq2seq collator,
plus token-classification pipeline spans, encoder-decoder generate,
and the T5 Unigram+Metaspace archetype in the direct-API layer.
31 tests, all green on the release wheel.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* golden production flows: chat serving, RAG, SFT

Story-shaped tests mirroring how production stacks drive the tokenizer:
chat serving (ChatML template + control tokens as added specials +
history truncation to a token budget + left-padded batch generate +
TextIteratorStreamer streaming == one-shot decode), RAG (token-budget
chunking via offsets with exact source spans + mean-pooled embeddings +
retrieval), and SFT (TRL-style completion-only masking, tokenizer
pickled through datasets.map(num_proc=2), Trainer steps). 34 tests
green on the release wheel.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* golden conformance layer; rename the scenario suite to e2e

tests/golden now holds true golden-master tests: goldens/*.jsonl record
everything the released wheel (0.23.1) produces on the 8 bench-model
tokenizers — ids, tokens, offsets, masks, word ids, decodes and pair
encodings for fixture excerpts and edge-case strings, plus ids digests
over the fixture corpora. generate.py regenerates them from .release/
(never hand-edited, guarded); test_golden.py replays the inputs on the
current build, one test per output domain per model, so failures read
as a conformance matrix. Opt-in via TOKENIZERS_GOLDEN (make golden)
until the rewrite is green, then the gate should be dropped.

The transformers scenario suite moves to tests/e2e (make e2e /
e2e-release) — flows there, values here. CI: e2e-release and
golden-freshness (regen must reproduce the committed goldens byte for
byte) required; one informational in-tree job runs both.

First catch, dev build vs release on llama-2: identical ids but
tokens[0] renders as '▁<s>' instead of '<s>' — the Prepend normalizer
leaks into the inserted special token's string.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* add huggingface_hub to the dev venv

from_pretrained needs it (the wheel's optional hub extra), so the
network-marked pretrained test failed in a fresh venv — it only worked
after `make examples` pulled the dep in via datasets.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* review pass: scope HF_TOKEN per step, unshadow offset loop var

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* iter Makefile

* fix tk-encode doctest: fence PipelinePostProcessor example as text

rustdoc compiles indented doc blocks as Rust doctests; the example is an
illustration, not compilable code.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix Python lint under ruff 0.16 defaults; pin ruff in CI

ruff 0.16.0 widened the default rule set, breaking CI with no code
change: import sorting (I001), loop vars captured by bench lambdas
(B023), implicit string concat in lists (ISC004), open() without a
context manager (SIM115), zip-for-pairwise (RUF007), and a blind
pytest.raises(Exception) (B017). Pin the CI ruff so rule-set changes
land deliberately.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>