Skip to content

Pull requests: huggingface/tokenizers

New pull request New
134 Open 929 Closed
134 Open 929 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Reviews
Filter by reviews
No reviews Review required Approved review Changes requested
Assignee
Filter by who’s assigned
Assigned to nobody Loading
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Pull requests list

docs: restore 2 dead link(s) via Wayback
#2238 opened Jul 23, 2026 by mirkosalvato1-ctrl Loading…
1
feat(pipeline+bindings): Support encoding pairs of sequences
#2237 opened Jul 23, 2026 by SBrandeis Contributor • Draft
perf(tk-encode): SIMD-first encode pipeline — fused split cache + de-virtualized BPE
#2234 opened Jul 23, 2026 by ArthurZucker Collaborator • Draft
bench: gigatoken per-step profile + take/have/skip analysis
#2233 opened Jul 23, 2026 by ArthurZucker Collaborator • Draft
1
implement ultra-basic, erroneous decode
#2232 opened Jul 22, 2026 by SBrandeis Contributor • Draft
1
feat(pipeline): Decode benches and stub
#2231 opened Jul 22, 2026 by SBrandeis Contributor Loading…
6
feat(bindings): Python bindings rework, plugging PipelineTokenizer
#2229 opened Jul 22, 2026 by SBrandeis Contributor • Draft
1
chore(deps): bump shell-quote from 1.8.1 to 1.10.0 in /bindings/node dependencies Pull requests that update a dependency file javascript Pull requests that update Javascript code
#2224 opened Jul 20, 2026 by dependabot Bot Loading…
1
feat(bpe): ScratchPool pattern + BPE word cache
#2223 opened Jul 20, 2026 by SBrandeis Contributor Loading…
30
chore(deps): bump brace-expansion from 1.1.13 to 1.1.16 in /bindings/node dependencies Pull requests that update a dependency file javascript Pull requests that update Javascript code
#2222 opened Jul 20, 2026 by dependabot Bot Loading…
1
bench(pipeline): add sebpop/upstream as a second reference series
#2221 opened Jul 20, 2026 by SBrandeis Contributor • Draft
1
Fix malformed BPE merges panicking during build
#2219 opened Jul 19, 2026 by 0xzr Loading…
1
Preserve backslashes in generated Python stub docstrings
#2218 opened Jul 18, 2026 by cupkk Loading…
1
5
Add a ParityBpeTrainer example and list it in the trainers API docs
#2217 opened Jul 18, 2026 by cimeister Contributor Loading…
Fix XSS in EncodingVisualizer HTML attributes
#2216 opened Jul 18, 2026 by varneeth01 Loading…
1
1
Fix dead wikitext-103 S3 link in quicktour.mdx
#2215 opened Jul 18, 2026 by RudrenduPaul Loading…
1
Fix token_to_sequence returning Some(0) for out-of-bounds index
#2214 opened Jul 17, 2026 by Osamaali313 Loading…
1
feat: multi threaded pipeline
#2213 opened Jul 17, 2026 by McPatate Member • Draft
1
Fix Metaspace decoder dropping internal spaces in the first token
#2212 opened Jul 17, 2026 by sohumt123 Loading…
Add missing return type hints to BaseTokenizer.save, save_model, to_str
#2211 opened Jul 17, 2026 by RudrenduPaul Loading…
Atomnorm
#2209 opened Jul 16, 2026 by ArthurZucker Collaborator Loading…
1
chore(deps-dev): bump websocket-driver from 0.7.4 to 0.7.5 in /tokenizers/examples/unstable_wasm/www dependencies Pull requests that update a dependency file javascript Pull requests that update Javascript code
#2208 opened Jul 15, 2026 by dependabot Bot Loading…
1
Pipeline-only tk-encode + tk-train on the pipeline (thin deps) — replaces #2200
#2207 opened Jul 15, 2026 by ArthurZucker Collaborator • Draft
fix: skip pre-tokenizer when is_pretokenized=True
#2204 opened Jul 15, 2026 by pctablet505 • Draft
1
Upgrade hf-hub to 1.0.0
#2203 opened Jul 15, 2026 by assafvayner • Draft
1
ProTip! Mix and match filters to narrow down what you’re looking for.