Cross platform universal tokenizer binding to HF and sentencepiece
templeblock/tokenizers-cpp
Folders and files
| Name | Name | Last commit date | ||
|---|---|---|---|---|
Repository files navigation
Releases
No releases published
Languages
- C++ 27.4%
- Rust 24.4%
- CMake 18.9%
- TypeScript 10.0%
- HTML 8.2%
- C 5.3%
- Other 5.8%