Name		Name	Last commit message	Last commit date
.github		.github
cmake		cmake
contrib/docker		contrib/docker
data		data
doc		doc
python		python
src		src
third_party		third_party
.gitignore		.gitignore
CMakeLists.txt		CMakeLists.txt
CONTRIBUTING.md		CONTRIBUTING.md
LICENSE		LICENSE
README.md		README.md
VERSION.txt		VERSION.txt
config.h.in		config.h.in
sentencepiece.pc.in		sentencepiece.pc.in

Repository files navigation

SentencePiece Parallel

This is not an official Google product.

Technical highlights

This fork utilizes multiple cores and smart buffering to divide-and-conquer large datasets. Takes a 10B token source from over an hour to under 10 minutes. Reduces memory overhead by ~50 percent.

About

Fork of Google's SentencePiece tokenizer that adds better memory management for large corpus volumes and increased throughput by parallelizing various parts.

1 star

0 watching

Report repository

Releases

Packages

Contributors

Languages