Skip to content

Commit 0af044c

Browse files
Update tokenizers requirement from 0.20 to 0.21
Updates the requirements on [tokenizers](https://github.com/huggingface/tokenizers) to permit the latest version. - [Release notes](https://github.com/huggingface/tokenizers/releases) - [Changelog](https://github.com/huggingface/tokenizers/blob/main/RELEASE.md) - [Commits](huggingface/tokenizers@v0.20.0...v0.21.0) --- updated-dependencies: - dependency-name: tokenizers dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com>
1 parent 2c9ccad commit 0af044c

File tree

1 file changed

+1
-1
lines changed

1 file changed

+1
-1
lines changed

crates/bpe/benchmarks/Cargo.toml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -23,4 +23,4 @@ bpe-openai = { path = "../../bpe-openai" }
2323
criterion = "0.5"
2424
rand = "0.8"
2525
tiktoken-rs = "0.6"
26-
tokenizers = { version = "0.20", features = ["http"] }
26+
tokenizers = { version = "0.21", features = ["http"] }

0 commit comments

Comments
 (0)