7.1 HIGH
- CVSS version (CVSS): 4.0
- Attack Vector (AV): Network (N)
- Attack Complexity (AC): Low (L)
- Attack Requirement (AT): None (N)
- Privileges Required (PR): None (N)
- User Interaction (UI): Passive (P)
- Vulnerable System Impact Confidentiality (VC): None (N)
- Vulnerable System Impact Integrity (VI): None (N)
- Vulnerable System Impact Availability (VA): High (H)
- Subsequent System Impact Confidentiality (SC): None (N)
- Subsequent System Impact Integrity (SI): None (N)
- Subsequent System Impact Availability (SA): None (N)
- Modified Attack Vector (MAV): Network (N)
- Modified Attack Complexity (MAC): Low (L)
- Modified Attack Requirement (MAT): None (N)
- Modified Privileges Required (MPR): None (N)
- Modified User Interaction (MUI): Passive (P)
- Modified Vulnerable System Impact Confidentiality (MVC): None (N)
- Modified Vulnerable System Impact Integrity (MVI): None (N)
- Modified Vulnerable System Impact Availability (MVA): High (H)
- Modified Subsequent System Impact Confidentiality (MSC): Negligible (N)
- Modified Subsequent System Impact Integrity (MSI): Negligible (N)
- Modified Subsequent System Impact Availability (MSA): Negligible (N)
- Safety (S): Not Defined (X)
- Automatable (AU): Not Defined (X)
- Recovery (R): Not Defined (X)
- Value Density (V): Not Defined (X)
- Vulnerability Response Effort (RE): Not Defined (X)
- Provider Urgency (U): Not Defined (X)
- Confidentiality Req. (CR): Not Defined (X)
- Integrity Req. (IR): Not Defined (X)
- Availability Req. (AR): Not Defined (X)
- Exploit Maturity (E): Not Defined (X)
by @LeSuisse Activity log
- Created suggestion
-
@LeSuisse
ignored
8 packages
- openvino-tokenizers
- python313Packages.curated-tokenizers
- python313Packages.pytorch-tokenizers
- python314Packages.curated-tokenizers
- python314Packages.pytorch-tokenizers
- python313Packages.openvino-tokenizers
- python314Packages.openvino-tokenizers
- pkgsRocm.python3Packages.pytorch-tokenizers
- @LeSuisse accepted
- @LeSuisse published on GitHub
tokenizers BpeBuilder Buffer Overflow via merge token
tokenizers (Hugging Face) is affected by an out-of-bounds buffer access in BpeBuilder::build (tokenizers/src/models/bpe/model.rs). When loading a tokenizer.json via Tokenizer::from_file/from_str, the builder sizes a scratch buffer to the longest vocabulary key, then writes each concatenated merge rule into it. A merge whose concatenated token exceeds the longest vocabulary key overruns the buffer, which Rust turns into a panic that aborts the process in Rust and FFI embeddings. This occurs at load time with no encoding required, so an attacker who supplies a crafted tokenizer.json can cause a denial of service. A secondary defect at the same location can cause a usize underflow (panic in debug, potential memory corruption in release) when continuing_subword_prefix is set and a merge token is shorter than the prefix. Observed in version 0.23.1.
References
-
-
https://github.com/huggingface/tokenizers/blob/v0.23.2/tokenizers/src/models/bp… technical-description
-
VulnCheck Advisory: tokenizers BpeBuilder Buffer Overflow via merge token third-party-advisory
Affected products
- =<0.23.2
Matching in nixpkgs
pkgs.python313Packages.tokenizers
Fast State-of-the-Art Tokenizers optimized for Research and Production
Ignored packages (8)
pkgs.openvino-tokenizers
OpenVINO Tokenizers - text tokenisation extensions for OpenVINO
-
nixos-unstable 2026.2.1.0
- nixpkgs-unstable 2026.2.1.0
- nixos-unstable-small 2026.3.1.0
pkgs.python313Packages.curated-tokenizers
Lightweight piece tokenization library
pkgs.python313Packages.pytorch-tokenizers
C++ implementations for various tokenizers (sentencepiece, tiktoken, etc.)
pkgs.python314Packages.curated-tokenizers
Lightweight piece tokenization library
pkgs.python314Packages.pytorch-tokenizers
C++ implementations for various tokenizers (sentencepiece, tiktoken, etc.)
pkgs.python313Packages.openvino-tokenizers
OpenVINO Tokenizers Python API
-
nixos-unstable 2026.2.1.0
- nixpkgs-unstable 2026.2.1.0
- nixos-unstable-small 2026.3.1.0
pkgs.python314Packages.openvino-tokenizers
OpenVINO Tokenizers Python API
-
nixos-unstable 2026.2.1.0
- nixpkgs-unstable 2026.2.1.0
- nixos-unstable-small 2026.3.1.0
Package maintainers
-
@GaetanLepage Gaetan Lepage <gaetan@glepage.com>