{"id":"CVE-2026-85670","aliases":["EUVD-2026-71126","GHSA-jrf5-f6jp-7vxc","RHLW-2026-b11abda52745af4c"],"status":"published","title":"tokenizers BpeBuilder Buffer Overflow via merge token","description":"tokenizers (Hugging Face) is affected by an out-of-bounds buffer access in BpeBuilder::build (tokenizers/src/models/bpe/model.rs). When loading a tokenizer.json via Tokenizer::from_file/from_str, the builder sizes a scratch buffer to the longest vocabulary key, then writes each concatenated merge rule into it. A merge whose concatenated token exceeds the longest vocabulary key overruns the buffer, which Rust turns into a panic that aborts the process in Rust and FFI embeddings. This occurs at load time with no encoding required, so an attacker who supplies a crafted tokenizer.json can cause a denial of service. A secondary defect at the same location can cause a usize underflow (panic in debug, potential memory corruption in release) when continuing_subword_prefix is set and a merge token is shorter than the prefix. Observed in version 0.23.1.","severity":{"version":"4.0","score":7.1,"vector":"CVSS:4.0/AV:N/AC:L/AT:N/PR:N/UI:P/VC:N/VI:N/VA:H/SC:N/SI:N/SA:N","level":"high","assessor":"cna","from":"cve"},"epss":{"day":"2026-11-10","score":0.00467,"percentile":0.40771,"model":"v2026.06.15"},"facts":{"in_kev":false,"known_exploited":false,"public_exploit":false,"fix_available":true},"cwes":["CWE-787"],"published":"2026-09-04T14:32:25Z","modified":"2026-10-28T02:30:33Z","fixes":[{"ecosystem":"Red Hat Lightwell:PyPI","package":"tokenizers","fixed_in":["0.23.1+rhlw.1"],"sources":["osv"]}],"affected":[{"source":"osv","ecosystem":"Red Hat Lightwell:PyPI","package":"tokenizers","ranges":[{"type":"ECOSYSTEM","events":[{"introduced":"0"},{"fixed":"0.23.1+rhlw.1"}]}],"versions":["0.0.1","0.0.10","0.0.11","0.0.12","0.0.13","0.0.2","0.0.3","0.0.4","0.0.5","0.0.6","0.0.7","0.0.8","0.0.9","0.1.0","0.1.1","0.10.0","0.10.1","0.10.2","0.10.3","0.11.0","0.11.1","0.11.2","0.11.3","0.11.4","0.11.5","0.11.6","0.12.0","0.12.1","0.13.0","0.13.1","0.13.2","0.13.3","0.14.0","0.14.1","0.15.0","0.15.1","0.15.2","0.19.0","0.19.1","0.2.0","0.2.1","0.20.0","0.20.1","0.20.2","0.20.3","0.20.4","0.20.4rc0","0.21.0","0.21.0rc0","0.21.1","0.21.1rc0","0.21.2","0.21.2rc0","0.21.4","0.22.0","0.22.0rc0","0.22.1","0.22.1rc0","0.22.2","0.22.2rc0","0.23.0rc0","0.23.1","0.23.1rc0","0.3.0","0.4.0","0.4.1","0.4.2","0.5.0","0.5.1","0.5.2","0.6.0","0.7.0","0.8.0","0.8.0.dev0","0.8.0.dev1","0.8.0.dev2","0.8.0rc1","0.8.0rc2","0.8.0rc3","0.8.0rc4","0.8.1","0.8.1rc1","0.8.1rc2","0.9.0","0.9.0.dev0","0.9.0.dev1","0.9.0.dev2","0.9.0.dev3","0.9.0.dev4","0.9.0rc1","0.9.0rc2","0.9.1","0.9.2","0.9.3","0.9.4"]}],"products":[{"vendor":"huggingface","product":"tokenizers"}],"references":[{"url":"https://github.com/huggingface/tokenizers/issues/2094","tags":["issue-tracking","exploit","report"]},{"url":"https://github.com/huggingface/tokenizers","tags":["product","package"]},{"url":"https://github.com/huggingface/tokenizers/blob/v0.23.2/tokenizers/src/models/bpe/model.rs","tags":["technical-description","article"]},{"url":"https://www.vulncheck.com/advisories/tokenizers-bpebuilder-buffer-overflow-via-merge-token","tags":["third-party-advisory","advisory"]},{"url":"https://nvd.nist.gov/vuln/detail/CVE-2026-85670","tags":["advisory"]},{"url":"https://github.com/advisories/GHSA-jrf5-f6jp-7vxc"},{"url":"https://github.com/CVEProject/cvelistV5/tree/main/cves/2026/85xxx/CVE-2026-85670.json","tags":["advisory"]}],"sources":{"cve":[{"id":"CVE-2026-85670","title":"tokenizers BpeBuilder Buffer Overflow via merge token","description":"tokenizers (Hugging Face) is affected by an out-of-bounds buffer access in BpeBuilder::build (tokenizers/src/models/bpe/model.rs). When loading a tokenizer.json via Tokenizer::from_file/from_str, the builder sizes a scratch buffer to the longest vocabulary key, then writes each concatenated merge rule into it. A merge whose concatenated token exceeds the longest vocabulary key overruns the buffer, which Rust turns into a panic that aborts the process in Rust and FFI embeddings. This occurs at load time with no encoding required, so an attacker who supplies a crafted tokenizer.json can cause a denial of service. A secondary defect at the same location can cause a usize underflow (panic in debug, potential memory corruption in release) when continuing_subword_prefix is set and a merge token is shorter than the prefix. Observed in version 0.23.1.","status":"published","published":"2026-09-04T14:32:25.391Z","modified":"2026-09-04T17:49:42.263Z","cvss":[{"version":"4.0","score":7.1,"vector":"CVSS:4.0/AV:N/AC:L/AT:N/PR:N/UI:P/VC:N/VI:N/VA:H/SC:N/SI:N/SA:N","level":"high","assessor":"cna"},{"version":"3.1","score":6.5,"vector":"CVSS:3.1/AV:N/AC:L/PR:N/UI:R/S:U/C:N/I:N/A:H","level":"medium","assessor":"cna"}],"cwes":["CWE-787"],"updated_at":"2026-11-02T14:19:52.410665Z"}],"euvd":[{"id":"EUVD-2026-71126","description":"tokenizers (Hugging Face) is affected by an out-of-bounds buffer access in BpeBuilder::build (tokenizers/src/models/bpe/model.rs). When loading a tokenizer.json via Tokenizer::from_file/from_str, the builder sizes a scratch buffer to the longest vocabulary key, then writes each concatenated merge rule into it. A merge whose concatenated token exceeds the longest vocabulary key overruns the buffer, which Rust turns into a panic that aborts the process in Rust and FFI embeddings. This occurs at load time with no encoding required, so an attacker who supplies a crafted tokenizer.json can cause a denial of service. A secondary defect at the same location can cause a usize underflow (panic in debug, potential memory corruption in release) when continuing_subword_prefix is set and a merge token is shorter than the prefix. Observed in version 0.23.1.","status":"published","published":"2026-09-04T14:32:25Z","modified":"2026-09-04T17:49:42Z","cvss":[{"version":"4.0","score":7.1,"vector":"CVSS:4.0/AV:N/AC:L/AT:N/PR:N/UI:P/VC:N/VI:N/VA:H/SC:N/SI:N/SA:N","level":"high"}],"updated_at":"2026-11-02T18:01:11.774269Z"}],"ghsa":[{"id":"GHSA-jrf5-f6jp-7vxc","title":"tokenizers (Hugging Face) is affected by an out-of-bounds buffer access in BpeBuilder::build ...","description":"tokenizers (Hugging Face) is affected by an out-of-bounds buffer access in BpeBuilder::build (tokenizers/src/models/bpe/model.rs). When loading a tokenizer.json via Tokenizer::from_file/from_str, the builder sizes a scratch buffer to the longest vocabulary key, then writes each concatenated merge rule into it. A merge whose concatenated token exceeds the longest vocabulary key overruns the buffer, which Rust turns into a panic that aborts the process in Rust and FFI embeddings. This occurs at load time with no encoding required, so an attacker who supplies a crafted tokenizer.json can cause a denial of service. A secondary defect at the same location can cause a usize underflow (panic in debug, potential memory corruption in release) when continuing_subword_prefix is set and a merge token is shorter than the prefix. Observed in version 0.23.1.","status":"published","published":"2026-09-04T15:36:15Z","modified":"2026-09-04T15:36:24Z","cvss":[{"version":"4.0","score":7.1,"vector":"CVSS:4.0/AV:N/AC:L/AT:N/PR:N/UI:P/VC:N/VI:N/VA:H/SC:N/SI:N/SA:N/E:X/CR:X/IR:X/AR:X/MAV:X/MAC:X/MAT:X/MPR:X/MUI:X/MVC:X/MVI:X/MVA:X/MSC:X/MSI:X/MSA:X/S:X/AU:X/R:X/V:X/RE:X/U:X","level":"high"},{"version":"3.1","score":6.5,"vector":"CVSS:3.1/AV:N/AC:L/PR:N/UI:R/S:U/C:N/I:N/A:H","level":"medium"}],"cwes":["CWE-787"],"updated_at":"2026-11-02T15:11:22.550434Z"}],"nvd":[{"id":"CVE-2026-85670","description":"tokenizers (Hugging Face) is affected by an out-of-bounds buffer access in BpeBuilder::build (tokenizers/src/models/bpe/model.rs). When loading a tokenizer.json via Tokenizer::from_file/from_str, the builder sizes a scratch buffer to the longest vocabulary key, then writes each concatenated merge rule into it. A merge whose concatenated token exceeds the longest vocabulary key overruns the buffer, which Rust turns into a panic that aborts the process in Rust and FFI embeddings. This occurs at load time with no encoding required, so an attacker who supplies a crafted tokenizer.json can cause a denial of service. A secondary defect at the same location can cause a usize underflow (panic in debug, potential memory corruption in release) when continuing_subword_prefix is set and a merge token is shorter than the prefix. Observed in version 0.23.1.","status":"published","published":"2026-09-04T15:17:44.993Z","modified":"2026-09-24T20:28:01.78Z","cvss":[{"version":"4.0","score":7.1,"vector":"CVSS:4.0/AV:N/AC:L/AT:N/PR:N/UI:P/VC:N/VI:N/VA:H/SC:N/SI:N/SA:N/E:X/CR:X/IR:X/AR:X/MAV:X/MAC:X/MAT:X/MPR:X/MUI:X/MVC:X/MVI:X/MVA:X/MSC:X/MSI:X/MSA:X/S:X/AU:X/R:X/V:X/RE:X/U:X","level":"high","assessor":"secondary"},{"version":"3.1","score":6.5,"vector":"CVSS:3.1/AV:N/AC:L/PR:N/UI:R/S:U/C:N/I:N/A:H","level":"medium","assessor":"secondary"}],"cwes":["CWE-787"],"updated_at":"2026-11-02T14:12:43.737972Z"}],"osv":[{"id":"CVE-2026-85670","title":"tokenizers BpeBuilder Buffer Overflow via merge token","description":"tokenizers (Hugging Face) is affected by an out-of-bounds buffer access in BpeBuilder::build (tokenizers/src/models/bpe/model.rs). When loading a tokenizer.json via Tokenizer::from_file/from_str, the builder sizes a scratch buffer to the longest vocabulary key, then writes each concatenated merge rule into it. A merge whose concatenated token exceeds the longest vocabulary key overruns the buffer, which Rust turns into a panic that aborts the process in Rust and FFI embeddings. This occurs at load time with no encoding required, so an attacker who supplies a crafted tokenizer.json can cause a denial of service. A secondary defect at the same location can cause a usize underflow (panic in debug, potential memory corruption in release) when continuing_subword_prefix is set and a merge token is shorter than the prefix. Observed in version 0.23.1.","status":"published","published":"2026-09-04T14:32:25.391Z","modified":"2026-10-28T02:30:33.192603323Z","cvss":[{"version":"4.0","score":7.1,"vector":"CVSS:4.0/AV:N/AC:L/AT:N/PR:N/UI:P/VC:N/VI:N/VA:H/SC:N/SI:N/SA:N","level":"high"}],"cwes":["CWE-787"],"updated_at":"2026-11-02T15:45:18.825071Z"},{"id":"RHLW-2026-b11abda52745af4c","title":"Lightwell Security Advisory: tokenizers 0.23.1+rhlw.1","description":"Red Hat Lightwell has released tokenizers 0.23.1+rhlw.1 with a backported security fix.\n\nCVE-2026-85670: tokenizers BpeBuilder Buffer Overflow via merge token\n\nThis patched artifact is a drop-in replacement for tokenizers 0.23.1 from PyPI.","status":"published","published":"2026-10-19T14:27:21Z","modified":"2026-10-27T08:45:03.134558402Z","cvss":[{"version":"4.0","score":7.1,"vector":"CVSS:4.0/AV:N/AC:L/AT:N/PR:N/UI:P/VC:N/VI:N/VA:H/SC:N/SI:N/SA:N","level":"high"},{"version":"3.1","score":6.5,"vector":"CVSS:3.1/AV:N/AC:L/PR:N/UI:R/S:U/C:N/I:N/A:H","level":"medium"}],"updated_at":"2026-11-02T15:52:58.636919Z"}]}}