SIMD-enabled utf-8 validation

<!-- homu-ignore:start --> *[View all comments](https://triagebot.infra.rust-lang.org/gh-comments/rust-lang/rust/issues/68455)* <!-- homu-ignore:end --> ## Introduction The ["Parsing Gigabytes of JSON per second"](https://branchfree.org/2019/02/25/paper-parsing-gigabytes-of-json-per-second/) post ([ArXiv - langdale, lemire](https://t.co/MgQINMJlNB?amp=1)) proposes a novel approach for parsing JSON that is fast enough that on many systems it moves the bottleneck to the disk and network instead of the parser. This is done through the clever use of SIMD instructions. Something that stood out to me from the post is that JSON is required to be valid utf-8, and they had come up with new algorithms to validate utf-8 using SIMD instructions that function *much* faster than conventional approaches. Since rustc does a *lot* of utf-8 validation (each `.rs` source file needs to be valid utf-8), it got me curious about what rustc currently does. Validation seems to be done by the following routine: https://github.com/rust-lang/rust/blob/2f688ac602d50129388bb2a5519942049096cbff/src/libcore/str/mod.rs#L1500-L1618 This doesn't appear to use SIMD anywhere, not even conditionally. But it's run a *lot*, so I figured it might be interesting to use a more efficient algorithm for. ## Performance improvements The post ["Validating UTF-8 strings using as little as 0.7 cycles per byte"](https://lemire.me/blog/2018/05/16/validating-utf-8-strings-using-as-little-as-0-7-cycles-per-byte/) shows

Posted by yoshuawuyts on Jan 22, 2020, 1:36 PM

Tags

• github (Network)

Related issues

• [UPDATE NOW] PHP RCE Vulnerability (CVE-2026-109618 and CVE-2026-109619)
• Server requests and latency increased after upgrading from Next.js 15 to 16
• Leak of Geedge Networks internal documents (100,000+ from Jira, Confluence, GitLab)
• Can't download videos from YouTube on cobalt.tools
• `google_service_networking_connection` produce an error when trying to destroy the resource
• WebGPU-based renderer for the editor