Breaking change proposal: Encoding.UTF8 singleton should not have a BOM
## tl;dr The `Encoding.UTF8` singleton currently says "please emit a BOM when writing." This is an anachronism. Nowadays, it should say "please _do not_ emit a BOM when writing." The `Encoding.UTF8` singleton should continue to perform U+FFFD substitution on invalid subsequences, just as it does today. ## Discussion More information: https://github.com/dotnet/standard/issues/260, https://github.com/dotnet/runtime/issues/7779, with further discussion at https://github.com/dotnet/runtime/issues/28218 Historically, the `Encoding.UTF8` singleton has been equivalent to `new UTF8Encoding(encoderShouldEmitUTF8Identifier: true, throwOnInvalidBytes: false)`. This is largely for historical reasons, as these types were introduced during a period when multiple different encodings were commonplace, and the world hadn't yet settled on UTF-8 as the de facto standard. Now, 20 years later, UTF-8 has cemented its place as the true winner, and many tools across Unix and Windows natively operate on UTF-8. But as mentioned in the above linked issues, these tools can fail if they encounter a BOM at the start of the data. The Unicode maintainers have also discussed recommending _against_ the use of BOMs by default unless explicitly required by the protocol or file format. * https://www.unicode.org/L2/L2021/21038-bom-guidance.pdf (guidance being drafted, but not yet adopted) * https://corp.unicode.org/pipermail/unicode/2020-October/009070.html (previous discussion on this issue which led to a
Posted by GrabYourPitchforks on Apr 15, 2021, 11:36 PM
Tags
Related issues
- [BUG] Cowork (Windows): all Plan9 shares fail with "Plan9 mount failed: invalid argument" after Windows update KB5124008 (26200.9445) — uninstalling the KB fixe
- Feature Request: Auto-resume CLI session when usage limit resets
- Please deobfuscate the Python
- [Bug] MLX version of Gemma 4 26B fails to load: "Model type gemma4 not supported" (macOS)
- # 233 Developers Discussed Abandoning the Office Add-in Platform — An Open Letter on the Office.js Stability, Security & Trust Crisis
- [FEATURE] Allow users to configure which shell the Bash tool uses or inherit initial shell
Raise real-world problems, find people facing the same issue, and discuss solutions.3,000 issues0 comments
reliability (Software)
[BUG] Cowork (Windows): all Plan9 shares fail with "Plan9 mount failed: invalid argument" after Windows update KB5124008 (26200.9445) — uninstalling the KB fixe
Feature Request: Auto-resume CLI session when usage limit resets
Please deobfuscate the Python
[Bug] MLX version of Gemma 4 26B fails to load: "Model type gemma4 not supported" (macOS)
# 233 Developers Discussed Abandoning the Office Add-in Platform — An Open Letter on the Office.js Stability, Security & Trust Crisis
[FEATURE] Allow users to configure which shell the Bash tool uses or inherit initial shell
Admin Materials and Surfaces
AMD Support
Breaking change proposal: Encoding.UTF8 singleton should not have a BOM
## tl;dr The `Encoding.UTF8` singleton currently says "please emit a BOM when writing." This is an anachronism. Nowadays, it should say "please _do not_ emit a BOM when writing." The `Encoding.UTF8` singleton should continue to perform U+FFFD substitution on invalid subsequences, just as it does today. ## Discussion More information: https://github.com/dotnet/standard/issues/260, https://github.com/dotnet/runtime/issues/7779, with further discussion at https://github.com/dotnet/runtime/issues/28218 Historically, the `Encoding.UTF8` singleton has been equivalent to `new UTF8Encoding(encoderShouldEmitUTF8Identifier: true, throwOnInvalidBytes: false)`. This is largely for historical reasons, as these types were introduced during a period when multiple different encodings were commonplace, and the world hadn't yet settled on UTF-8 as the de facto standard. Now, 20 years later, UTF-8 has cemented its place as the true winner, and many tools across Unix and Windows natively operate on UTF-8. But as mentioned in the above linked issues, these tools can fail if they encounter a BOM at the start of the data. The Unicode maintainers have also discussed recommending _against_ the use of BOMs by default unless explicitly required by the protocol or file format. * https://www.unicode.org/L2/L2021/21038-bom-guidance.pdf (guidance being drafted, but not yet adopted) * https://corp.unicode.org/pipermail/unicode/2020-October/009070.html (previous discussion on this issue which led to a
Comments (0)
No comments yet. Start the discussion.