Tokenization overhaul
The current tokenisation story of VS Code is based on TM grammars, which are pretty powerful, but we are running into their limits if we want to do something more than a top-down scanner can do. Also, once you implement a TM interpreter, you realise how inefficient the way in which regular expressions must be evaluated is and how TM grammars were not meant to do much more than simple colouring using just a few rules... The fact that we now have these complex grammars than end up producing beautiful tokens is more of a testament to the amazing computing power available to us than to the design of the TM grammar semantics. At the time when TM grammars were introduced and became popular there were no language servers available which understand the semantics of a language. Therefore, TM grammars were also used to colour semantic constructs. The introduction of LSP has brought us language servers for many languages and it we want to leverage this power to reduce the complexity of the tokenizer/classifier. There is already effort under way to specify how such API might look under LSP at https://github.com/microsoft/vscode-languageserver-node/pull/367 In any case, for smarter languages where we offer great extensions, such as for TypeScript or C++, we have noticed two different patterns emerge to try and compensate for these limitations. --- #### Complex TM grammar approach (TypeScript) This approach was taken by TypeScript, where we now have immense regular expressions, which a
Posted by alexdima on Jul 10, 2019, 8:59 PM
Tags
Related issues
- [BUG] Cowork (Windows): all Plan9 shares fail with "Plan9 mount failed: invalid argument" after Windows update KB5124008 (26200.9445) — uninstalling the KB fixe
- Feature Request: Auto-resume CLI session when usage limit resets
- Please deobfuscate the Python
- [Bug] MLX version of Gemma 4 26B fails to load: "Model type gemma4 not supported" (macOS)
- # 233 Developers Discussed Abandoning the Office Add-in Platform — An Open Letter on the Office.js Stability, Security & Trust Crisis
- [FEATURE] Allow users to configure which shell the Bash tool uses or inherit initial shell
Raise real-world problems, find people facing the same issue, and discuss solutions.3,500 issues0 comments
reliability (Software)
[BUG] Cowork (Windows): all Plan9 shares fail with "Plan9 mount failed: invalid argument" after Windows update KB5124008 (26200.9445) — uninstalling the KB fixe
Feature Request: Auto-resume CLI session when usage limit resets
Please deobfuscate the Python
[Bug] MLX version of Gemma 4 26B fails to load: "Model type gemma4 not supported" (macOS)
# 233 Developers Discussed Abandoning the Office Add-in Platform — An Open Letter on the Office.js Stability, Security & Trust Crisis
[FEATURE] Allow users to configure which shell the Bash tool uses or inherit initial shell
Admin Materials and Surfaces
AMD Support
Tokenization overhaul
The current tokenisation story of VS Code is based on TM grammars, which are pretty powerful, but we are running into their limits if we want to do something more than a top-down scanner can do. Also, once you implement a TM interpreter, you realise how inefficient the way in which regular expressions must be evaluated is and how TM grammars were not meant to do much more than simple colouring using just a few rules... The fact that we now have these complex grammars than end up producing beautiful tokens is more of a testament to the amazing computing power available to us than to the design of the TM grammar semantics. At the time when TM grammars were introduced and became popular there were no language servers available which understand the semantics of a language. Therefore, TM grammars were also used to colour semantic constructs. The introduction of LSP has brought us language servers for many languages and it we want to leverage this power to reduce the complexity of the tokenizer/classifier. There is already effort under way to specify how such API might look under LSP at https://github.com/microsoft/vscode-languageserver-node/pull/367 In any case, for smarter languages where we offer great extensions, such as for TypeScript or C++, we have noticed two different patterns emerge to try and compensate for these limitations. --- #### Complex TM grammar approach (TypeScript) This approach was taken by TypeScript, where we now have immense regular expressions, which a
Comments (0)
No comments yet. Start the discussion.