Syntax highlighting in the playground

[aspiwack]

It would be nice to have syntax highlighting of code samples (both source and formatted) and of the queries in the playground.

Considering that we already load tree-sitter grammar for our language (and for Tree-sitter query files). It should, in principle, be natural to use these grammars for syntax highlighting.

[torhovland]

The ironic part is that the Ace editor component we are using comes with highlighting for over 100 languages built-in, but it's not based on Tree-sitter. Instead it's based on this specification.

So the easiest way to do this would be to rely on their existing highlighting for most of our languages, and then define highlighting for Nickel and the Tree-sitter query language.

But perhaps we rather want to do it the hard way, using Tree-sitter?

[torhovland]

Built-in OCaml, for example:

[aspiwack]

If most of our languages come with an already existing syntax specification for Ace, then let's just do the easy thing first. And think about potential improvement based on tree-sitter grammars in the future. It is ironic, as you say, but having reasonable syntax highlighting is what matters, so the shortest path to there is what we ought to do.

[torhovland]

Now, even if we wanted to do it the hard way, it doesn't look feasible with this Ace editor. It really insists of defining highlighting using its Regex-based approach.

The question, then, is if we should be looking at alternative editor components, better now than later.

[aspiwack]

It doesn't cost much to do a little research. How did you choose Ace editor?

[torhovland]

I saw the Nickel playground uses it.

[aspiwack]

Fair enough 🙂

@amesgen (for Ormolu) and @yannham did you do research in webcomponents for code editing? If so could you share?

[torhovland]

Some options:

Insisting on Regex:

• Ace - A comprehensive web-based code editor that also includes powerful syntax highlighting capabilities.
• Monaco Editor - The code editor that powers VS Code, this provides syntax highlighting among many other features.
• Refractor - Lightweight, robust, elegant virtual syntax highlighting using Prism.
• Lowlight - Virtual syntax highlighting for virtual DOMs and non-HTML things.

Need to map parsed nodes to a set of proprietary to highlighting tags:

• CodeMirror - A versatile text editor implemented in JavaScript for the browser. It's specialized for editing code and features over 100 languages.

Can only output HTML, not be used in an editor:

• tree-sitter-hast - NPM package to convert tree-sitter parsed syntax trees to syntax-highlighted hast (Hypertext Abstract Syntax Trees).
• Rainbow - A small, lightweight library for syntax highlighting.
• SHJS (Syntax Highlighting in JavaScript) - A JavaScript program which highlights source code passages in HTML documents.
• SyntaxHighlighter - A fully functional self-contained code syntax highlighter developed in JavaScript.
• Prism - Prism is a lightweight, extensible syntax highlighter, built with modern web standards in mind. It’s used in millions of websites, including some of those you visit daily.
• Highlight.js - Syntax highlighting for the Web
• starry-night - Syntax highlighting, like what GitHub uses to highlight code, but free and open source and JavaScript!
• React Syntax Highlighter - Syntax highlighting component for React using the seriously super amazing lowlight and refractor by wooorm

[torhovland]

Given that the Nickel team uses Ace and have implemented highlighting for it, and that the Tree-sitter query language probably can be sorted out with a lightly modified Scheme mode, perhaps Ace isn't such a bad choice after all. We can always fall back to plain text mode, which for now we may have to do for the new Ocamllex language.

[yannham]

Highlighting is tiring. We have no less than 6 grammars for Nickel:

• the LALRPOP one for the interpreter
• the Vim one for the vim plugin - at the time we started it, the tree-sitter builtin support in neovim wasn't exactly streamlined
• the emacs plugin where the community apparently prefers to have a bare-bone entry plugin and let people do more complex stuff if they want LSP or tree-sitter based highligting 🤷‍♂️
• The Ace.js syntax for the editor in the playground
• A Prism.js syntax for highlighting the code
• The tree-sitter grammar for generic highligthing and Topiary

It's frustrating and annoying to maintain. On factor was that tree-sitter is not always fully integrated enough with editor to make it the default way (like, you need a recent version of neovim or to install third party plugins).

I believe I chose the Ace editor after a quick review of the existing solutions at the time, based on the features, on highlighting capabilities, the possibility to show errors directly in-code, and also probably not least based on which ones were a drop-in component (almost) in React/Gatsby, so that I could just write a few lines and get a working editor on the website. I also didn't have many days to spend on the website and although I've done some fullstack development a bit before, I wouldn't describe myself exactly as a web developer, so I might have cut corners or make suboptimal choices.

There is an open issue about replacing Ace.js with Monaco, in particular because we do an error messages mambo-jumbo to mimic LSP diagnostics by hand, but because Monaco is basically the VSCode engine, we might get full LSP features out of the box, and also re-use the VSCode extension highlighing. Oh, by the way, I forgot this one, so we actually maintain 7 different Nickel grammars...

[amesgen]

I chose ace for Ormolu Live via a very similar process as @yannham described above 😆 (another point in favor was that it has Purescript bindings; Ormolu Live is Purescript for the UI + WASM for the actual formatting).

Ace highlighting for Haskell is far from perfect, but seemed acceptable for a formatter playground. Previously, we used ghc-syntax-highlighter to format the output, which is great as the highlighting is correct by directly using the GHC parser, but isn't easy to plug that into a JS code editor. CodeMirror might be flexible enough, but I haven't tried it.

Minor comment on Monaco: It does not work in mobile contexts; probably not a huge deal, but I think it can sometimes be neat to be able to try simple examples on mobile.