Provide generator fragment for TextMate grammars

[kthoms]

It would be useful if Xtext could provide a generator fragment that creates a tmLanguage TextMate file for the grammar. This can be useful for the creation of language server extensions that make use of TextMate grammars like VSCode.
See also #1327

[cdietrich]

TODO:

• do we want to support JSON or Apples Dict XML / Property List (plist)
• what besides keywords (and maybe comments and strings) do we want to support

[miklossy]

Hint: The YAKINDU Solidity-IDE project already contains a hand-written TextMate grammar, see solidity.tmLanguage. This could serve as a good starting point when implementing such a generator fragment.

[rubenporras]

Hi,

we are thinking about writing one generator fragment for TextMate grammars for our codebase. We would start with something simple (e.g. keywords, maybe comments and strings as suggested above). I think it would be nice if we could contribute it, so I would like to check beforehand what would be the criteria for inclussion.

• Would something as simple as just keywords (maybe comments and strings) be good enough?
• Any preference regarding library to be used. It looks like other eclispe code is using https://github.com/google/gson, so that one comes first to my mind.
• If possible, we would try to aim for Xtext 2.26 release, if we are not too late.

Do you have any input?

[cdietrich]

having keywords, strings, ints, comments should be enhough
i assume no library at all would be best. (just rich strings as for the other templates)
@szarnekow what do you think`?

[szarnekow]

@rubenporras A contribution would be very welcome. This has been a long standing issue and it's great that you're volunteering to look into it.

Can you please help me understand why you'd need a library that serializes gson for that fragment?
I thought textmate grammars are either a sort of xml dialect or a JSON file - would you plan to provide a model along with the serialization to text? Would you consider this one to be some kind of API?

If it's not too cumbersome, a plain text generation is probably the best fit for the existing abstractions.

Keywords as a starter would be fine.
Strings and comments as well as other terminals can be added in a subsequent step.

A few questions:

• The textmate grammar belongs to the LSP client which is usually not contained within the Xtext project. How would you define the target directory for the generation?

• What would be the idea to specialize the tm grammar for other terminal rules?

• When you say comments and strings: Would that be based on the definition in Terminals.xtext?

• Some languages are case insensitive. Would this be supported by the keyword generation?

• Do you have an idea how the generated rules could be unit tested given the specific dialect of regexes in textmate grammars?

[rubenporras]

Hi @szarnekow,

• My initial thought would be put the textmate grammar on the ui plug-in, on a syntax folder. When using LSP, I think we will have an UI project for each DSL, even if minimal, right?
• Plain text is fine for me, no problem. I had in mind to have keywords as a starter.
• Having strings and comments based in Terminals.xtext would be nice if possible, but have not looked at it yet, so I would put it out of scope for now. I have no idea so far about the specialization, do you think is it necessary for the start?
• Yes, I would like the possibility to support case insensitive.
• No, no idea about testing, I did not thought about that yet. Would you have any idea?

[cdietrich]

@rubenporras with you ui plugin: do you mean the .ide or the .ui plugin. in most textmate usescases like vscode there wont be a .ui plugin i guess.
am not sure if tm grammars can be merged or if the generation should be a generate once.

[rubenporras]

@cdietrich , yes, some use cases might not have such .ui plugin, or maybe since it is small (basically the TextMate grammar and the plugin.xml to register a bunch of extension points, like a custom DocumentProvider), one would prefer to have the grammars for all supported DSLs in one plugin (i am actually thinking about that for our product). I can imagine, we could make it configurable.
Regarding if grammars can be merged, at least there is an include option, so if strings and comments are the same for all DSLs in a family, a base TextMate grammar could be generated. I was planning that as long temr for our product, specially if I would write the grammars by hand.

[jnt0r]

Just found this. Is there any update?

[cdietrich]

not from my side

[rubenporras]

Hi @cdietrich , we have by now written this generator and have used it internally, similar to #2639, it is not fully generic but a bit trimmed down to simplify the implementation given our needs.

Since we are now not working on it actively, I propose that we close the issue, unless you are interested in our implementation. If you are interested, I can go through the code, post the simplifications we did, and if you are still interested, see if we can do a PR.

[szarnekow]

A first version of it is available on main in https://github.com/eclipse/xtext/tree/main/org.eclipse.xtext.xtext.generator/src/org/eclipse/xtext/xtext/generator/textmate

[szarnekow]

@rubenporras Maybe you had something different in mind. We'd be happy to apply any improvements to this first version of the fragment.

[iTob191]

Maybe this will help someone else. To generate a textmate grammar, add

fragment = TextMateHighlightingFragment {
    absolutePath = "<output directory>"
}

inside the StandardLanguage block of your mwe2 workflow.