feat: Add support for tokens in toml files

[xitij2000]

Description

Currently, the only way to specify tokens is using JSON, however there are some unique benefits to using the TOML format for design tokens. TOML is a format for configuration files that resembles the ini but can support richer nested data. It has been embraced as the default format by Python for pyproject files and by Rust for it's Cargo.toml files.

There are a couple of reasons why it's particularly good for the usecase of design tokens.

1. It has a simpler way of representing deeply nested data

Design tokens are deeply nested and in JSON it can get a bit clumsy to represent this. To take a particularly deeply nested value, we have other.content.form.control.checkbox.indicator.icon-checked.base which is 8 levels deep. The JSON for such deeply nested values can be pretty verbose and can be hard to follow. In comparison with TOML this is just:

[other.content.form.control.checkbox.indicator.icon-checked.base]
value = "..."
type = "..."
source = "..."

The above dot notiation very easily maps to the final variable name, and you can quickly do a search for such nested data without checking the full heirarchy.

2. The order of nested TOML documents can be arbitrary

In the above example, if you wanted to add value for other.content.form you can simply do that at any point
in the document. Unlike JSON where you'd need to have it nested in the correct hierarchy.

3. Changes in nesting can be done much quicker

If in the above example, you wanted to change from the checkbox.indicator to checkbox-indicator or move from other.content.form to other.form only for this value without affecting other values, you can do that with just a single change without impacting any otherpart of the hierarchy. With JSON if you rename an attribute that automatically affects all the values in that heirarchy, so you'd have to move this bit of the document to a new attribute. TOML really simplifies refactoring.

4. TOML has native support for comments

While design tokens can support comments via field, this is part of the document and will be processed as such. This is not a huge issue, but still a nice to have.

NOTE: This PR merely adds a way to support TOML for design tokens. It is not a recommendation to switch the entirety of the token system to use toml instead of JSON.

While doing some work on design tokens I had to deal with a lot of cases where I'd expressed the hierarchy of values incorrectly and it was a pain to make the changes. I found this much easier with toml so began converting JSON to TOML to make the changes and then convert back. I figured it would be nice to simply add support to Paragon itself since it was a small enough change.

Testing

You can test this by creating a minimal setup for design tokens.

• Create a new directory with the following directory structure inside it: themes/light . (e.g. /tmp/toml-tokens/
• Inside the light folder create a toml file with some design tokens, for example:

   [color.testing]
   value = "#123456"

• Check out this version of Paragon, install the dependencies and run: ./bin/paragon-scripts.js build-tokens --source /tmp/toml-tokens/ --build-dir /tmp/toml-tokens/build --source-tokens-only
• Check the output build directory in the folder and you'll see a file called themes/light/variables.css that has a css variable called --pgn-color-testing with the above value.

Deploy Preview

NA

Merge Checklist

• If your update includes visual changes, have they been reviewed by a designer? Send them a link to the Netlify deploy preview, if applicable.
• Does your change adhere to the documented style conventions?
• Do any prop types have missing descriptions in the Props API tables in the documentation site (check deploy preview)?
• Were your changes tested using all available themes (see theme switcher in the header of the deploy preview, under the "Settings" icon)?
• Were your changes tested in the example app?
• Is there adequate test coverage for your changes?
• Consider whether this change needs to reviewed/QA'ed for accessibility (a11y). If so, please request an a11y review for the PR in the #wg-paragon Open edX Slack channel.

Post-merge Checklist

• Verify your changes were released to NPM at the expected version.
• If you'd like, share your contribution in #show-and-tell.
• 🎉 🙌 Celebrate! Thanks for your contribution.

[openedx-webhooks]

Thanks for the pull request, @xitij2000! Please note that it may take us up to several weeks or months to complete a review and merge your PR.

Feel free to add as much of the following information to the ticket as you can:

• supporting documentation
• Open edX discussion forum threads
• timeline information ("this must be merged by XX date", and why that is)
• partner information ("this is a course on edx.org")
• any other information that can help Product understand the context for the PR

All technical communication about the code itself will be done via the GitHub pull request interface. As a reminder, our process documentation is here.

Please let us know once your PR is ready for our review and all tests are green.

[netlify]

✅ Deploy Preview for paragon-openedx ready!

Built without sensitive environment variables

Name	Link
🔨 Latest commit	f325bb3
🔍 Latest deploy log	https://app.netlify.com/sites/paragon-openedx/deploys/6641f9bfd9266500080dc5dd
😎 Deploy Preview	https://deploy-preview-3047--paragon-openedx.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[DanielVZ96]

Tested and LGTM

[bradenmacdonald]

@adamstankiewicz Could we get your thoughts on this?

[brian-smith-tcril]

@bradenmacdonald I have added this to the agenda for tomorrow's Paragon WG meeting.

[brian-smith-tcril]

We discussed this in the working group meeting last week and the benefits of toml are absolutely worth merging this for!

NOTE: This PR merely adds a way to support TOML for design tokens. It is not a recommendation to switch the entirety of the token system to use toml instead of JSON.

This lines up perfectly with what was discussed in the working group - in the meeting notes we noted:

• Supporting toml for brand package overrides makes a lot of sense
  • Specifically being able to set one.value.nested.quite.deeply without creating a giant empty json structure is a huge benefit.
• We should keep the base tokens in json
  • The style dictionary documentation uses json which makes cross-referencing easier
  • The nested structure is beneficial in with working with all of the tokens and not just a subset

[brian-smith-tcril]

Thank you so much for this! Sorry it took so long to discuss/review it!

[adamstankiewicz]

LGTM

[openedx-webhooks]

@xitij2000 🎉 Your pull request was merged! Please take a moment to answer a two question survey so we can improve your experience in the future.

[openedx-semantic-release-bot]

🎉 This PR is included in version 23.0.0-alpha.2 🎉

The release is available on:

• npm package (@alpha dist-tag)
• GitHub release

Your semantic-release bot 📦🚀

[xitij2000]

Thanks for looking into this and merging it!