A transition towards use of MyST instead of rST?

[consideRatio]

I've never really become comfortable with rST, but in the past Markdown wasn't a good option for documentation in projects because it lacked the ability to reference other files etc in a good way. This has changed with MyST though.

MyST with Sphinx is very similar to using rST with Sphinx. The key difference is the use of Markdown syntax instead of rST syntax, where the Markdown syntax has been complemented with rST features described as roles and directives. A key point with the use of MyST, is that Sphinx will be agnostic to if MyST has parsed markdown files into the format that Sphinx processes, or if it was an rST parser. Due to this, as @TomAugspurger pointed out:

Things like the dask-sphinx-theme will still work just fine, regardless of whether the source is in rst or markdown.

---

In the JupyterHub org, we have a few projects that have successfully transitioned MyST markdown with Sphinx instead of rST with Sphinx, and it has been a great success in my mind. For me, it has become easier to maintain docs in those projects and I believe it helps our Jupyter ecosystem contributors contribute to docs a bit easier as most of anyone working with Jupyter knows Markdown already.

By opening this issue, I hope to spark a discussion about the use of rST, MyST, or potentially other alternatives. If there is support for it, I'd also be happy to put in work to pilot a transition from rST to MyST in the dask/dask-gateway project.

Related

• JupyterHub Helm chart use of MyST
• MyST Markdown reference
• Transition rST to MyST? dask-gateway#427

[mrocklin]

cc @jacobtomlinson @jsignell

[jacobtomlinson]

MyST is awesome and I would be excited to see it used more. I am generally in favour of this, but have some conflicted feelings too.

I've never really become comfortable with rST

I felt this way for a long time. Its only in the last year or so that I finally feel equally comfortable between RST and Markdown.

I think for many folks (who aren't full-time OSS Python maintainers) Markdown feels more comfortable as it is used in so many other places.

I hope to spark a discussion about the use of rST, MyST, or potentially other alternatives.

I have worked on other projects that use Markdown for documentation using recommonmark. It always felt clunky and bolted on and I've never been 100% happy with it.

I've been watching MyST with great excitement for a while as it seems to solve all the problems I had with recommonmark and it seems that even that project has been deprecated in favour of MyST.

A transition towards use of MyST instead of rST?

Personally I'm pretty on board with using MyST in new projects and greenfield development. However, I'm not convinced that migrating existing rST docs is worth the effort.

I think the only benefit of a migration would be that docs would be slightly easier for novice users to contribute to.

My other slight concern is that this would mean docstrings and docs would be written in two different markup languages. It's not a huge issue but it still means contributors need to know how to write rST to some level.

[jrbourbeau]

Thanks for bringing this up @consideRatio!

Personally I'm pretty on board with using MyST in new projects and greenfield development. However, I'm not convinced that migrating existing rST docs is worth the effort.

Perhaps @consideRatio could elaborate on what this migration looked like for projects in the JupyterHub org. For example, do all existing .rst files need to be converted to .md?

[consideRatio]

Thanks for such quick feedback on this everyone!

Perhaps @consideRatio could elaborate on what this migration looked like for projects in the JupyterHub org. For example, do all existing .rst files need to be converted to .md?

The process can be gradual and doesn't have to force rST files to be transformed. With MyST, you can have .md files next to .rST.

The process can be described in a few steps I think.

1. The myst-parser is declared to be used when sphinx observes .md files.
   Example of this: switching to myst markdown in docs jupyterhub/zero-to-jupyterhub-k8s#1628
2. Any individual .rST file can now be replaced with a .md file if wanted
   Example of this using rst-to-myst: docs: 100% MyST Markdown jupyterhub/zero-to-jupyterhub-k8s#1974
3. The conversion process can make use of:
   • A tool like rst-to-myst to make the initial transformation of rST to Markdown.
     When I did this latest a while back, the tool was an alpha state but seem more mature now.
   • A tool like prettier can be used to autoformat markdown files as well as python/yaml code within code blocks.
     I have been very happy about the use of prettier for Markdown and YAML across the JupyterHub org so far, but with regards to this transformation it's absolutely not needed but perhaps there are some tweaks needed in the formatting that could be solved by prettier.

Personally I'm pretty on board with using MyST in new projects and greenfield development. However, I'm not convinced that migrating existing rST docs is worth the effort.

I've enjoyed the implicit JupyterHub policy along the lines that whoever wants to make such conversions are blessed to do so if they put in the work, but that there is no common push across the org to transform it all. For me, this has enabled me to transform projects I work often in 100% to MyST which I've considered worth it personally. Practically, @choldgraf introduced us to MyST and the jupyterhub/zero-to-jupyterhub-k8s project piloted its use. After a while, other projects followed and some remains untouched still.

My other slight concern is that this would mean docstrings and docs would be written in two different markup languages. It's not a huge issue but it still means contributors need to know how to write rST to some level.

Oh, yeah hmmm, docstrings from .py files may need to be written systematically in one language per project I think. I don't know so well how the autodoc that extracts docstrings and helps Sphinx render it works in detail and if one could signal if one want to parse a docstring as Markdown or rST.

[choldgraf]

A quick thought on API docs - I believe the best way to use this is to just use an "eval_rst" directive for API docs. Then it will parse the contents of docstrings as rst as expected.
Unfortunately IIRC the autodoc extensions all hard code assumptions about rst (see sphinx-doc/sphinx#8018 (comment) for example)

I will also note (as an admittedly biased perspective) that I've found maintaining markdown documentation to be just much "nicer" than maintaining rst. I dunno how to put my finger on it, but it goes beyond just "easier for beginners"

[choldgraf]

(I just realized I might start a flame war about rst vs md, i apologize for that... These are just my perspectives not objective truth!)

[dhirschfeld]

docstrings and docs would be written in two different markup languages.

For me, it's very rare that I use any sphinx roles in my (numpydoc) docstrings (if I do, it's usually .. math::).

What does get me is the use of double backticks to get code formatting which is different to markdown (and therefore surprising to most). At least that concern can be addressed by using the literal default role

[jacobtomlinson]

I think the TLDR for my feeling are, a 100% jump to MyST would make things much better, but a 50/60/70/80% jump to MyST would make things worse. And docstrings/autodoc hold us back from 100%.

---

I'm super on board with "markdown for all the things!" as a general life goal. My worries are mainly based on current tooling and conventions.

I wasn't aware of rst-to-myst. That looks awesome! I retract my comment about migrations not being worth the effort.

However.

In projects like dask-cloudprovider there is a lot of documentation that lives in the docstring with narrative and supporting documentation written in rST in the Sphinx docs. It makes heavy use of autodoc.

If I put the AWS page through rst-to-myst I get a lot of eval_rst sections. See gist. If you look at the docstring for EC2Cluster which is used by autodoc in that file it contains much of the content and has many rST directives and features in it.

This kind of situation was why I wasn't super happy with recommonmark in projects like opsdroid. You have a bunch of rST still within your code, and then some markdown in the docs dir which is also littered with rST in eval_rst blocks. It would be much cleaner/readable/maintainable to stick to rST, despite markdown being more pleasant to write.

If we could use MyST in docstrings and it worked with autodoc I would be 100% on board. But I feel like a half way move like this makes things worse.

Core projects like dask/dask and dask/distributed have way more narrative documentation and the API docs are just a small section, so in those situations maybe it would be worth the shift to markdown. I dunno.

[dhirschfeld]

As a user of mkdocs-material the biggest pain point is api docs and, from my investigation, for markdown there are no api doc tools that are as mature as those for sphinx. I believe the current contender is:

• https://github.com/mkdocstrings/mkdocstrings

They've struggled with similar questions over in httpx so the discussion there is probably worth a read:

• On Sphinx vs. MkDocs encode/httpx#1220