inline links to *.md files no longer work, starting with v0.19.0

[jclerman]

Describe the bug

context
When I create the following Markdown file:

my-docs.md

## Overview

Here I would like to refer you to the [details](details.md).

(there is another Markdown file details.md, in the same directory).

expectation
I expected a link to be created, leading to the 2nd file (details.md).

I verified that I am following the guidance in the documentation, here, specifically:

[text](relative/path/myfile.md) work as previously, to link to files, but they can also be relative to source directory: [text](/path/from/srcdir/myfile.md). You can also use <project:file.md>

bug
But instead I get an error when I try to build my documentation:

Warning, treated as error:
/Users/***/my-docs.md:3:'myst' cross-reference target not found: 'details.md' [myst.xref_missing]

problem
I can't figure out how to insert links between Markdown files using myst-parser >=0.19.

Reproduce the bug

See summary above. I invoke sphinx via:

sphinx-build -M html "." "_build" -W

List your environment

Python 3.8.7
MacOS 12.6.3 (Monterey)

sphinx 5.3.0  # also happens with 6.x
myst-parser 1.0.0  # also happens with 0.19.0

[jclerman]

After much experimentation, I found that omitting the .md from affected links "solves" the problem - in that myst-parser will now generate the correct links (presumably because they point to the HTML file generated from the original markdown target file). However, that breaks the same links for other markdown viewers (e.g. github, or PyCharm IDE) - basically, the links become nonstandard, whereas in myst-parser 0.18.x, the same links were usable by both markdown viewers, and in the generated sphinx docs.

Is there a way to preserve the portability we had in 0.18? Was this change intentional?

[chrisjsewell]

Hi @jclerman,

These links are certainly working in the tests and in the documented examples: https://myst-parser.readthedocs.io/en/latest/syntax/cross-referencing.html#examples

Perhaps you could link to your project, where this is appears to be not working, or provide a minimal working example?

I found that omitting the .md from affected links "solves" the problem

yeh this is more of a workaround than a solution 😅

[chrisjsewell]

Did you also try <project:details.md>, what does that do?

[jclerman]

Hi @chrisjsewell, this is a private project so I can't simply link to it, unfortunately. I will see about creating a minimal example that shows the issue. In the meantime (or maybe instead?) here is my Sphinx conf.py - do you see anything in there that might be causing this issue?

"""Configuration file for the Sphinx documentation builder."""

import os
import sys
from typing import Any, List, MutableMapping

import toml

sys.path.insert(0, os.path.abspath("../"))
DOCS_ROOT_DIR = os.path.dirname(os.path.abspath(__file__))

# -- Project information -----------------------------------------------------
pyproject_toml_path = os.path.join(DOCS_ROOT_DIR, "..", "pyproject.toml")
pyproject_toml: MutableMapping[str, Any] = toml.load(pyproject_toml_path)

project = pyproject_toml["tool"]["poetry"]["name"]
copyright = "2021, Jeff Lerman"
author = "Jeff Lerman"

# The full version, including alpha/beta/rc tags
release = pyproject_toml["tool"]["poetry"]["version"]


# -- General configuration ---------------------------------------------------

# Add any Sphinx extension module names here, as strings. They can be
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
# ones.
# See https://brendanhasz.github.io/2019/01/05/sphinx.html for some pointers.
extensions = [
    "sphinx.ext.autodoc",
    "sphinx.ext.napoleon",
    "sphinx_rtd_theme",
    "myst_parser",
    "sphinx.ext.todo",
    "sphinx.ext.autosectionlabel",
]
autodoc_preserve_defaults = True
napoleon_google_docstring = True  # Enable parsing of Google-style pydocs.
napoleon_numpy_docstring = True  # Enable parsing of NumPy-style pydocs.
napoleon_use_ivar = True  # to correctly handle Attributes header in class pydocs
# Prefix document path to section labels, to use:
# `path/to/file:heading` instead of just `heading`
autosectionlabel_prefix_document = True

# Add any paths that contain templates here, relative to this directory.
templates_path = ["_templates"]

# List of patterns, relative to source directory, that match files and
# directories to ignore when looking for source files.
# This pattern also affects html_static_path and html_extra_path.
exclude_patterns = ["_build", "Thumbs.db", ".DS_Store"]

# List of warnings to suppress.
suppress_warnings = ["myst.header"]

# -- Options for HTML output -------------------------------------------------

# The theme to use for HTML and HTML Help pages.  See the documentation for
# a list of builtin themes.
#
html_theme = "sphinx_rtd_theme"

# Add any paths that contain custom static files (such as style sheets) here,
# relative to this directory. They are copied after the builtin static files,
# so a file named "default.css" will overwrite the builtin "default.css".
html_static_path: List[str] = []

[jclerman]

Apologies for posting it this way - but attached is a (very small) tgz file containing a small (may not quite be minimal) example demonstrating the issue.

myst-issue.tgz

To use (one way to do it):

• unpack the archive (tar -xzvf filename.tgz)
• from the root of the archive, do poetry shell
• poetry install
• cd docs
• make html

[n-peugnet]

I have a similar problem, since v0.19 but only in a multi-language Sphinx project, when compiling a translated version and with links in the form [text](path/to/doc.md#auto-header-link).

Maybe I should open a separated issue for this?

[jclerman]

Hi @chrisjsewell , I don't mean to nag at all - but not sure if you still need more info on this (it's still tagged that way) or if the info I provided was sufficient to reproduce the issue?

[NSeydoux]

If it can help, inrupt/solid-client-notifications-js#552 reproduces the issue, and even if this specific PR upgrades myst-parser to 1.0.0, the problem also starts with 0.19.0.

[agoose77]

I'm a new maintainer here, getting up to speed. Without a deep knowledge of what's going on here 0.19 was the first release IIRC to include new reference/link resolution syntax. It might be that we don't opt-in to that in all cases, breaking existing usage. I'll take a look at this, and report back!

[chrisjsewell]

example demonstrating the issue

Heya @jclerman, to paraphrase your example, you have, files:

docs/
  index.rst
  general/
    overview.md
    algorithm.md

and then:

docs/index.rst

.. include:: general/overview.md
   :parser: myst_parser.sphinx_

docs/general/overview.md

Here is the [algorithm](../general/algorithm.md)

The include directive effectively works somewhat like a copy/paste into the parent document, so in essence you have:

docs/index.md

Here is the [algorithm](../general/algorithm.md).

Hopefully, you can see here that your relative document link is now incorrectly resolved to general/algorithm.md rather than docs/general/algorithm.md.
This is expected (see e.g. sphinx-doc/sphinx#6805 (comment)).

If you are including documents into other documents, then relative document paths must either be relative to the including/parent document, or you can use absolute paths, starting with /, which are always resolved relative to the source folder, e.g. [algorithm](/general/algorithm.md) will always resolve to docs/general/algorithm.md independent of the actual document it is in.

Does make sense?

[chrisjsewell]

@n-peugnet and @NSeydoux if your issue is different to what I describe above, then yeh it would probably be best if you open a separate issue, ideally also with a similar reporducible example thanks

[jclerman]

Hi @chrisjsewell, thanks. Using absolute paths does work, but I would really like to be able to use paths that are relative to the *.md file in which they are written - or at least some way to have absolute paths that use the repository root as their root, so that xrefs continue to work with other parsers (that don't consider /docs to be the root) too. My project's top-level README.md links to some of these files, which means that a reader in the GitHub GUI might also click to them and read them - and I'd like for their links not to be dead in that context. With myst-parser 0.18.x, that workflow was supported.

(my original response here included a complaint about continued problems with relative xrefs, but I now understand that those are explained by your explanation above)

It occurs to me that some of this confusion could be resolved elegantly if the xrefs were, during parsing, resolved (to absolute paths) before any include directives were processed.

[jclerman]

One more thought on this: It seems like myst-parser 18.x must have supported relative paths in a way that tolerated going "above" the root, since in my example, I can have a link ../general/algorithm.md from overview.md, even though overview.md is included from index.rst. From index.rst, .. in the relative link copied from overview.md points to a directory one level above the root. That part must be silently tolerated/ignored, so that effectively, just general/algorithm.md is used. That has the effect of allowing that relative link to be resolved (as intended) by myst-parser, and also to be resolved by something (like GitHub) that reads the relative link with respect to the original location of overview.md.

The tolerant behavior must have been removed from myst-parser in the switch to 19.x - if it were restored, my use-case would be addressed (and so would anyone's who wanted to have links that are interpretable by both Sphinx/myst-parser after an include, and other tools that need to resolve the links from their original locations).

[iva2k]

I'm a new user trying out MyST with sphinx in an existing project. I had high hopes for it especially for the promised links translations. But I encountered exactly the same problem with links relative to the .md file they are located in not working as promised.

Looks like there was a regression in v0.19, and now it is at v2.0 with the same problem remaining, and no activity here for good 10 months.

Is there any plan to fix this issue? What would it take?

Is there any workaround that won't break .md files for Github (Besides downgrading to v0.18)?

[Snoopy1866]

@chrisjsewell if try to include the .md file and it include a relative path which links to another .md file in the parent folder, WARNING still happends.

For example, I have files:

docs/
  index.md
README.md
README-en.md

in index.md:

```{include} ../README.md

```

in README.md:

[English](README-en.md)

run docs/make html, output warning:

C:\Users\wtwang\Documents\GitHub\PyStatPower\PyStatPower\README.md:18: WARNING: 'myst' cross-reference target not found: 'README-en.md' [myst.xref_missing]

It seems that there's no solution in this scenario, whether I use an absolute path or a relative path, since the source folder is consistently set to docs.