Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Option to preserve comments in the AST #1967

Open
pawamoy opened this issue Apr 21, 2024 · 5 comments · May be fixed by #2037
Open

Option to preserve comments in the AST #1967

pawamoy opened this issue Apr 21, 2024 · 5 comments · May be fixed by #2037

Comments

@pawamoy
Copy link
Contributor

pawamoy commented Apr 21, 2024

Hello, thanks for this wonderful piece of software 🙂

I'm maintaining mkdocstrings, which uses Jinja templates to render API documentation for different languages. Recently I had this idea of doing the same thing (collecting and rendering API docs) for... Jinja templates themselves 😛 It's just an idea for now, but I'll be tracking my progress here: mkdocstrings/mkdocstrings#661.

I see that Jinja provides the Environment.parse method, which builds an AST of the Jinja source: this is fantastic. I'll be able to visit such trees to build useful information and store it in structures that make it easy to render them in other Jinja templates.

However... Jinja comments are not preserved 😢 This #719 (comment) says that they're probably discarded by the lexer.

I can already extract lots of useful information from templates, but I had in mind to allow template writers to use Jinja comments to document variables, filters, blocks, macros, whatever, directly in the template, just like we document attributes, functions, classes, etc., directly within Python code.

For this I would need an option to tell Environment.parse to preserve the comments in the generated AST. Do you think that is feasible? I can definitely free up some of my time to work on this.

In any case, what do you think of this idea of auto-documentation for Jinja templates? I think that would be a wonderful tool in the Jinja ecosystem 😄 I personally have lots of Jinja templates that I would document this way 🙂

@pawamoy
Copy link
Contributor Author

pawamoy commented Apr 21, 2024

I'm happy to maintain a patched lexer/parser in an external package if you don't feel comfortable implementing this option in core. I'd greatly appreciate some guidance on how to change the lexer/parser to preserve comments 🙂

I suppose I must somehow stop ignoring comments in the lexer, and also create a new Comment node that could be added to the AST.

@pawamoy
Copy link
Contributor Author

pawamoy commented Apr 21, 2024

OK I managed to get something working 🙂

Basically:

  • remove comment tokens from ignored set
  • add a comment node
  • handle comment tokens in subparse
ignored_tokens = frozenset(
    [
        # TOKEN_COMMENT_BEGIN,
        # TOKEN_COMMENT,
        # TOKEN_COMMENT_END,
        TOKEN_WHITESPACE,
        TOKEN_LINECOMMENT_BEGIN,
        TOKEN_LINECOMMENT_END,
        TOKEN_LINECOMMENT,
    ]
)
class Comment(Stmt):
    """A template comment."""

    fields = ("data",)
    data: str
# in Parser.subparse
elif token.type == "comment_begin":
    flush_data()
    next(self.stream)
    body.append(nodes.Comment(next(self.stream).value))
    self.stream.expect("comment_end")

Printing the AST gives something like:

Template(body=[Comment(data=' Admonitions template. '), ...])

@pawamoy
Copy link
Contributor Author

pawamoy commented Oct 4, 2024

Bump 🙂 Would maintainers be interested in such functionality? I think it would allow nice developments in the Jinja ecosystem 🙂

@davidism
Copy link
Member

davidism commented Oct 4, 2024

Yeah, I'm open for reviewing a PR for this, but I just have very limited time to focus on new features for Jinja right now.

@pawamoy
Copy link
Contributor Author

pawamoy commented Oct 7, 2024

Fantastic ☺️ No problem, I'll do my best to send the most reviewable PR and most easily maintained code 😋

@pawamoy pawamoy linked a pull request Oct 10, 2024 that will close this issue
4 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants