Extending the parser for a Python DSL (Snakemake)

[mbhall88]

Snakemake is a workflow management system used very heavily in the biology research community, among others. It is essentially an extension of Python, with some Snakemake-specific syntax. For example

def foo()
    return x + 42

# snakemake-specific syntax
rule bar:
    input: "input.txt"
    output: "output.txt"
    shell:
        """
        echo output > {output}
        """

I am a co-dev and co-maintainer of a tool to format Snakefiles, which essentially just passed the code to black, converting snakemake-specific syntax into python analogs, and then converting back after the formatting.

However, this parsing is getting harder and harder to maintain as we just have a hacky parser. What I would ultimately like to do is extend and existing python parser to include the snakemake syntax keywords such as rule, include, etc. This would allow for much easier parsing and bug fixing. We currently have lots of hacky fixes which are building up into a headache.

I noticed that ruff has essentially implemented a python parser from scratch and was wondering how simple you think it would be for me to create a snakemake parser that extends this? Any direction/advice you could provide would be so helpful. I have played around with lark but had a hard time trying to get it to work.

If it were possible to extend ruff's parser, would that then make it easy to plug into ruff's formatting and we could then just pass the snakemake code through to ruff for formatting instead of black?

Sorry I know this has been a bit rambling, I hope it is somewhat clear what I am asking...

[MichaReiser]

Hi @mbhall88

I am a co-dev and co-maintainer of a tool to format Snakefiles, which essentially just passed the code to black, converting snakemake-specific syntax into python analogs, and then converting back after the formatting.

I'm always impressed by what people build. Nice that this works :)

I noticed that ruff has essentially implemented a python parser from scratch and was wondering how simple you think it would be for me to create a snakemake parser that extends this?

Forking Ruff's parser and extending it to support the snakefile syntax shouldn't be too much work. You probably want to add a few custom statement types in

ruff/crates/ruff_python_parser/src/parser/statement.rs

Lines 112 to 146 in 5109b50

	pub(super) fn parse_statement(&mut self) -> Stmt {
	let start = self.node_start();
	match self.current_token_kind() {
	TokenKind::If => Stmt::If(self.parse_if_statement()),
	TokenKind::For => Stmt::For(self.parse_for_statement(start)),
	TokenKind::While => Stmt::While(self.parse_while_statement()),
	TokenKind::Def => Stmt::FunctionDef(self.parse_function_definition(vec![], start)),
	TokenKind::Class => Stmt::ClassDef(self.parse_class_definition(vec![], start)),
	TokenKind::Try => Stmt::Try(self.parse_try_statement()),
	TokenKind::With => Stmt::With(self.parse_with_statement(start)),
	TokenKind::At => self.parse_decorators(),
	TokenKind::Async => self.parse_async_statement(),
	token => {
	if token == TokenKind::Match {
	// Match is considered a soft keyword, so we will treat it as an identifier if
	// it's followed by an unexpected token.
	match self.classify_match_token() {
	MatchTokenKind::Keyword => {
	return Stmt::Match(self.parse_match_statement());
	}
	MatchTokenKind::KeywordOrIdentifier => {
	if let Some(match_stmt) = self.try_parse_match_statement() {
	return Stmt::Match(match_stmt);
	}
	}
	MatchTokenKind::Identifier => {}
	}
	}
	self.parse_single_simple_statement()
	}
	}
	}

I don't think the Ruff parser uses any error recovery today that would be incompatible or cause problems with the syntax I see above but there's a risk that we might add error recovery that would result in skipping valid snakemake syntax (but only if there was some invalid syntax before it).

Forking a parser comes with its own challenge if you want to stay up to date with new Python syntax, and we don't make any stability guarantees regarding our parser/AST API.

I suspect what you would want most is for Ruff to add a snakemake dialect to its parser. That's something we would have to consider thoroughly because I used to maintain an error-recoverable JavaScript/TypeScript parser and supporting multiple dialects adds a lot of complexity. The main issue I see with snakemake is that Python could introduce new syntax that leads to ambiguity with snakemake in the future and that would put Ruff in a rough spot ;) This problem did not exist with TypeScript/JavaScript because the JS standard committee agreed not to introduce any new syntax that would break TypeScript.

I don't think snakemake has enough community today to warrant the added complexity on our side.

[mbhall88]

Thank you for the detailed response!

I certainly would not advocate adding snakemake dialect to your parser. I guess I was hoping there was a way I could use the ruff parser as a dependency and then somehow extend it. This way, as you say, I get all the Python updates you implement for free. However this sounds like it might not be possible? This was sort of why I had tried something like lark as it should be possible to extend an existing grammar. Maybe I need to debug through issues with them…

trying to rack my brains for a solution leveraging the trait system but can’t seem to think of anything at the moment (though I would consider my rust intermediate at best).

[MichaReiser]

No, I don't see a way of making our parser extensible in the sense that it is possible to plug in additional syntax support because it is all hand written and adding new syntax may also require changes to other parse rules (e.g. rule bar: requires a lookahead when seeing an identifier to decide if this is an expression statement or a snakefiles statement)