explore: alternative CSS selector parsers

[flavorjones]

The CSS selector parser we have is complex, and selector parsing is really a separable concern from Nokogiri proper. It would be nice if we were able to use an existing parser.

(Side note: the generation of XPath from the CSS is a Nokogiri concern, though, since the generated xpath query is often tightly coupled to the version of libxml or the C extension. Perhaps we can spin this off as a separate gem/concern at some point, but it would need to be pluggable to do nokogiri-specific xpath things and I don't feel like that's worth the effort right now.)

Some things to look at that generate ASTs for CSS:

• https://github.com/rgrove/crass
• https://github.com/ruby-syntax-tree/syntax_tree-css

I'd also like to fix some outstanding bugs in the current implementation:

• [bug] CSS nth-of-type and friends does not match browser behavior #3238
• :eq css selector should index from zero #652
• nokogiri's implementation of :first is different than jQuery's. #628
• CSS foo~:nth-child(2) gives incorrect XPath #707

though, note that the behavior changes to fix these bugs probably justify a 2.0 major release, because it's going to break existing apps.

And then I think we can also introduce some new features:

• CSS case insensitive attribute selector operator #2036
• :scope pseudoclass not supported #1889
• [Nokogiri should accept multiple selectors as arguments to the :not CSS pseudo-class #3207

[flavorjones]

I've looked at Crass a bit yesterday and today, but it's not returning a fine-enough-grained AST for selectors; we'd have to use the tokens and implement some sort of parser to make it work.

Looking at syntax_tree-css, it's incomplete but is definitely a well-formed AST for selectors. I've started kicking the tires and making basic improvements to see how far I can take it.

[flavorjones]

PRs against syntax_tree-css to get it to where we can integrate it:

• Improve selectors, fix complex selector parsing issues ruby-syntax-tree/syntax_tree-css#58
• additional CSS selector formatting ruby-syntax-tree/syntax_tree-css#59
• refactor: Improve how whitespace is consumed in complex selectors ruby-syntax-tree/syntax_tree-css#60
• Include an explicit Combinator in the AST for implicit descendant ruby-syntax-tree/syntax_tree-css#61
• Parse pseudo-class function arguments ruby-syntax-tree/syntax_tree-css#62

PRs against nokogiri with this goal in mind:

• Tidy the CSS parser API #3218

[flavorjones]

More PRs against Nokogiri (broke up #3218 as too-big):

• deprecate: Nokogiri::CSS.parse #3222
• fix: css pseudo-class error raised by XPathVisitor #3223
• improve documentation for CSS.xpath_for, and reorganize css tests #3224
• move prefix and ns into XPathVisitor #3225
• extract CSS selector cache into CSS::SelectorCache #3226
• reorganize css tests (part 2) #3227

[flavorjones]

I've got a branch where the hand-written parser work is progressing, in case anybody wants to follow along: https://github.com/sparklemotion/nokogiri/tree/2560-start-custom-css-parser