ExceptionGroup ergonomics in pytest.param(..., marks=xfail(raises=...))

[Zac-HD]

Suppose you have a parameterized test, some params of which are expected to raise (e.g.) IndexError:

return pytest.param(
    ...,
    marks=pytest.mark.xfail(raises=IndexError, strict=True),
)

If your test is async though, you'll actually raise an ExceptionGroup(..., [IndexError]) (maybe with even more nesting), so the tests will still fail. Can we make this more ergonomic? What would a parametrize-aware RaisesGroup-like thing look like?

See also #11538 and #11671.

[jakkdl]

I might be seeing nails everywhere, but I would do

return pytest.param(
    ...,
    marks=pytest.mark.xfail(raises=RaisesGroup(IndexError), strict=True),
)

and afaict the only code doing the matching is

pytest/src/_pytest/skipping.py

Lines 280 to 281 in d9a2fd0

	if raises is not None and not isinstance(call.excinfo.value, raises):
	rep.outcome = "failed"

where it would be straightforward to add

if isinstance(raises, RaisesGroup) and not raises.matches(call.excinfo.value):
  rep.outcome = "failed"
elif raises is not None [...]
  ...

This of course depends on #11671 (or a version that fully implements trio.RaisesGroup functionality).

[jakkdl]

Actually, I don't see why this is about pytest.param at all. If pytest.mark.xfail(raises=...) got support for RaisesGroup (or equivalent) it should Just Work ™️

@pytest.mark.xfail(raises=RaisesGroup(ValueError))
def test_foo():
    raise ExceptionGroup("foo", [ValueError])

#11671 has fallen behind the trio implementation, and is stalled by people wanting to have opinions on the interface design, otherwise I'd resolve this issue in it right away.