0.15 is extrmemely slow and fails on lots of tests

[samuelcolvin]

I'm not really sure what's going on, but pydantic/pydantic-ai#657 should demonstrate the problem - all it's doing is upgrading from 0.14 to 0.15 and everything starts breaking.

It's possible that the tests on IsNow are failing just because inline-snapshot comparison is so slow that the datetime's being compared to are no longer close to "now".

It's worth saying that due to #174 I'm currently having to switch back and fourth between diffferent versions to run all tests and generate snapshots! @15r10nk I'd really appreciate some help on this.

I've tried the latest release (v0.18 I think) and i'm see the exact same behaviour as 0.15.

[15r10nk]

I will look into it

[samuelcolvin]

thank you.

[15r10nk]

I made some progress.

The tests in this branch pydantic/pydantic-ai#658 are working for cpython>=3.11.

I found the following problems.

ModelResponse.from_text

I assume that you (or someone else) changed the generated snapshot code to make use of ModelResponse.from_text. inline-snapshot tried to inspect the code and failed because the dataclass had no content attribute for example. This should not lead to a crash of inline-snapshot.

I fixed it for now by adding Is(...) which tells inline-snapshot that this code is created
by the user and that it should not touch it.

    assert result.all_messages() == snapshot(
        [
            ModelRequest(parts=[UserPromptPart(content='Hello', timestamp=IsNow(tz=timezone.utc))]),
            Is(ModelResponse.from_text(content='Hello world', timestamp=IsNow(tz=timezone.utc))),
            ModelRequest(parts=[UserPromptPart(content='Hello', timestamp=IsNow(tz=timezone.utc))]),
            Is(ModelResponse.from_text(content='Hello world', timestamp=IsNow(tz=timezone.utc))),
        ]
    )

I would also like to know why you changed this code. Was there some problem with the generated code? I think about adding a way to tell inline-snapshot that it should use alternative class methods to construct a object. This would allow inline-snapshot to fix the value if it changes later. But this has nothing to do with this bug and is maybe something for a new feature in the future.

IsNow

You where right IsNow slows inline-snapshot down. I currently don't know why this happen, but I found a interesting work around.

import dirty_equals

OldIsNow = dirty_equals.IsNow


def newIsNow(*a: Any, **ka: Any):
    # this has a big effect on the test runtime
    # tests are passing in both cases

    # ~ 4min
    # (the delta is required to make the tests work)
    # return OldIsNow(*a,**ka,delta=50)

    # 16 sec
    return dirty_equals.AnyThing()


dirty_equals.IsNow = newIsNow

I will try to figure the difference between AnyThing and IsNow out.

[15r10nk]

@samuelcolvin I found the reason for the slow tests.
Do you remember this samuelcolvin/dirty-equals#105 (comment)? Turns out that this also affects isinstance/subclass checks.
I also almost forgot it because I fixed inline-snapshot to prevent the crashes related to this issue.

Can you please create a new dirty-equals release?

[samuelcolvin]

Sorry about that, I'll do a release asap.

[samuelcolvin]

Regarding ModelResponse.from_text, happy to switch to standard init, no idea why it's done that way, I suspect it was changed in a big refactor.

[samuelcolvin]

dirty-equals v0.9 released.

[15r10nk]

Regarding ModelResponse.from_text, happy to switch to standard init, no idea why it's done that way, I suspect it was changed in a big refactor.

I have reverted it to the representation used by inline-snapshot. The tests are working again.
I will soon release a new inline-snapshot version.

I also found it very cumbersome to change the timestamps back to IsNow, which inspired me to #177. Let me know if you have similar issues when you are working with inline-snapshot.