perf(duckdb): speedup timestamp conversion by avoiding conversion to object #9556
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
cpcloud
added
performance
|
Still about 5x left that we're losing, but that can be done in a follow up. |
|
Ok found a few of the remaining Xs in a bug 🐞 |
|
Added a benchmark, which actually times out on |
cpcloud
force-pushed
the
duckdb-timestamp-perf
branch
from
7a2c13d
to
1bf015b
Compare
cpcloud
force-pushed
the
duckdb-timestamp-perf
branch
from
1bf015b
to
2757636
Compare
cpcloud
commented
Jul 12, 2024
| @@ -200,8 +200,10 @@ def convert_Boolean(cls, s, dtype, pandas_type): | |||
|
|
|||
| @classmethod | |||
| def convert_Timestamp(cls, s, dtype, pandas_type): | |||
| if isinstance(dtype, pd.DatetimeTZDtype): | |||
There was a problem hiding this comment.
This check was always False, because dtype is always an Ibis type.
cpcloud
force-pushed
the
duckdb-timestamp-perf
branch
4 times, most recently
from
7fdbba2
to
532c76e
Compare
cpcloud
force-pushed
the
duckdb-timestamp-perf
branch
from
532c76e
to
b7e6deb
Compare
gforsyth
approved these changes
Jul 13, 2024
There was a problem hiding this comment.
This looks good to me, and I think the shift from ns to us precision is a non-issue here, since DuckDB doesn't support ns timestamps in the first-place, so it's mostly a very small change in the representation of the results, but the values aren't fundamentally changing.
14 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Redo of #9554. The slow bit here was object conversion of timestamps, which seemed to only be hit by this one test.