Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Alternative concat way several times faster than with_columns() when adding many columns to the df #19503

Closed
2 tasks done
Chuck321123 opened this issue Oct 28, 2024 · 0 comments · Fixed by #19701
Closed
2 tasks done

Alternative concat way several times faster than with_columns() when adding many columns to the df #19503

Chuck321123 opened this issue Oct 28, 2024 · 0 comments · Fixed by #19701
Assignees
@ritchie46
Labels
bug Something isn't working needs triage Awaiting prioritization by a maintainer python Related to Python Polars

Comments

@Chuck321123
Copy link

Chuck321123 commented Oct 28, 2024 •
edited
Loading

Checks

  • I have checked that this issue has not already been reported.
  • I have confirmed this bug exists on the latest version of Polars.

Reproducible example

import polars as pl
import numpy as np

# Set the number of rows and columns
num_rows = 10000
num_columns = 10000

np.random.seed(42)

data1 = {f"col_{i}": np.random.rand(num_rows) for i in range(num_columns)}
df1 = pl.DataFrame(data1)

data2 = {f"feature_{i}": np.random.rand(num_rows) for i in range(num_columns)}
df2=pl.DataFrame(data2)

%timeit df1.with_columns(df2)

%timeit pl.concat([df1,df2], how="horizontal")

Log output

1.28 s ± 31.8 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
1.85 ms ± 57.8 μs per loop (mean ± std. dev. of 7 runs, 100 loops each)

Issue description

So when adding multiple columns, concat is way faster than the normal with_columns() method. Would be nice if these two methods were equally as fast.

Expected behavior

That these functions become equally as fast

Installed versions

--------Version info---------
Polars:              1.12.0
Index type:          UInt32
Platform:            Windows-11-10.0.22631-SP0
Python:              3.13.0 | packaged by Anaconda, Inc. | (main, Oct  7 2024, 21:21:52) [MSC v.1929 64 bit (AMD64)]
LTS CPU:             False

----Optional dependencies----
adbc_driver_manager  <not installed>
altair               <not installed>
cloudpickle          3.1.0
connectorx           <not installed>
deltalake            <not installed>
fastexcel            <not installed>
fsspec               <not installed>
gevent               <not installed>
great_tables         <not installed>
matplotlib           3.9.2
nest_asyncio         1.6.0
numpy                2.1.2
openpyxl             3.1.5
pandas               2.2.3
pyarrow              18.0.0
pydantic             <not installed>
pyiceberg            <not installed>
sqlalchemy           <not installed>
torch                <not installed>
xlsx2csv             <not installed>
xlsxwriter           <not installed>
@Chuck321123 Chuck321123 added bug Something isn't working needs triage Awaiting prioritization by a maintainer python Related to Python Polars labels Oct 28, 2024
@ritchie46 ritchie46 mentioned this issue Nov 8, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ritchie46 ritchie46

Labels
bug Something isn't working needs triage Awaiting prioritization by a maintainer python Related to Python Polars
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

perf: Fix quadratic 'with_columns' behavior pola-rs/polars
2 participants
@ritchie46 @Chuck321123