Performance Issue - Polars DataFrame Behaves Depending on Data Source (Which Shouldn't)

[hanepudding]

Checks

• I have checked that this issue has not already been reported.
• I have confirmed this bug exists on the latest version of Polars.

Reproducible example

Polars DataFrame Behaves Depending on Data Source (Which Shouldn't)

Workarounds found, but need deeper investigation.
This behavior triggered when I am dealing with really large dataframe(in my case 28 million rows) and tons of .csv files, so hopefully I can describe the problem clearly without having to upload my dataset.

What happened

result_problematic.equals(result_good) #True, but...
result_problematic.write_parquet("problematic.parquet") # 1m 38.1s
result_good.write_parquet("good.parquet") # 6.1s

_test1 = pl.read_parquet("problematic.parquet") # 3.7s
_test2 = pl.read_parquet("good.parquet") # 0.6s
_test1.equals(_test2) # True

How to reproduce this problematic .parquet file:

import polars as pl
import datetime as dt

# old data source, shape: (27_796_178, 14)
df_old = pl.read_parquet("some_file.parquet")

# problematic dataframe, shape: (1_144_920, 14)
df_to_append_problematic = pl.read_csv("minato/*/*.csv").with_columns(
    pl.col("datetime","trading_date").str.to_datetime(time_unit = "ns"),
).filter(
    pl.col("trading_date") > dt.datetime(2024,8,6)
)

# somehow good dataframe, also in shape: (1_144_920, 14)
pl.read_csv("minato/*/*.csv").write_parquet("just_csvs.parquet")  # ???
df_to_append_good = pl.read_parquet("just_csvs.parquet").with_columns(
    pl.col("datetime","trading_date").str.to_datetime(time_unit = "ns"),
).filter(
    pl.col("trading_date") > dt.datetime(2024,8,6)
)

result_problematic = pl.concat([df_old, df_to_append_problematic])
result_good = pl.concat([df_old, df_to_append_good])

Workaround:

pl.read_csv("minato/*/*.csv").write_parquet("just_csvs.parquet")
df_good = pl.read_parquet("just_csvs.parquet")
# simply pl.read_csv("minato/*/*.csv") will return a BAD guy

Log output

No response

Issue description

See code section for details.
Tl;dr,

result_problematic.equals(result_good) #True, but...
result_problematic.write_parquet("problematic.parquet") # 1m 38.1s !
result_good.write_parquet("good.parquet") # 6.1s

_test1 = pl.read_parquet("problematic.parquet") # 3.7s !
_test2 = pl.read_parquet("good.parquet") # 0.6s
_test1.equals(_test2) # True

Expected behavior

DataFrame's behavior should be independent from data source once loaded to python.

Installed versions

--------Version info---------
Polars:              1.12.0
Index type:          UInt32
Platform:            Windows-11-10.0.26100-SP0
Python:              3.13.0 (tags/v3.13.0:60403a5, Oct  7 2024, 09:38:07) [MSC v.1941 64 bit (AMD64)]
LTS CPU:             False

----Optional dependencies----
adbc_driver_manager  <not installed>
altair               <not installed>
cloudpickle          <not installed>
connectorx           <not installed>
deltalake            <not installed>
fastexcel            <not installed>
fsspec               <not installed>
gevent               <not installed>
great_tables         <not installed>
matplotlib           3.9.2
nest_asyncio         1.6.0
numpy                2.1.3
openpyxl             3.1.5
pandas               2.2.3
pyarrow              18.0.0
pydantic             <not installed>
pyiceberg            <not installed>
sqlalchemy           2.0.36
torch                <not installed>
xlsx2csv             <not installed>
xlsxwriter           <not installed>

[J-Meyers]

Does rechunking the csv dataframe prior to operations help?