fix: ignore crc files when checking if provided path correspond to a valid delta table

[guillotjulien]

Description

The current implementation of is_delta_table_location only check the first file in the _delta_log folder to check for the presence of a commit or checkpoint file. This creates issue when checking tables created using the Spark driver since .crc files are created and appear before commit files in the directory.

This MR modifies the behavior of is_delta_table_location so that files are checked in a loop. .crc files are ignored and the loop execution stops as soon as a commit, checkpoint or invalid file is found.

Related Issue(s)

• closes is_deltatable() returns False for tables in S3 created by delta-spark #3115

[codecov]

Codecov Report

Attention: Patch coverage is 89.33333% with 8 lines in your changes missing coverage. Please review.

Project coverage is 72.26%. Comparing base (c56d6c0) to head (27b0151).
Report is 2 commits behind head on main.

Files with missing lines	Patch %	Lines
crates/core/src/logstore/mod.rs	88.57%	2 Missing and 6 partials ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #3122   +/-   ##
=======================================
  Coverage   72.26%   72.26%           
=======================================
  Files         134      134           
  Lines       42903    42973   +70     
  Branches    42903    42973   +70     
=======================================
+ Hits        31003    31056   +53     
- Misses       9923     9927    +4     
- Partials     1977     1990   +13     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[ion-elgreco]

These are valid files?

00001.crc.crc
00001.json.crc

[guillotjulien]

Not from what I observed. The only .crc file generated without . prefix follow the ^\d+\.crc$ pattern. We can further restrict the regex like this: ^(\.\d+(\.crc|\.json)|\d+)\.crc$.

It should cover every case:

[ion-elgreco]

Why isn't it just r"^\d+\.crc$"?

I have not seen .json.crc or .crc.crc before tbh

[guillotjulien]

Those get generated when I write a delta table locally using Spark 3.5.4 and delta-spark 3.3.0. iirc I also saw them with Spark 3.2 / delta-spark 2.3.0, so I assume they're legit?

pyspark --packages io.delta:delta-spark_2.12:3.3.0 \
  --conf "spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension" \
  --conf "spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog"


> spark.sql("select 1 as id, 'value' as value").write.format("delta").save("./test/resources/test-table")

[ion-elgreco]

I guess :P I just wanted some confirmation on this, because I hadn't seen them before

[ion-elgreco]

Thanks for the fix!