TableNotFoundError on DeltaTable.create(...)

[rspears74]

Environment

Delta-rs version: 0.15.1

Binding: Python

Environment:

• Cloud provider: AWS
• OS: MacOS
• Other: Running in Github Action

---

Bug

What happened:
I am calling DeltaTable.create with an S3 path. I get a TableNotFoundError.

What you expected to happen:
I expect a table to be created at the path I'm specifying.

How to reproduce it:
Use DeltaTable.create(...) (maybe with an S3 path?)

More details:

[ion-elgreco]

It would help if you can provide more details, what is the exact code you're running. Something that we can reproduce as well

[rspears74]

There's not much more to show.

schema = '''{
  "name": "id",
  "type": "string",
  "nullable": false,
  "metadata": {}
}'''
storage = {"AWS...": "...", ...}
table = DeltaTable.create(delta_table_s3_path, schema, mode='overwrite', storage_options=storage)

The main point here is that DeltaTable.create should never throw a TableNotFoundError as it doesn't even make sense in context. It should either create a new table, or say that a table already exists, but then if you create it in overwrite mode, it should work.

[rspears74]

This might have a similar problem as found in #2197 - which is essentially that it's unable to handle creating a new "folder" in s3.

[ion-elgreco]

@rspears74 the reason it throws a tableNotFoundError is because at the end of creation it's trying to read the table after creation.

I also use S3 but I don't run into this issue so perhaps it's some permission issue?

[rspears74]

@ion-elgreco See this comment #2195 (comment)

[ion-elgreco]

@rspears74 have you tried it against the latest version also?

It's not straightforward what the issue is because I don't have this experience with S3, albeit it's not the AWS s3

[rspears74]

@ion-elgreco I have not, I'm no longer working with delta-rs (this was over a month ago, and the most recent version at the time was 0.15.1). I don't have time to investigate further right now, but I expect you could potentially reproduce my issue by trying to create a table in an S3 folder that doesn't yet exist - following the example in this comment in the other issue I opened. I've worked around these issues for the time being.