DeltaTable.hisotry(): return timestamp column as datetime (not epoch integer)

[keen85]

Description

When accessing the history of a Delta Table via delta-spark, the version timestamp is returned as TimestampType column.
delta-rs is different here, it returns an epoch timestamp as integer:

• https://delta-io.github.io/delta-rs/usage/examining-table/#history
• https://delta-io.github.io/delta-rs/api/delta_table/#deltalake.DeltaTable.history

I suggest to make behavior of delta-rs consistent to delta-rs and return proper Python datetime value.

Use Case
Make it easier for users to interpret the result of DeltaTable.hisotry().

Conversion in Python is fairly easy:

dt = DeltaTable("../rust/tests/data/simple_table")
ts_epoch = dt.history(1)[0]["timestamp"]
ts_datetime = datetime.utcfromtimestamp(ts / 1000.0)

However, this would be a breaking change 😬

Related Issue(s)
I found nothing

[ion-elgreco]

That would actually require quite some changes, the CommitInfo struct holds the timestamp as Option.

@roeap Any idea's on this?

[roeap]

I would not be too concerned about breaking that, as it does reduce the source of some errors that might happen (i.e. Millis / micros etc.)

Commit Infos are generally annoing though, as there is no real convention / guarantees for it. delta-rs just does kind of what spark does, but who know what other writers are doing 😆.

So parsing all of that might be difficult, since the contents of a field are not really known to us unless we wrote it ourselves.

[rtyler]

There is no guaranteed schema of commitInfo in the protocol, my vote here would be to do nothing rather than chase compatibility with undefined Spark behavior