[Python] Using unify_schema() during schema evolution fails

[PoojaRavi1105]

Describe the usage question you have. Please include as many useful details as possible.

I am trying to merge multiple parquet files using pyarrow's unify_schema(). During schema evolution, the same field has 2 different data type structures. What is the best way to handle schema evolution in such cases while making sure there is no loss of data?

Component(s)

Parquet, Python

[AlenkaF]

As per the documentation: "Note that two fields with different types will fail merging."
https://arrow.apache.org/docs/python/generated/pyarrow.unify_schemas.html

What are you using to merge the parquet files once you get the unified schema?

[PoojaRavi1105]

I'm reading the tables using pyarrow.parquet.read_table() and storing them in a list. After this, I'm looping through the whole list of tables and I'm using pyarrow.parquet.ParquetWriter.write_table() to write the merged parquet file. This is further being uploaded to S3 bucket.
Do we have a feature being planned to handle schema promotion? This would be a helpful feature to have in such scenarios.

[AlenkaF]

I see. There are some issues already opened connected to this. I suggest reading through:

• [C++][Dataset] Schema evolution in Dataset scanning #26923
• python - read multiple parquets that have different schema? #35569

Let me know if you get the information you are looking for.

[mapleFU]

@PoojaRavi1105

1. Currently parquet dataset doesn't support iceberg style schema evolution using unify_schema
2. But when you set the schema in dataset explicitly yourself, it's able to be read.

[sergun]

@PoojaRavi1105

1. Currently parquet dataset doesn't support iceberg style schema evolution using unify_schema
2. But when you set the schema in dataset explicitly yourself, it's able to be read.

reg. 2

It doesn't work for me in case of adding / removing columns (pyarrow==15.0.0). E.g.

1.parquet has scheme:

        pa.schema([
            ('id', pa.int64()),
            ('x', pa.int64()),
            ('a', pa.struct([
                ('y', pa.int64()),
            ])),
        ])

2.parquet has scheme:

        pa.schema([
            ('id', pa.int64()),
            ('y', pa.int64()),
            ('a', pa.struct([
                ('x', pa.int64()),
            ])),
        ])

When I read them by manually merged schema:

    dataset = ds.dataset(["1.parquet", "2.parquet",], schema=
        pa.schema([
            ('id', pa.int64()),
            ('x', pa.int64()),
            ('y', pa.int64()),
            ('a', pa.struct([
                ('x', pa.int64()),
                ('y', pa.int64()),
            ])),
        ])
    )

get:

pyarrow.lib.ArrowTypeError: struct fields don't match or are in the wrong order: Input fields: struct<x: int64> output fields: struct<x: int64, y: int64>
Guys, @AlenkaF are there some plans to support this in the roadmap?