support file filter on external table

[ftong2020]

What type of enhancement is this?

User experience

What does the enhancement do?

When using greptimedb external table, user can use a path to include multiple files. On MacOS, OS itself will add .DS_Store file to the folder when it is indexed. Then, external table will complain invalid file format because .DS_Store is also included.

It will be very handy if external table can support glob filters or at least some form of wildcard matching.

Implementation challenges

No response

[WenyXu]

Hi @ftong2020,
We support the pattern option when creating an external table. Here's an example:

CREATE EXTERNAL TABLE city (
            host string,
            ts timestamp,
            cpu float64 default 0,
            memory float64,
            TIME INDEX (ts),
            PRIMARY KEY(host)
) WITH (location='/var/data/', format='parquet', pattern='*.parquet');

For more details:
https://docs.greptime.com/reference/sql/create/#create-external-table