[PECO-239] Apache Arrow support

[kravets-levko]

PECO-239 Add Apache Arrow support

• Basic Apache arrow support
• TSparkArrowTypes support
• Unit tests
• E2E tests
• Have Arrow feature always enabled
• Option to control if native arrow types should be used or not
• Return Arrow result as regular javascript object (convert from internal apache-arrow classes)
  • Should be compatible with existing JSON result handler
  • If native types disabled - should parse JSON-serialized fields similarly to existing JSON result handler
  • Tests

Arrow support is always enabled. User can choose whether to represent complex types as JSON-encoded strings or as native Arrow objects by setting a options.useArrowNativeTypes option of IDBSQLSession.executeStatement (defaults to true)

TSparkArrowTypes.intervalTypesAsArrow is not currently supported by apache-arrow (see apache/arrow#21815) and therefore not exposed to user. PySQL also doesn't support this option, so at least we're consistent here

[andrefurlan-db]

Before merging this I'd like to review the interfaces and tests with @rcypher-databricks and Go driver.

[andrefurlan-db]

a few surprises here.
Missing:
tests that confirms that arrow on or off do not change the result.
tests to ensure that pagination logic still works with arrow.

[kravets-levko]

@andrefurlan-db I updated tests so they cover most data types handling. Also, added a separate test to make sure that reponse with multiple arrow batches is also handled as it should. Please take a look. Thank you!

[andrefurlan-db]

Almost there! Added a few comments.

[andrefurlan-db]

are you sure? I thought map[any]any was the type definition. So you could have a struct as key, no?

[kravets-levko]

this is for compatibility with column results - maps are json-serialized as plain objects which could have only string keys. we could try to use JS built-in Map class to represent maps, but for column results it probably will have some limitations

[andrefurlan-db]

Have you run any tests with a map using complex data type as key?

[kravets-levko]

I mentioned it in Readme in tests/fixtures folder. In short: complex types used as map keys produce malformed json (similar to date/time values), so they could be only handled with Arrow + native types enabled. And even in this case, since JS objects cannot have complex type keys, they will remain stringified. So it will look something like this:

{ 
  '{"s": "hello"}': ..., 
  '{"s": "world"}': ..., 
}

[andrefurlan-db]

@rcypher-databricks , what is the other serialization issue with arrow disabled?

[kravets-levko]

those cases are the ones I stumbled at. there's even a thread in Slack about date serialization. if you know other similar issues - I'd like to check them as well.

[rcypher-databricks]

The only other similar issue I ran into was dealing with intervals when arrow native types are turned on. Setting the intervalTypesAsArrow flag to false doesn't apply to intervals that are nested in complex types.

[andrefurlan-db]

looks good

[susodapop]

Stamp. I updated the PAT for e2e tests and kicked-off a new execution.