Release v1.3.0-incubating

[pan3793]

Apache Kyuubi(Incubating) v1.3.0-incubating Release Notes

Highlight

Fully verified with the latest Spark 3.0/3.1
Support build on JDK8(default), JDK11
Provide enhanced AQE in Kyuubi Spark Extensions (for Spark 3.1)
Provide several auxiliary SQL functions as a supplement to Spark's built-in Functions
Major improvements of Spark on Kubernetes, include documents, integration tests, tools, etc.
Support engine/session level configurable initialize SQL statements
Support custom authentication implementation
Support stop Spark engine through Web UI
Support single session mode
Several improvements of observability
Fully tested with the all data lake platforms Delta, Iceberg
Fully tested with MiniYARNCluster

基于最新的 Spark 3.0/3.1 版本做了充分的验证
支持使用 JDK 8(默认), JDK 11 构建
在 Kyuubi Spark Extensions 模块中提供了增强的 AQE (目前支持 Spark 3.1)
提供了一些辅助的 SQL 函数作为 Spark 内置函数的补充
显著增强了 Spark 与 Kubernetes 的集成, 涵盖文档、集成测试、工具等
支持配置 Engine/Session 级别的初始化 SQL 语句列表
支持自定义认证插件
支持从 Spark Web UI 上关停 engine
支持单会话模式
提供更多的指标和事件输出，增强可观测性
在数据湖平台 Delta 和 Iceberg 之上进行了充分的测试
在 MiniYARNCluster 之上进行了充分的测试

Functionality

[1.2][KYUUBI #631] Add kyuubi-spark-extensions-3-1 module
[KYUUBI #657][KYUUBI #659] Provides auxiliary SQL functions as supplement to Spark's built-in Functions
[KYUUBI #897] Support stop Spark engine through Web UI
[KYUUBI #762] Add Kyuubi Hive JDBC Module
[KYUUBI #926] Support add repartition for OptimizedCreateHiveTableAsSelectCommand
[KYUUBI #670] KyuubiSQLException support sqlState and errorCode
[KYUUBI #707] Respect spark.sql.catalogImplementation in engine
[KYUUBI #641] Support custom authentication
[KYUUBI #751] Support single session mode
[1.2][KYUUBI #457] Support configurable initialize sql statement for engine startup
[KYUUBI #857] Support configurable initialize SQL statement for each session

Integration

[KYUUBI #755] Default build against Spark 3.1
[KYUUBI #693][KYUUBI #760] Support build against JDK11
[KYUUBI #768][K8S] Add module shuffle-files-cleaner
[KYUUBI #844][K8S] Exposing Kyuubi as a NodePort service
[KYUUBI #859][K8S] Add kubernetes configmap
[KYUUBI #656][K8S] Document how to deploy Kyuubi engines on Kubernetes
[KYUUBI #776][K8S] Run kyuubi on Kubernetes - make basic dockerfile and example
[KYUUBI #444][KYUUBI #854][TEST][K8S] Add minikube for K8s integration test
[KYUUBI #712][LDAP] Add a new config property authentication.ldap.guidKey
[KYUUBI #643][TEST] Test Detla 1.0.0 with Spark 3.1
[KYUUBI #948][TEST] Test Iceberg 0.12.0 with Spark 3.0/3.1
[KYUUBI #527][TEST][YARN] Test with MiniYARNCluster

Observability

[KYUUBI #737] Add monitor module
[KYUUBI #727] Add the listener for each SQL
[KYUUBI #767] Add statement event tracking

Performance

[1.2][KYUUBI #629] Refine distributed lock to lock only engine bootstrap

Others

[1.2][KYUUBI #416] Detected Dependency List Change for better dependency management
[KYUUBI #487][KYUUBI #566] Add TPC-DS DDL test cases
[KYUUBI #871] Change module name kyuubi-main to kyuubi-server

[yaooqinn]

Thanks for driving this