-
Notifications
You must be signed in to change notification settings - Fork 924
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Bug] BuildQuery should respect normal node's input #5594
Labels
Comments
This was referenced Nov 1, 2023
18 tasks
zhaohehuhu
pushed a commit
to zhaohehuhu/incubator-kyuubi
that referenced
this issue
Feb 5, 2024
…nput # 🔍 Description ## Issue References 🔗 This pull request fixes apache#5594 ## Describe Your Solution 🔧 For case ``` def filter_func(iterator): for pdf in iterator: yield pdf[pdf.id == 1] df = spark.read.table("test_mapinpandas") execute_result = df.mapInPandas(filter_func, df.schema).show() ``` The logical plan is ``` GlobalLimit 21 +- LocalLimit 21 +- Project [cast(id#5 as string) AS id#11, name#6] +- MapInPandas filter_func(id#0, name#1), [id#5, name#6] +- HiveTableRelation [`default`.`test_mapinpandas`, org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, Data Cols: [id#0, name#1], Partition Cols: []] ``` When handle `MapInPandas`, we didn't match its input with `HiveTableRelation`, cause we miss input table's columns. This pr fix this In this pr, we remove the branch of each project such as `Project`, `Aggregate` etc, handle it together. ## Types of changes 🔖 - [x] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ For case ``` def filter_func(iterator): for pdf in iterator: yield pdf[pdf.id == 1] df = spark.read.table("test_mapinpandas") execute_result = df.mapInPandas(filter_func, df.schema).show() ``` We miss column info of table `test_mapinpandas` #### Behavior With This Pull Request 🎉 We got privilege object of table `test_mapinpandas` with it's column info. #### Related Unit Tests --- # Checklists ## 📝 Author Self Checklist - [x] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project - [x] I have performed a self-review - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) ## 📝 Committer Pre-Merge Checklist - [x] Pull request title is okay. - [x] No license issues. - [x] Milestone correctly set? - [x] Test coverage is ok - [x] Assignees are selected. - [x] Minimum number of approvals - [x] No changes are requested **Be nice. Be informative.** Closes apache#5787 from AngersZhuuuu/KYUUBI-5594-approach2. Closes apache#5594 e085455 [Angerszhuuuu] Update RangerSparkExtensionSuite.scala 49f09fb [Angerszhuuuu] Update RangerSparkExtensionSuite.scala 4781f75 [Angerszhuuuu] Update PrivilegesBuilderSuite.scala 9e9208d [Angerszhuuuu] Update V2JdbcTableCatalogRangerSparkExtensionSuite.scala 626d3dd [Angerszhuuuu] Update RangerSparkExtensionSuite.scala 3d69997 [Angerszhuuuu] Update PrivilegesBuilderSuite.scala 6eb4b8e [Angerszhuuuu] Update RangerSparkExtensionSuite.scala 61efb8a [Angerszhuuuu] update 794ebb7 [Angerszhuuuu] Merge branch 'master' into KYUUBI-5594-approach2 a236da8 [Angerszhuuuu] Update PrivilegesBuilderSuite.scala 74bd3f4 [Angerszhuuuu] Update RangerSparkExtensionSuite.scala 4acbc42 [Angerszhuuuu] Merge branch 'KYUUBI-5594-approach2' of https://github.com/AngersZhuuuu/incubator-kyuubi into KYUUBI-5594-approach2 266f7e8 [Angerszhuuuu] update a6c7845 [Angerszhuuuu] Update PrivilegesBuilder.scala d785d5f [Angerszhuuuu] Merge branch 'master' into KYUUBI-5594-approach2 014ef3b [Angerszhuuuu] Update PrivilegesBuilder.scala 7e1cd37 [Angerszhuuuu] Merge branch 'master' into KYUUBI-5594-approach2 71d2661 [Angerszhuuuu] update db95941 [Angerszhuuuu] update 490eb95 [Angerszhuuuu] update 70d110e [Angerszhuuuu] Merge branch 'master' into KYUUBI-5594-approach2 e6a5877 [Angerszhuuuu] Update PrivilegesBuilder.scala 5ff22b1 [Angerszhuuuu] Update PrivilegesBuilder.scala e684301 [Angerszhuuuu] Update PrivilegesBuilder.scala 594b202 [Angerszhuuuu] Update PrivilegesBuilder.scala 2f87c61 [Angerszhuuuu] Update RangerSparkExtensionSuite.scala 1de8c1c [Angerszhuuuu] Update PrivilegesBuilder.scala ad17255 [Angerszhuuuu] Update PrivilegesBuilderSuite.scala 4f5e850 [Angerszhuuuu] update 64349ed [Angerszhuuuu] Update PrivilegesBuilder.scala 11b7a4c [Angerszhuuuu] Update PrivilegesBuilder.scala 9a58fb0 [Angerszhuuuu] update d0b022e [Angerszhuuuu] Update RuleApplyPermanentViewMarker.scala e0f28a6 [Angerszhuuuu] Merge branch 'master' into KYUUBI-5594 0ebdd5d [Angerszhuuuu] Merge branch 'master' into KYUUBI-5594 8e53236 [Angerszhuuuu] update 3bafa7c [Angerszhuuuu] update d6e984e [Angerszhuuuu] update b00bf5e [Angerszhuuuu] Update PrivilegesBuilder.scala 8214228 [Angerszhuuuu] update 93fc689 [Angerszhuuuu] Merge branch 'master' into KYUUBI-5594 04184e3 [Angerszhuuuu] update 0bb7624 [Angerszhuuuu] Revert "Revert "Update PrivilegesBuilder.scala"" f481283 [Angerszhuuuu] Revert "Update PrivilegesBuilder.scala" 9f87182 [Angerszhuuuu] Revert "Update PrivilegesBuilder.scala" 29b67c4 [Angerszhuuuu] Update PrivilegesBuilder.scala 8785ad1 [Angerszhuuuu] Update PrivilegesBuilder.scala 270f21d [Angerszhuuuu] Update RangerSparkExtensionSuite.scala 60872ef [Angerszhuuuu] Update RangerSparkExtensionSuite.scala c34f32e [Angerszhuuuu] Merge branch 'master' into KYUUBI-5594 86fc475 [Angerszhuuuu] Update PrivilegesBuilder.scala 404f1ea [Angerszhuuuu] Update PrivilegesBuilder.scala dcca394 [Angerszhuuuu] Update PrivilegesBuilder.scala c2c6fa4 [Angerszhuuuu] Update PrivilegesBuilder.scala 6f6a36e [Angerszhuuuu] Merge branch 'master' into KYUUBI-5594]-AUTH]BuildQuery-should-respect-normal-node's-input 4dd47a1 [Angerszhuuuu] update c549b6a [Angerszhuuuu] update 80013b9 [Angerszhuuuu] Update PrivilegesBuilder.scala 3cbba42 [Angerszhuuuu] Update PrivilegesBuilder.scala Authored-by: Angerszhuuuu <[email protected]> Signed-off-by: Cheng Pan <[email protected]>
zhaohehuhu
pushed a commit
to zhaohehuhu/incubator-kyuubi
that referenced
this issue
Mar 21, 2024
…nput # 🔍 Description ## Issue References 🔗 This pull request fixes apache#5594 ## Describe Your Solution 🔧 For case ``` def filter_func(iterator): for pdf in iterator: yield pdf[pdf.id == 1] df = spark.read.table("test_mapinpandas") execute_result = df.mapInPandas(filter_func, df.schema).show() ``` The logical plan is ``` GlobalLimit 21 +- LocalLimit 21 +- Project [cast(id#5 as string) AS id#11, name#6] +- MapInPandas filter_func(id#0, name#1), [id#5, name#6] +- HiveTableRelation [`default`.`test_mapinpandas`, org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, Data Cols: [id#0, name#1], Partition Cols: []] ``` When handle `MapInPandas`, we didn't match its input with `HiveTableRelation`, cause we miss input table's columns. This pr fix this In this pr, we remove the branch of each project such as `Project`, `Aggregate` etc, handle it together. ## Types of changes 🔖 - [x] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan 🧪 #### Behavior Without This Pull Request ⚰️ For case ``` def filter_func(iterator): for pdf in iterator: yield pdf[pdf.id == 1] df = spark.read.table("test_mapinpandas") execute_result = df.mapInPandas(filter_func, df.schema).show() ``` We miss column info of table `test_mapinpandas` #### Behavior With This Pull Request 🎉 We got privilege object of table `test_mapinpandas` with it's column info. #### Related Unit Tests --- # Checklists ## 📝 Author Self Checklist - [x] My code follows the [style guidelines](https://kyuubi.readthedocs.io/en/master/contributing/code/style.html) of this project - [x] I have performed a self-review - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) ## 📝 Committer Pre-Merge Checklist - [x] Pull request title is okay. - [x] No license issues. - [x] Milestone correctly set? - [x] Test coverage is ok - [x] Assignees are selected. - [x] Minimum number of approvals - [x] No changes are requested **Be nice. Be informative.** Closes apache#5787 from AngersZhuuuu/KYUUBI-5594-approach2. Closes apache#5594 e085455 [Angerszhuuuu] Update RangerSparkExtensionSuite.scala 49f09fb [Angerszhuuuu] Update RangerSparkExtensionSuite.scala 4781f75 [Angerszhuuuu] Update PrivilegesBuilderSuite.scala 9e9208d [Angerszhuuuu] Update V2JdbcTableCatalogRangerSparkExtensionSuite.scala 626d3dd [Angerszhuuuu] Update RangerSparkExtensionSuite.scala 3d69997 [Angerszhuuuu] Update PrivilegesBuilderSuite.scala 6eb4b8e [Angerszhuuuu] Update RangerSparkExtensionSuite.scala 61efb8a [Angerszhuuuu] update 794ebb7 [Angerszhuuuu] Merge branch 'master' into KYUUBI-5594-approach2 a236da8 [Angerszhuuuu] Update PrivilegesBuilderSuite.scala 74bd3f4 [Angerszhuuuu] Update RangerSparkExtensionSuite.scala 4acbc42 [Angerszhuuuu] Merge branch 'KYUUBI-5594-approach2' of https://github.com/AngersZhuuuu/incubator-kyuubi into KYUUBI-5594-approach2 266f7e8 [Angerszhuuuu] update a6c7845 [Angerszhuuuu] Update PrivilegesBuilder.scala d785d5f [Angerszhuuuu] Merge branch 'master' into KYUUBI-5594-approach2 014ef3b [Angerszhuuuu] Update PrivilegesBuilder.scala 7e1cd37 [Angerszhuuuu] Merge branch 'master' into KYUUBI-5594-approach2 71d2661 [Angerszhuuuu] update db95941 [Angerszhuuuu] update 490eb95 [Angerszhuuuu] update 70d110e [Angerszhuuuu] Merge branch 'master' into KYUUBI-5594-approach2 e6a5877 [Angerszhuuuu] Update PrivilegesBuilder.scala 5ff22b1 [Angerszhuuuu] Update PrivilegesBuilder.scala e684301 [Angerszhuuuu] Update PrivilegesBuilder.scala 594b202 [Angerszhuuuu] Update PrivilegesBuilder.scala 2f87c61 [Angerszhuuuu] Update RangerSparkExtensionSuite.scala 1de8c1c [Angerszhuuuu] Update PrivilegesBuilder.scala ad17255 [Angerszhuuuu] Update PrivilegesBuilderSuite.scala 4f5e850 [Angerszhuuuu] update 64349ed [Angerszhuuuu] Update PrivilegesBuilder.scala 11b7a4c [Angerszhuuuu] Update PrivilegesBuilder.scala 9a58fb0 [Angerszhuuuu] update d0b022e [Angerszhuuuu] Update RuleApplyPermanentViewMarker.scala e0f28a6 [Angerszhuuuu] Merge branch 'master' into KYUUBI-5594 0ebdd5d [Angerszhuuuu] Merge branch 'master' into KYUUBI-5594 8e53236 [Angerszhuuuu] update 3bafa7c [Angerszhuuuu] update d6e984e [Angerszhuuuu] update b00bf5e [Angerszhuuuu] Update PrivilegesBuilder.scala 8214228 [Angerszhuuuu] update 93fc689 [Angerszhuuuu] Merge branch 'master' into KYUUBI-5594 04184e3 [Angerszhuuuu] update 0bb7624 [Angerszhuuuu] Revert "Revert "Update PrivilegesBuilder.scala"" f481283 [Angerszhuuuu] Revert "Update PrivilegesBuilder.scala" 9f87182 [Angerszhuuuu] Revert "Update PrivilegesBuilder.scala" 29b67c4 [Angerszhuuuu] Update PrivilegesBuilder.scala 8785ad1 [Angerszhuuuu] Update PrivilegesBuilder.scala 270f21d [Angerszhuuuu] Update RangerSparkExtensionSuite.scala 60872ef [Angerszhuuuu] Update RangerSparkExtensionSuite.scala c34f32e [Angerszhuuuu] Merge branch 'master' into KYUUBI-5594 86fc475 [Angerszhuuuu] Update PrivilegesBuilder.scala 404f1ea [Angerszhuuuu] Update PrivilegesBuilder.scala dcca394 [Angerszhuuuu] Update PrivilegesBuilder.scala c2c6fa4 [Angerszhuuuu] Update PrivilegesBuilder.scala 6f6a36e [Angerszhuuuu] Merge branch 'master' into KYUUBI-5594]-AUTH]BuildQuery-should-respect-normal-node's-input 4dd47a1 [Angerszhuuuu] update c549b6a [Angerszhuuuu] update 80013b9 [Angerszhuuuu] Update PrivilegesBuilder.scala 3cbba42 [Angerszhuuuu] Update PrivilegesBuilder.scala Authored-by: Angerszhuuuu <[email protected]> Signed-off-by: Cheng Pan <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Code of Conduct
Search before asking
Describe the bug
For case
The logical plan is
When handle
MapInPandas
, we didn't match its input withHiveTableRelation
, cause we miss input table's columnsAffects Version(s)
master, 1.8.0
Kyuubi Server Log Output
No response
Kyuubi Engine Log Output
No response
Kyuubi Server Configurations
No response
Kyuubi Engine Configurations
No response
Additional context
No response
Are you willing to submit PR?
The text was updated successfully, but these errors were encountered: