Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dremio jdbc plugin #162

Closed
wants to merge 7 commits into from
Closed

Conversation

iNikitaGricenko
Copy link
Contributor

@iNikitaGricenko iNikitaGricenko commented Oct 16, 2023

What changes are being made and why?

  • Added dreamio query task and Trigger using the JDBC drive.
  • Added Query task using the Arrow Flight SQL JDBC drive.

How the changes have been QAed?

Dreamio query was tested through Dremio Test Drive and Dremio OSS (Docker)
Arrow Flight SQL was tested through Dremio account, adding source that supports Apache Arrow Flight sql

id : dremio-listen
namespace: dev

triggers:
  - id: watch
    type: io.kestra.plugin.jdbc.dremio.Trigger
    url: "jdbc:arrow-flight-sql://localhost:31010/?useEncryption=false"
    username: "dremio"
    password: "dremio123"
    sql: "SELECT * from \"dremio-test\".\"postgres.public\".departments"
    fetch: true
    interval: PT10S

tasks:
  - id: query
    type: io.kestra.plugin.jdbc.dremio.Query
    url: "jdbc:arrow-flight-sql://localhost:31010/?useEncryption=false"
    username: "dremio"
    password: "dremio123"
    sql: "SELECT * from \"dremio-test\".\"postgres.public\".departments"
    fetchOne: true

Setup Instructions

Copy link
Member

@loicmathieu loicmathieu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think using both drivers is useful as you can do the same with the Dremio driver and the Apache Flight SQL driver.

We will discuss internally if we need to change something about that.

@Getter
@NoArgsConstructor
@Schema(
title = "Query a Dremio database through Apache Arrow Flight sql driver."
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
title = "Query a Dremio database through Apache Arrow Flight sql driver."
title = "Query a Dremio database through Apache Arrow Flight SQL."

@Plugin(
examples = {
@Example(
title = "Send a sql query to a Dremio direct database and fetch a row as outputs using Apache Arrow Flight sql driver",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
title = "Send a sql query to a Dremio direct database and fetch a row as outputs using Apache Arrow Flight sql driver",
title = "Send a SQL query to a Dremio database and fetch a row as outputs using Apache Arrow Flight SQL",

}
),
@Example(
title = "Send a sql query to a Dremio coordinator and fetch a row as outputs using Apache Arrow Flight sql driver",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
title = "Send a sql query to a Dremio coordinator and fetch a row as outputs using Apache Arrow Flight sql driver",
title = "Send a SQL query to a Dremio coordinator and fetch a row as outputs using Apache Arrow Flight SQL",

@Plugin(
examples = {
@Example(
title = "Send a sql query to a Dremio database and fetch a row as outputs",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
title = "Send a sql query to a Dremio database and fetch a row as outputs",
title = "Send a SQL query to a Dremio database and fetch a row as outputs",

@Plugin(
examples = {
@Example(
title = "Wait for a sql query to return results and iterate through rows",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
title = "Wait for a sql query to return results and iterate through rows",
title = "Wait for a SQL query to return results and iterate through rows",

"",
"triggers:",
" - id: watch",
" type: io.kestra.plugin.jdbc.clickhouse.Trigger",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
" type: io.kestra.plugin.jdbc.clickhouse.Trigger",
" type: io.kestra.plugin.jdbc.dremio.Trigger",

@anna-geller
Copy link
Member

closes kestra-io/kestra#1854

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants