feat: Add heartbeat mechanism for failure discovery #2
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…4083) With primitive keys coming the QTT test needs to be able to test the full schema is correct, not just the value schema.
First of a few commits to start introducing support for primitive keys in different query types. This commit opens the door for CT/CS statements with primitive keys, (`STRING`, `INT`, `BIGINT`, `BOOLEAN` and `DOUBLE`), and for using those sources in non-join, non-aggregate and non-partition-by queries.
Extends qtt to build the query plan and deserialize/serialize it before executing. Should be enough to make sure plans are serializable and executable until we improve qtt.
* chore: partition-by primitive key support Fixes: confluentinc#4092 WIP: This commit gets `PARTITION BY` clauses working with primitive key types. However, it does disable a couple of join until confluentinc#4094 has been completed. BREAKING CHANGE: A `PARTITION BY` now changes the SQL type of `ROWKEY` in the output schema of a query. For example, consider: ```sql CREATE STREAM INPUT (ROWKEY STRING KEY, ID INT) WITH (...); CREATE STREAM OUTPUT AS SELECT ROWKEY AS NAME FROM INPUT PARTITION BY ID; ``` Previously, the above would have resulted in an output schema of `ROWKEY STRING KEY, NAME STRING`, where `ROWKEY` would have stored the string representation of the integer from the `ID` column. With this commit the output schema will be `ROWKEY INT KEY, NAME STRING`.
Adding the next release milestone (0.7.0 at this time) to every bug report so that they are triaged weekly and can be prioritized accordingly.
the milestone field seems to have no effect, reverting.
* refactor: nicer spec for aggregation steps This patch cleans up the aggregation step to specify a list of non-aggregate column references instead of a count of non-aggregate columns.
confluentinc#4105) * chore: split ARRAYCONTAINS into JSON_ARRAY_CONTAINS and ARRAY_CONTAINS BREAKING CHANGE: the ARRAYCONTAINS function now needs to be referenced as either JSON_ARRAY_CONTAINS or ARRAY_CONTAINS depending on the intended param types
…#4116) BREAKING CHANGE: commands that were persisted with RUN SCRIPT will no longer be executable
* chore: group-by primitive key support Fixes: confluentinc#4092 This commit gets `GROUP BY` clauses working with primitive key types. BREAKING CHANGE: A `GROUP BY` on single expressions now changes the SQL type of `ROWKEY` in the output schema of the query to match the SQL type of the expression. For example, consider: ```sql CREATE STREAM INPUT (ROWKEY STRING KEY, ID INT) WITH (...); CREATE TABLE OUTPUT AS SELECT COUNT(*) AS COUNT FROM INPUT GROUP BY ID; ``` Previously, the above would have resulted in an output schema of `ROWKEY STRING KEY, COUNT BIGINT`, where `ROWKEY` would have stored the string representation of the integer from the `ID` column. With this commit the output schema will be `ROWKEY INT KEY COUNT BIGINT`. BREAKING CHANGE: Any`GROUP BY` expression that resolves to `NULL`, including because a UDF throws an exception, now results in the row being excluded from the result. Previously, as the key was a `STRING` a value of `"null"` could be used. With other primitive types this is not possible. As key columns must be non-null any exception is logged and the row is excluded.
…c#4310) * test: push queries work with non-windowed primitive keys Fixes: confluentinc#4123
I followed the instructions to build and run locally, and fatal errors weren't written to stdout - took me a while to realise why the process was just exiting :) Found the error in the `ksql.log` log in the logs folder.
This patch pins the jetty client version to the jetty version specified by confluent/common.
and fixed small bug in test framework
…c#4313) This commit: 1. exposes the window type of the key of a query/source, i.e. `HOPPING`, `TUMBLING` `SESSION` or none. 2. makes the status of a query easier to find. 3. fixes a bug that meant the statement text of a query was not displayed in the CLI. BREAKING CHANGE: The response from the RESTful API has changed for some commands with this commit: the `SourceDescription` type no longer has a `format` field. Instead it has `keyFormat` and `valueFormat` fields. ## `SHOW QUERY` changes: Response now includes a `state` property for each query that indicates the state of the query. e.g. ```json { "queryString" : "create table OUTPUT as select * from INPUT;", "sinks" : [ "OUTPUT" ], "id" : "CSAS_OUTPUT_0", "state" : "Running" } ``` The CLI output was: ``` ksql> show queries; Query ID | Kafka Topic | Query String CSAS_OUTPUT_0 | OUTPUT | CREATE STREAM OUTPUT WITH (KAFKA_TOPIC='OUTPUT', PARTITIONS=1, REPLICAS=1) AS SELECT * FROM INPUT INPUT EMIT CHANGES; CTAS_CLICK_USER_SESSIONS_5 | CLICK_USER_SESSIONS | CREATE TABLE CLICK_USER_SESSIONS WITH (KAFKA_TOPIC='CLICK_USER_SESSIONS', PARTITIONS=1, REPLICAS=1) AS SELECT CLICKSTREAM.USERID USERID, COUNT(*) COUNT FROM CLICKSTREAM CLICKSTREAM WINDOW SESSION ( 300 SECONDS ) GROUP BY CLICKSTREAM.USERID EMIT CHANGES; For detailed information on a Query run: EXPLAIN <Query ID>; ``` and is now: ``` Query ID | Status | Kafka Topic | Query String CSAS_OUTPUT_0 | RUNNING | OUTPUT | CREATE STREAM OUTPUT WITH (KAFKA_TOPIC='OUTPUT', PARTITIONS=1, REPLICAS=1) AS SELECT *FROM INPUT INPUTEMIT CHANGES; For detailed information on a Query run: EXPLAIN <Query ID>; ``` Note the addition of the `Status` column and the fact that `Query String` is now longer being written across multiple lines. ## `DESCRIBE <source>;` changes: old CLI output: ``` ksql> describe CLICK_USER_SESSIONS; Name : CLICK_USER_SESSIONS Field | Type ROWTIME | BIGINT (system) ROWKEY | INTEGER (system) USERID | INTEGER COUNT | BIGINT For runtime statistics and query details run: DESCRIBE EXTENDED <Stream,Table>; ``` New CLI output: ``` ksql> describe CLICK_USER_SESSIONS; Name : CLICK_USER_SESSIONS Field | Type ROWTIME | BIGINT (system) ROWKEY | INTEGER (system) (Window type: SESSION) USERID | INTEGER COUNT | BIGINT For runtime statistics and query details run: DESCRIBE EXTENDED <Stream,Table>; ``` Note the addition of the `Window Type` information. The extended version of the command has also changed. Old output: ``` ksql> describe extended CLICK_USER_SESSIONS; Name : CLICK_USER_SESSIONS Type : TABLE Key field : USERID Key format : STRING Timestamp field : Not set - using <ROWTIME> Value Format : JSON Kafka topic : CLICK_USER_SESSIONS (partitions: 1, replication: 1) Statement : CREATE TABLE CLICK_USER_SESSIONS WITH (KAFKA_TOPIC='CLICK_USER_SESSIONS', PARTITIONS=1, REPLICAS=1) AS SELECT CLICKSTREAM.USERID USERID, COUNT(*) COUNT FROM CLICKSTREAM CLICKSTREAM WINDOW SESSION ( 300 SECONDS ) GROUP BY CLICKSTREAM.USERID EMIT CHANGES; Field | Type ROWTIME | BIGINT (system) ROWKEY | INTEGER (system) USERID | INTEGER COUNT | BIGINT Queries that write from this TABLE ----------------------------------- CTAS_CLICK_USER_SESSIONS_5 (RUNNING) : CREATE TABLE CLICK_USER_SESSIONS WITH (KAFKA_TOPIC='CLICK_USER_SESSIONS', PARTITIONS=1, REPLICAS=1) AS SELECT CLICKSTREAM.USERID USERID, COUNT(*) COUNT FROM CLICKSTREAM CLICKSTREAM WINDOW SESSION ( 300 SECONDS ) GROUP BY CLICKSTREAM.USERID EMIT CHANGES; For query topology and execution plan please run: EXPLAIN <QueryId> Local runtime statistics ------------------------ (Statistics of the local KSQL server interaction with the Kafka topic CLICK_USER_SESSIONS) ``` New output: ``` ksql> describe extended CLICK_USER_SESSIONS; Name : CLICK_USER_SESSIONS Type : TABLE Key field : USERID Timestamp field : Not set - using <ROWTIME> Key format : KAFKA Value format : JSON Kafka topic : CLICK_USER_SESSIONS (partitions: 1, replication: 1) Statement : CREATE TABLE CLICK_USER_SESSIONS WITH (KAFKA_TOPIC='CLICK_USER_SESSIONS', PARTITIONS=1, REPLICAS=1) AS SELECT CLICKSTREAM.USERID USERID, COUNT(*) COUNT FROM CLICKSTREAM CLICKSTREAM WINDOW SESSION ( 300 SECONDS ) GROUP BY CLICKSTREAM.USERID EMIT CHANGES; Field | Type ROWTIME | BIGINT (system) ROWKEY | INTEGER (system) (Window type: SESSION) USERID | INTEGER COUNT | BIGINT Queries that write from this TABLE ----------------------------------- CTAS_CLICK_USER_SESSIONS_5 (RUNNING) : CREATE TABLE CLICK_USER_SESSIONS WITH (KAFKA_TOPIC='CLICK_USER_SESSIONS', PARTITIONS=1, REPLICAS=1) AS SELECT CLICKSTREAM.USERID USERID, COUNT(*) COUNTFROM CLICKSTREAM CLICKSTREAMWINDOW SESSION ( 300 SECONDS ) GROUP BY CLICKSTREAM.USERIDEMIT CHANGES; For query topology and execution plan please run: EXPLAIN <QueryId> Local runtime statistics ------------------------ (Statistics of the local KSQL server interaction with the Kafka topic CLICK_USER_SESSIONS) ``` Note: the change from `Key format` of `STRING` to `KAFKA`. The output of `Window Type` information for windowed schemas and outputing sql statements on a single line.
fixed tests use application_server config to determine local host address fixed compile issues added extra tests test fixed failing test added debug logging, made critical section smaller addressed almogs comments added return
vpapavas
pushed a commit
that referenced
this pull request
Mar 27, 2023
* Revert "Revert "[KSE-1612] Upgrade netty to 4.1.86 and vertx to 4.3.7 (confluentinc#9764)" (confluentinc#9766)" This reverts commit 1ecd4f7. * Fix KsqlClientTest Update exception expectations in KsqlClientTest according to the upgraded vert.x. * Update origin in CorsTest as per OWASP Include schema into the ORIGIN, see vert-x3/vertx-web@c667f07
vpapavas
pushed a commit
that referenced
this pull request
Mar 27, 2023
* Revert "Revert "[KSE-1612] Upgrade netty to 4.1.86 and vertx to 4.3.7 (confluentinc#9764)" (confluentinc#9766)" This reverts commit 1ecd4f7. * Fix KsqlClientTest Update exception expectations in KsqlClientTest according to the upgraded vert.x. * Update origin in CorsTest as per OWASP Include schema into the ORIGIN, see vert-x3/vertx-web@c667f07
vpapavas
pushed a commit
that referenced
this pull request
Mar 27, 2023
* Revert "Revert "[KSE-1612] Upgrade netty to 4.1.86 and vertx to 4.3.7 (confluentinc#9764)" (confluentinc#9766)" This reverts commit 1ecd4f7. * Fix KsqlClientTest Update exception expectations in KsqlClientTest according to the upgraded vert.x. * Update origin in CorsTest as per OWASP Include schema into the ORIGIN, see vert-x3/vertx-web@c667f07
vpapavas
pushed a commit
that referenced
this pull request
Mar 27, 2023
* Revert "Revert "[KSE-1612] Upgrade netty to 4.1.86 and vertx to 4.3.7 (confluentinc#9764)" (confluentinc#9766)" This reverts commit 1ecd4f7. * Fix KsqlClientTest Update exception expectations in KsqlClientTest according to the upgraded vert.x. * Update origin in CorsTest as per OWASP Include schema into the ORIGIN, see vert-x3/vertx-web@c667f07
vpapavas
pushed a commit
that referenced
this pull request
Mar 27, 2023
* Revert "Revert "[KSE-1612] Upgrade netty to 4.1.86 and vertx to 4.3.7 (confluentinc#9764)" (confluentinc#9766)" This reverts commit 1ecd4f7. * Fix KsqlClientTest Update exception expectations in KsqlClientTest according to the upgraded vert.x. * Update origin in CorsTest as per OWASP Include schema into the ORIGIN, see vert-x3/vertx-web@c667f07
vpapavas
pushed a commit
that referenced
this pull request
Mar 27, 2023
* Revert "Revert "[KSE-1612] Upgrade netty to 4.1.86 and vertx to 4.3.7 (confluentinc#9764)" (confluentinc#9766)" This reverts commit 1ecd4f7. * Fix KsqlClientTest Update exception expectations in KsqlClientTest according to the upgraded vert.x. * Update origin in CorsTest as per OWASP Include schema into the ORIGIN, see vert-x3/vertx-web@c667f07
vpapavas
pushed a commit
that referenced
this pull request
Mar 27, 2023
* Revert "Revert "[KSE-1612] Upgrade netty to 4.1.86 and vertx to 4.3.7 (confluentinc#9764)" (confluentinc#9766)" This reverts commit 1ecd4f7. * Fix KsqlClientTest Update exception expectations in KsqlClientTest according to the upgraded vert.x. * Update origin in CorsTest as per OWASP Include schema into the ORIGIN, see vert-x3/vertx-web@c667f07
vpapavas
pushed a commit
that referenced
this pull request
Mar 27, 2023
* Revert "Revert "[KSE-1612] Upgrade netty to 4.1.86 and vertx to 4.3.7 (confluentinc#9764)" (confluentinc#9766)" This reverts commit 1ecd4f7. * Fix KsqlClientTest Update exception expectations in KsqlClientTest according to the upgraded vert.x. * Update origin in CorsTest as per OWASP Include schema into the ORIGIN, see vert-x3/vertx-web@c667f07
vpapavas
pushed a commit
that referenced
this pull request
Mar 27, 2023
* Revert "Revert "[KSE-1612] Upgrade netty to 4.1.86 and vertx to 4.3.7 (confluentinc#9764)" (confluentinc#9766)" This reverts commit 1ecd4f7. * Fix KsqlClientTest Update exception expectations in KsqlClientTest according to the upgraded vert.x. * Update origin in CorsTest as per OWASP Include schema into the ORIGIN, see vert-x3/vertx-web@c667f07
vpapavas
pushed a commit
that referenced
this pull request
Mar 27, 2023
This reverts commit 3c9b295.
vpapavas
pushed a commit
that referenced
this pull request
Mar 27, 2023
This reverts commit a2cbd00.
vpapavas
pushed a commit
that referenced
this pull request
Mar 27, 2023
This reverts commit e1993fa.
vpapavas
pushed a commit
that referenced
this pull request
Mar 27, 2023
This reverts commit 5a1ff37.
vpapavas
pushed a commit
that referenced
this pull request
Mar 27, 2023
This reverts commit 1bea324.
vpapavas
pushed a commit
that referenced
this pull request
Mar 27, 2023
This reverts commit 8c59d45.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Added
HeartbeatHandlerand two resourcesHeartbeatResourceandClusterStatusResourceTesting done
Check of tests for
HeartbeatHandlerandHeartbeatResourceReviewer checklist