v4.0.0

.gitignore

@@ -25,6 +25,7 @@ artifacts/
 work/
 ci/
 e2e-results/
+test_summary.json
 
 # Editor
 .idea

CHANGELOG.md

@@ -1,10 +1,69 @@
 # Changelog
 
-## Unreleased
+## 4.0.0
 
 ### Features
 
-- Add a new optional `path` setting to specify an additional URL path. [#512](https://github.com/grafana/clickhouse-datasource/pull/512)
+Version 4.0.0 contains major revisions to the query builder and datasource configuration settings.
+
+#### Query Builder
+
+- Completely rebuilt query builder to have specialized editors for Table, Logs, Time Series, and Traces.
+- Completely rebuilt SQL generator to support more complicated and dynamic queries.
+- Updated query builder options structure to be clearer and support more complex queries.
+- Updated database/table selector to be in a more convenient location. Database and table options are automatically selected on initial load.
+- Upgraded query builder state management so queries stay consistent when saving/editing/sharing queries.
+- Separated Table and Time Series query builders. Table view operates as a catch-all for queries that don't fit the other query types.
+- Combined "format" into the query type switcher for simplicity. The query tab now changes the builder view and the display format when on the Explore page. This includes the raw SQL editor.
+- Added an OTEL switch for logs and trace views. This will allow for quicker query building for those using the OTEL exporter for ClickHouse.
+- Updated Time Series query builder with dedicated Time column. Default filters are added on-load.
+- Added an `IS ANYTHING` filter that acts as a placeholder for easily editing later (useful for query templates/bookmarks on the Explore page.)
+- Added better support for Map types on the Filter editor.
+- LIMIT editor can now be set to 0 to be excluded from the query.
+- Table and Time Series views now have a simple / aggregate mode, depending on the query complexity.
+- Updated the logs histogram query to use the new query builder options and column hints.
+- Added Logs query builder with dedicated Time, Level, and Message columns. Includes OTEL switch for automatically loading OTEL schema columns. Default filters are added on-load.
+- Added Trace query builder with dedicated trace columns. Includes OTEL switch for automatically loading OTEL schema columns. Default filters are added on-load.
+- Updated data panel filtering to append filters with column hints. Visible in logs view when filtering by a specific level. Instead of referencing a column by name, it will use its hint.
+- Order By now lists aggregates by their full name + alias.
+- Order By column allows for custom value to be typed in.
+- Aggregate column name allows for custom value to be typed in.
+- Filter editor allows for custom column names to be typed in.
+- Increased width of filter value text input.
+- Columns with the `Map*` type now show a `[]` at the end to indicate they are complex types. For example, `SpanAttributes[]`.
+- Filter editor now has a dedicated field for map key. You can now select a map column and its key separately. For example, `SpanAttributes['key']`.
+- Map types now load a sample of options when editing the `key` for the map. This doesn't include all unique values, but for most datasets it should be a convenience.
+- Added column hints, which offers better linking across query components when working with columns and filters. For example, a filter can be added for the `Time` column, even without knowing what the time column name is yet. This enables better SQL generation that is "aware" of a column's intended use.
+
+### Plugin Backend
+- Added migration logic for `v3` configs going to `v4+`. This is applied when the config is loaded when building a database connection.
+- `$__timeFilter`, `$__fromTime`, and `$__toTime` macros now convert to `DateTime64(3)` for better server-side type conversion. Also enables millisecond precision time range filtering.
+
+#### Datasource Configuration
+- Added migration script for `v3.x` configurations to `v4+`. This runs automatically when opening/saving the datasource configuration.
+- Renamed config value `server` to `host`.
+- Renamed config value `timeout` to the more specific `dial_timeout`.
+- Updated labeling for port selection. The default port will now change depending on native/http and secure/unsecure setting.
+- Rearranged fields and sections to flow better for initial setup of a new datasource.
+- Added plugin version to config data for easier config version migrations in the future.
+- Added fields for setting default values for database/table.
+- Added section for setting default log database/table/columns. Includes OTEL. These are used when using the log query builder.
+- Added section for setting default trace database/table/columns. Includes OTEL. These are used when using the trace query builder.
+- Added OTEL switches for logs/traces for quicker query building. OTEL defaults to the latest version, and will auto update if kept on this setting.
+- Increased width of inputs for typically long values (server URL, path, etc.)
+- Allow adding custom HTTP headers with either plain text or secure credentials. [#633](https://github.com/grafana/clickhouse-datasource/pull/633)
+- Add `path` setting to specify an additional URL path when using the HTTP protocol. [#512](https://github.com/grafana/clickhouse-datasource/pull/512)
+
+### Fixes
+
+- Queries will now remain consistent when reloading/editing a previously saved query.
+- Fixed default Ad-Hoc filters. [#650](https://github.com/grafana/clickhouse-datasource/pull/650)
+- Fixed Ad-Hoc filters parsing numeric fields. [#629](https://github.com/grafana/clickhouse-datasource/pull/629)
+- Fixed majority of usability quirks with redesigned query builder.
+
+### Upgrades
+
+- Updated all dependencies to latest compatible versions (Includes Dependabot PRs)
 
 ## 3.3.0
 

DEV_GUIDE.md

@@ -70,3 +70,47 @@ docker run -d -p 8443:8443 -p 9440:9440 -p 9000:9000 -p 8123:8123 --name secure-
 docker exec -it secure-clickhouse-server bash
 cp /etc/clickhouse-server/my-own-ca.crt /usr/local/share/ca-certificates/root.ca.crt
 update-ca-certificates
+
+# Code Structure / Notes
+
+## Column Hints
+
+Column hints are used within the query builder and SQL generator to enable flexible and dynamic queries.
+
+Here's an example of some column hints:
+```js
+ColumnHint.Time
+ColumnHint.LogMessage
+ColumnHint.LogLevel
+ColumnHint.TraceId
+```
+
+The easiest example is the time hint (`ColumnHint.Time`). When building a Logs query, we need to know what the primary log time column is:
+
+```ts
+const logTimeColumn: SelectedColumn = { name: 'my_time_column_on_my_table', hint: ColumnHint.Time, alias: 'logTime' };
+```
+
+Using the column hint, we can add an `ORDER BY` statement to the query without having to know the actual column name:
+
+```ts
+const logsOrderBy: OrderBy = { name: '', hint: ColumnHint.Time, dir: OrderByDirection.ASC };
+```
+
+Notice how `name` can be left empty, this is because the SQL generator knows to find the final column/alias by the time hint:
+
+```ts
+// Input options
+const queryBuilderOptions: QueryBuilderOptions = {
+  table: 'logs',
+  columns: [logTimeColumn],
+  orderBy: [logsOrderBy],
+  . . .
+};
+```
+```sql
+-- Final output from SQL generator
+SELECT my_time_column_on_my_table as logTime FROM logs ORDER BY logTime ASC
+```
+
+By adding a simple hint, we can apply filters, orderBys, and other behaviors to the SQL generator without having to reference specific columns. This simplifies the UI logic and user experience by reducing the number of places where a column name needs to be updated.

README.md

@@ -1,63 +1,63 @@
-# ClickHouse data source for Grafana
+# Official ClickHouse data source for Grafana
 
+The ClickHouse data source plugin allows you to query and visualize ClickHouse data in Grafana.
 
 <img alt="Grafana Dashboard Screenshot - Query Analysis" src="https://github.com/grafana/clickhouse-datasource/assets/5509570/d129936e-afac-4002-8963-61c15825c154" width="400" >
 
 <img alt="Grafana Dashboard Screenshot - Data Analysis" src="https://github.com/grafana/clickhouse-datasource/assets/5509570/5911f72b-0a52-4e1e-9cd4-3905ac0623cd" width="400" >
 
-
 ## Version compatibility
 
-Users on `v8.x` of Grafana are encouraged to continue to use `v2.2.0` of the plugin. 
-Users on `v9.x` and higher of Grafana can use `v3` however it is `beta` and may contain bugs.
+Users on Grafana `v9.x` and higher of Grafana can use `v4`.
+Users on Grafana `v8.x` are encouraged to continue using `v2.2.0` of the plugin.
 
-The ClickHouse data source plugin allows you to query and visualize ClickHouse
-data from within Grafana.
 
-**As of 2.0 this plugin will only support ad hoc filters when using ClickHouse 22.7+**
+\* *As of 2.0 this plugin will only support ad hoc filters when using ClickHouse 22.7+*
 
 ## Installation
 
-For detailed instructions on how to install the plugin on Grafana Cloud or
-locally, please checkout the [Plugin installation docs](https://grafana.com/docs/grafana/latest/plugins/installation/).
+For detailed instructions on how to install the plugin on Grafana Cloud or locally,
+please checkout the [Plugin installation docs](https://grafana.com/docs/grafana/latest/plugins/installation/).
 
 ## Configuration
 
 ### ClickHouse user for the data source
 
 Set up an ClickHouse user account with [readonly](https://clickhouse.com/docs/en/operations/settings/permissions-for-queries#settings_readonly) permission and access to
-databases and tables you want to query. Please note that Grafana does not
-validate that queries are safe. Queries can contain any SQL statement. For
-example, statements like `ALTER TABLE system.users DELETE WHERE name='sadUser'`
+databases and tables you want to query.
+Please note that Grafana does not validate that queries are safe. Queries can contain any SQL statement.
+For example, statements like `ALTER TABLE system.users DELETE WHERE name='sadUser'`
 and `DROP TABLE sadTable;` would be executed.
 
 To configure a readonly user, follow these steps:
 1. Create a `readonly` user profile following the [Creating Users and Roles in ClickHouse](https://clickhouse.com/docs/en/operations/access-rights) guide.
 2. Ensure the `readonly` user has enough permission to modify the `max_execution_time` setting required by the underlying [clickhouse-go client](https://github.com/ClickHouse/clickhouse-go/).
 3. If you're using a public Clickhouse instance, it's not recommended to set `readonly=2` in the `readonly` profile. Instead, leave `readonly=1` and set the constraint type of `max_execution_time` to [changeable_in_readonly](https://clickhouse.com/docs/en/operations/settings/constraints-on-settings) to allow modification of this setting.
+
 ### ClickHouse protocol support
 
-The plugin supports both `HTTP` and `Native` (default) transport protocols. This can be enabled in the configuration via the `protocol` configuration parameter. Both protocols exchange data with ClickHouse using optimized native format.
+The plugin supports both `Native` (default) and `HTTP` transport protocols.
+This can be enabled in the configuration via the `protocol` configuration parameter.
+Both protocols exchange data with ClickHouse using optimized native format.
 
-Note that the default ports for `HTTP/s` and `Native` differ:
+Note that the default ports for `HTTP/S` and `Native` differ:
 
 - HTTP - 8123
 - HTTPS - 8443
 - Native - 9000
 - Native with TLS - 9440
 
-### Manual configuration
+### Manual configuration via UI
 
-Once the plugin is installed on your Grafana instance, follow [these
-instructions](https://grafana.com/docs/grafana/latest/datasources/add-a-data-source/)
+Once the plugin is installed on your Grafana instance, follow
+[these instructions](https://grafana.com/docs/grafana/latest/datasources/add-a-data-source/)
 to add a new ClickHouse data source, and enter configuration options.
 
 ### With a configuration file
 
-It is possible to configure data sources using configuration files with
-Grafana’s provisioning system. To read about how it works, including all the
-settings that you can set for this data source, refer to [Provisioning Grafana
-data sources](https://grafana.com/docs/grafana/latest/administration/provisioning/#data-sources).
+It is possible to configure data sources using configuration files with Grafana’s provisioning system.
+To read about how it works, refer to
+[Provisioning Grafana data sources](https://grafana.com/docs/grafana/latest/administration/provisioning/#data-sources).
 
 Here are some provisioning examples for this data source using basic authentication:
 
@@ -69,27 +69,50 @@ datasources:
     jsonData:
       defaultDatabase: database
       port: 9000
-      server: localhost
+      host: localhost
       username: username
       tlsSkipVerify: false
       # tlsAuth: <bool>
       # tlsAuthWithCACert: <bool>
       # secure: <bool>
-      # timeout: <seconds>
+      # dialTimeout: <seconds>
       # queryTimeout: <seconds>
       # protocol: <native|http>
+      # defaultTable: <string>
+      # httpHeaders:
+      # - name: X-Example-Header
+      #   secure: false
+      #   value: <string>
+      # - name: Authorization
+      #   secure: true
+      # logs:
+      #   defaultDatabase: <string>
+      #   defaultTable: <string>
+      #   otelEnabled: <bool>
+      #   otelVersion: <string>
+      #   timeColumn: <string>
+      #   ...Column: <string>
+      # traces:
+      #   defaultDatabase: <string>
+      #   defaultTable: <string>
+      #   otelEnabled: <bool>
+      #   otelVersion: <string>
+      #   durationUnit: <seconds|milliseconds|microseconds|nanoseconds>
+      #   traceIdColumn: <string>
+      #   ...Column: <string>
     secureJsonData:
       password: password
       # tlsCACert: <string>
       # tlsClientCert: <string>
       # tlsClientKey: <string>
+      # secureHttpHeaders.Authorization: <string>
 ```
 
 ## Building queries
 
-The query editor allows you to query ClickHouse to return time series or
-tabular data. Queries can contain macros which simplify syntax and allow for
-dynamic parts.
+Queries can be built using the raw SQL editor or the query builder.
+Queries can contain macros which simplify syntax and allow for
+dynamic SQL generation.
 
 ### Time series
 
@@ -170,11 +193,10 @@ WHERE $__timeFilter(date_time)
 
 | Macro                                        | Description                                                                                                                                                                         | Output example                                                        |
 |----------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------|
-| *$__timeFilter(columnName)*                  | Replaced by a conditional that filters the data (using the provided column) based on the time range of the panel in seconds                                                         | `time >= '1480001790' AND time <= '1482576232' )`                     |
+| *$__timeFilter(columnName)*                  | Replaced by a conditional that filters the data (using the provided column) based on the time range of the panel in milliseconds                                                         | `time >= toDateTime64(1480001790/1000, 3) AND time <= toDateTime64(1482576232/1000, 3) )`                     |
 | *$__dateFilter(columnName)*                  | Replaced by a conditional that filters the data (using the provided column) based on the date range of the panel                                                                    | `date >= '2022-10-21' AND date <= '2022-10-23' )`                     |
-| *$__timeFilter_ms(columnName)*               | Replaced by a conditional that filters the data (using the provided column) based on the time range of the panel in milliseconds                                                    | `time >= '1480001790671' AND time <= '1482576232479' )`               |
-| *$__fromTime*                                | Replaced by the starting time of the range of the panel casted to DateTime                                                                                                          | `toDateTime(intDiv(1415792726371,1000))`                              |
-| *$__toTime*                                  | Replaced by the ending time of the range of the panel casted to DateTime                                                                                                            | `toDateTime(intDiv(1415792726371,1000))`                              |
+| *$__fromTime*                                | Replaced by the starting time of the range of the panel casted to `DateTime64(3)`                                                                                                          | `toDateTime64(1415792726371/1000, 3)`                              |
+| *$__toTime*                                  | Replaced by the ending time of the range of the panel casted to `DateTime64(3)`                                                                                                            | `toDateTime64(1415792726371/1000, 3)`                              |
 | *$__interval_s*                              | Replaced by the interval in seconds                                                                                                                                                 | `20`                                                                  |
 | *$__timeInterval(columnName)*                | Replaced by a function calculating the interval based on window size in seconds, useful when grouping                                                                               | `toStartOfInterval(toDateTime(column), INTERVAL 20 second)`           |
 | *$__timeInterval_ms(columnName)*             | Replaced by a function calculating the interval based on window size in milliseconds, useful when grouping                                                                          | `toStartOfInterval(toDateTime64(column, 3), INTERVAL 20 millisecond)` |

cspell.config.json

@@ -10,6 +10,8 @@
     "mage_output_file.go"
   ],
   "words": [
+    "concats",
+    "traceid",
     "aggregatable",
     "aheads",
     "apikey",