Skip to content

Commit

Permalink
chore: add openGemini v1.3.0 release note (#138)
Browse files Browse the repository at this point in the history 
Signed-off-by: xiangyu5632 <[email protected]>
  • Loading branch information
@xiangyu5632
xiangyu5632 authored Nov 17, 2024
1 parent cc6962d commit b9205cf
Show file tree
Hide file tree
Showing 4 changed files with 342 additions and 199 deletions.
140 changes: 41 additions & 99 deletions src/guide/versions/Release_notes_1.3.md
View file
Original file line number Diff line number Diff line change
@@ -1,126 +1,68 @@
---
title: Release notes(v1.3)
order: 1
title: Release notes(v1.3.0)
order: 2
---

## Features
openGemini v1.3.0 further improves the stability and reliability of cluster data replica based on v1.3.0-rc.1 and fixes multiple bugs found in v1.2.0.

1. **Use PromQL to query data directly from openGemini**
## Future

PromQL is a powerful, concise querying language for Prometheus.
1. `SHOW MEASUREMENTS` support *`limit`* and *`offset`* clauses

![image-20240828100203674](../../../static/img/guide/version/image-20240828100203674.png)

**Examples**

This example selects all time series that have the `http_requests_total` metric name:

```shell
// PromQL
http_requests_total

// openGemini HTTP API
curl -i -XPOST 'http://127.0.0.1:8086/api/v1/query?query=http_requests_total'
```

This example selects only those time series with the `http_requests_total` metric name that also have the `job` label set to `apiserver` and their `handler` label set to `/api/comments`:

```shell
// PromQL
http_requests_total{job="apiserver", handler="/api/comments"}

// openGemini HTTP API
curl -i -XPOST 'http://127.0.0.1:8086/api/v1/query?' --data-urlencode 'query=http_requests_total{job="apiserver", handler="/api/comments"}'
```

This example select all the values we have recorded within the last 5 minutes for all time series that have the metric name `http_requests_total` and a `job` label set to `apiserver`:

```shell
// PromQL
http_requests_total{job="apiserver}[5m]
// openGemini HTTP API
curl -i -XPOST 'http://127.0.0.1:8086/api/v1/query?' --data-urlencode 'query=http_requests_total{job="apiserver"}[5m]'
```
Related Reading [Use PromQL](https://docs.opengemini.org/guide/query_data/promql.md)
2. **Data Replication**
In openGemini v1.2.0 and earlier versions, data has no data replication. However, starting from v1.3.0, openGemini supports data replications, meeting the data reliability requirements of most scenarios such as industry, energy, and the Internet of Things.
![image-20240828101727172](../../../static/img/guide/version/image-20240828101727172.png)
Related Reading [Data Replication](https://docs.opengemini.org/guide/features/replication.md)
3. **Backup & Restore**
Both stand-alone and clustered support data backup, including full backup and incremental backup. In addition, we also support offline data recovery, which can be restored to the original node or to a new node. Data is backed up locally, and backup to a remote backup server is not supported.
![image-20240828105008977](../../../static/img/guide/version/image-20240828105008977.png)
Related Reading [Backup & Restroe](https://docs.opengemini.org/guide/maintenance/back_restore.html)
4. **Support Prometheus to pull runtime monitor metrics data from openGemini**
```shell
curl -i -XPOST 'http://127.0.0.1:8086/metrics'
```
5. **A new command `show cluster` for get cluster status**
2. A new command, ‘show measurements detail with measurement = xxx’, is added to view the **Index**, **Shard key**, **Engine Type**, **Tag Keys**, and **Field keys** of measurement. Previously, at least 3 or 4 commands were required to query the above information.

```sql
> show cluster
time status hostname nodeID nodeType availability
---- ------ -------- ------ -------- ------------
1725071376777471503 alive 127.0.0.3:8091 1 meta available
1725071376777471503 alive 127.0.0.1:8091 2 meta available
1725071376777471503 alive 127.0.0.2:8091 3 meta available
1725071376777471503 alive 127.0.0.1:8400 4 data available
1725071376777471503 alive 127.0.0.2:8400 5 data available
1725071376777471503 alive 127.0.0.3:8400 6 data available
> show measurements detail with measurement = mst
name: mst
tags: Retention policy=rp0
+--------------------------------+
| Detail |
+--------------------------------+
| INDEX: <nil> |
| SHARD KEY: <nil> |
| ENGINE TYPE: tsstore |
| TAG KEYS: country, name |
| FIELD KEYS: address(string), |
| age(float), alive(boolean), |
| height(integer) |
+--------------------------------+
1 columns, 5 rows in set
```

## Fix Bug

1. In v1.2.0, ts-meta has a memory leak. After running for a period of time, its memory usage will be obviously abnormal.

## Performance
*--- This problem has been fixed in v1.3.0. It is recommended that you upgrade as soon as possible.*

1. **Topn() performance optimization **
2. In v1.2.0, the data subscription function will cause abnormal data to be written to the target database, such as adding abnormal column data.

Optimize the time range query scenarios. for example `select topn(http_requests_total, 10) from mst1 where time >=now()-2h` , In practical application, the average latency of 100 concurrent queries is reduced from 256ms to 68ms, and the end-to-end performance is improved by 2 to 4 times.
*--- This problem has been fixed in v1.3.0*

2. **Data writing optimization**
3. In v1.2.0, the query result data may be inaccurate when querying with `limit=xx & order by desc` conditions.

For data writing scenarios where the number of TAGs is huge, the TAG Value is particularly long, or the TAG array is relatively large, the CPU overhead can be reduced by 20% by optimizing WAL network transmission, Cache compression, and index update code logical, etc. In practical application, write performance increased from 370,000 rows/s to 600,000 rows/s
*--- This problem has been fixed in v1.3.0*

Configuration
4. In v1.2.0, the following query scenarios may cause process panic.

```toml
[index]
cache-compress-enable = true
bloom-filter-enable = true
```
3. **Batch query optimization**
Scenario: `select count(*) from (select field1 where tag1='t1' and field2='f2')`

*--- This issue has been fixed in v1.3.0*

For batch query scenarios, such as dashboard applications, most query conditions are the same. By caching the inverted index query results, In practical application, 10 million time series , the batch query performance is improved by nearly 100%.
5. In v1.3.0-rc.1, in a cluster data replicas scenario, killing one of the ts-stores may cause large fluctuations in write latency.

4. **`show tag values` performance optimization**
*--- This issue has been fixed in v1.3.0*

Optimizing query strategies can improve performance by 5-10 times when querying the tag values with high cardinality and limits.
6. In v1.3.0-rc.1, in a multi-copy scenario, if one of the three ts-store nodes loses power or loses network connection, it will cause write timeouts for a long time.

5. **Memtable query performance optimization**
*--- This issue has been fixed in v1.3.0*

The massive time series real-time data make memtable cache many data. At the same time, there have a large concurrent query, which leads to high CPU and poor query performance. By reducing data deep copies and GC optimization, the query performance can be further improved.
7. In v1.3.0-rc.1 stand-alone , writing data may cause process panic when using the text index

6. **Multi-Field query optimization**
*--- This issue has been fixed in v1.3.0*

In scenarios where there are many field filters in the query conditions, there will be problems such as high query latency and high CPU overhead. By optimizing the index scanning algorithm, the CPU consumption in the Index Scan stage of multi-field scanning scenarios is reduced from 75.36% to 0.53%.
## Special Note

## Bug Fix
1. openGemini v1.3.0 is compatible with v1.2.0 and can be upgraded directly from v1.2.0 to v1.3.0 without data replicas.

1. Fixed the issue that RP cannot be specified when deleting a table #666
2. Fixed the issue that occurring some failed when deleting measurements in batches #482
3. Fixed the issue that the command line text was wrong when viewing the history commands in ts-cli #651
4. Fix ts-store restart panic #481
5. Fixed the issue of text index file loading failure #700
2. The cluster data replica write performance of openGemini v1.3.0 is not good enough, and the community is working on optimizing it. We do not recommend using this feature for projects with high-performance requirements.
126 changes: 126 additions & 0 deletions src/guide/versions/Release_notes_1.3.rc.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,126 @@
---
title: Release notes(v1.3-rc.1)
order: 1
---

## Features

1. **Use PromQL to query data directly from openGemini**

PromQL is a powerful, concise querying language for Prometheus.

![image-20240828100203674](../../../static/img/guide/version/image-20240828100203674.png)

**Examples**

This example selects all time series that have the `http_requests_total` metric name:

```shell
// PromQL
http_requests_total

// openGemini HTTP API
curl -i -XPOST 'http://127.0.0.1:8086/api/v1/query?query=http_requests_total'
```

This example selects only those time series with the `http_requests_total` metric name that also have the `job` label set to `apiserver` and their `handler` label set to `/api/comments`:

```shell
// PromQL
http_requests_total{job="apiserver", handler="/api/comments"}

// openGemini HTTP API
curl -i -XPOST 'http://127.0.0.1:8086/api/v1/query?' --data-urlencode 'query=http_requests_total{job="apiserver", handler="/api/comments"}'
```

This example select all the values we have recorded within the last 5 minutes for all time series that have the metric name `http_requests_total` and a `job` label set to `apiserver`:

```shell
// PromQL
http_requests_total{job="apiserver}[5m]
// openGemini HTTP API
curl -i -XPOST 'http://127.0.0.1:8086/api/v1/query?' --data-urlencode 'query=http_requests_total{job="apiserver"}[5m]'
```
Related Reading [Use PromQL](https://docs.opengemini.org/guide/query_data/promql.md)
2. **Data Replication**
In openGemini v1.2.0 and earlier versions, data has no data replication. However, starting from v1.3.0, openGemini supports data replications, meeting the data reliability requirements of most scenarios such as industry, energy, and the Internet of Things.
![image-20240828101727172](../../../static/img/guide/version/image-20240828101727172.png)
Related Reading [Data Replication](https://docs.opengemini.org/guide/features/replication.md)
3. **Backup & Restore**
Both stand-alone and clustered support data backup, including full backup and incremental backup. In addition, we also support offline data recovery, which can be restored to the original node or to a new node. Data is backed up locally, and backup to a remote backup server is not supported.
![image-20240828105008977](../../../static/img/guide/version/image-20240828105008977.png)
Related Reading [Backup & Restroe](https://docs.opengemini.org/guide/maintenance/back_restore.html)
4. **Support Prometheus to pull runtime monitor metrics data from openGemini**
```shell
curl -i -XPOST 'http://127.0.0.1:8086/metrics'
```
5. **A new command `show cluster` for get cluster status**
```sql
> show cluster
time status hostname nodeID nodeType availability
---- ------ -------- ------ -------- ------------
1725071376777471503 alive 127.0.0.3:8091 1 meta available
1725071376777471503 alive 127.0.0.1:8091 2 meta available
1725071376777471503 alive 127.0.0.2:8091 3 meta available
1725071376777471503 alive 127.0.0.1:8400 4 data available
1725071376777471503 alive 127.0.0.2:8400 5 data available
1725071376777471503 alive 127.0.0.3:8400 6 data available
```
## Performance
1. **Topn() performance optimization **
Optimize the time range query scenarios. for example `select topn(http_requests_total, 10) from mst1 where time >=now()-2h` , In practical application, the average latency of 100 concurrent queries is reduced from 256ms to 68ms, and the end-to-end performance is improved by 2 to 4 times.
2. **Data writing optimization**
For data writing scenarios where the number of TAGs is huge, the TAG Value is particularly long, or the TAG array is relatively large, the CPU overhead can be reduced by 20% by optimizing WAL network transmission, Cache compression, and index update code logical, etc. In practical application, write performance increased from 370,000 rows/s to 600,000 rows/s
Configuration
```toml
[index]
cache-compress-enable = true
bloom-filter-enable = true
```
3. **Batch query optimization**
For batch query scenarios, such as dashboard applications, most query conditions are the same. By caching the inverted index query results, In practical application, 10 million time series , the batch query performance is improved by nearly 100%.
4. **`show tag values` performance optimization**
Optimizing query strategies can improve performance by 5-10 times when querying the tag values with high cardinality and limits.
5. **Memtable query performance optimization**
The massive time series real-time data make memtable cache many data. At the same time, there have a large concurrent query, which leads to high CPU and poor query performance. By reducing data deep copies and GC optimization, the query performance can be further improved.
6. **Multi-Field query optimization**
In scenarios where there are many field filters in the query conditions, there will be problems such as high query latency and high CPU overhead. By optimizing the index scanning algorithm, the CPU consumption in the Index Scan stage of multi-field scanning scenarios is reduced from 75.36% to 0.53%.
## Bug Fix
1. Fixed the issue that RP cannot be specified when deleting a table #666
2. Fixed the issue that occurring some failed when deleting measurements in batches #482
3. Fixed the issue that the command line text was wrong when viewing the history commands in ts-cli #651
4. Fix ts-store restart panic #481
5. Fixed the issue of text index file loading failure #700
Loading

0 comments on commit b9205cf

Please sign in to comment.