Skip to content

Commit

Permalink
Delete unnecessary information.
Browse files Browse the repository at this point in the history
  • Loading branch information
dongwenxin1 committed Jul 27, 2020
1 parent 7304626 commit 1f474c0
Show file tree
Hide file tree
Showing 11 changed files with 14 additions and 22 deletions.
6 changes: 3 additions & 3 deletions sql/xsql/docs/docs/datasources/druid.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -14,8 +14,8 @@ Druid接入XSQL的配置继承了[Configurations](../configurations/common.md)
```
xsql.conf文件:
spark.xsql.datasource.mydruid.type DRUID
spark.xsql.datasource.mydruid.uri http://r883.dfs.shbt.qihoo.net:8082
spark.xsql.datasource.mydruid.coordinator.uri r883.dfs.shbt.qihoo.net:8081
spark.xsql.datasource.mydruid.uri http://druidhostname:8082
spark.xsql.datasource.mydruid.coordinator.uri druidhostname:8081
spark.xsql.datasource.mydruid.user xxxx
spark.xsql.datasource.mydruid.password xxxx
spark.xsql.datasource.mydruid.version 0.10.1
Expand Down Expand Up @@ -238,4 +238,4 @@ TAKE BACK RETURN 382377564 749815.1810177844 14995638
COLLECT COD 382410465 749926.601011185 14995241
DELIVER IN PERSON 382437327 749621.3410196826 14994611
NONE 382512680 750010.1210143827 15000562
```
```
2 changes: 1 addition & 1 deletion sql/xsql/docs/docs/datasources/hbase.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ HBase是一个分布式的、面向列的开源数据库，旨在为非结构化
spark.xsql.datasources hbase_ds_name
spark.xsql.default.datasource hbase_ds_name
spark.xsql.datasource.hbase_ds_name.type hbase
spark.xsql.datasource.hbase_ds_name.host jlxx.sys.lyct.qihoo.net,jlxx.sys.lyct.qihoo.net,jlxx.sys.lyct.qihoo.net
spark.xsql.datasource.hbase_ds_name.host hostname1,hostname2,hostname3
spark.xsql.datasource.hbase_ds_name.port 2181
# 配置元数据存储文件名称，需要放置在SPARK_CONF_DIR中
spark.xsql.datasource.hbase_ds_name.schemas hbase.schemas
Expand Down
3 changes: 1 addition & 2 deletions sql/xsql/docs/docs/performance_report/elasticsearch.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,6 @@ Elasticsearch的性能测试报告分为基于TPCDS的性能测试报告和基
​
**虚拟机版本**： Java HotSpot(TM) 64-Bit Server VM (build 25.60-b23, mixed mode)
​
**测试机器**： client01v.qss.zzzc.qihoo.net、clientadmin.dfs.shbt.qihoo.net

**XSQL配置**：

Expand Down Expand Up @@ -157,4 +156,4 @@ Elasticsearch的性能测试报告分为基于TPCDS的性能测试报告和基
### **结论**

- XSQL [Pushdown]相比于直接调用Elasticsearch的API，执行性能仅有约30毫秒的损耗。
- XSQL借助于Spark执行时，执行效率很低。
- XSQL借助于Spark执行时，执行效率很低。
1 change: 0 additions & 1 deletion sql/xsql/docs/docs/performance_report/hbase.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,6 @@ HBase的性能测试报告主要是基于TPCDS的性能测试报告。
​
**虚拟机版本**： Java HotSpot(TM) 64-Bit Server VM (build 25.60-b23, mixed mode)
​
**测试机器**： client01v.qss.zzzc.qihoo.net、clientadmin.dfs.shbt.qihoo.net

**XSQL配置**：

Expand Down
3 changes: 1 addition & 2 deletions sql/xsql/docs/docs/performance_report/hive.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,6 @@ Hive的性能测试报告分为基于TPCDS的性能测试报告和基于业务
​
**虚拟机版本**： Java HotSpot(TM) 64-Bit Server VM (build 25.60-b23, mixed mode)
​
**测试机器**： client01v.qss.zzzc.qihoo.net、clientadmin.dfs.shbt.qihoo.net

**XSQL配置**：

Expand Down Expand Up @@ -146,4 +145,4 @@ Hive的性能测试报告分为基于TPCDS的性能测试报告和基于业务
### **结论**

- XSQL相比于Hive，执行性能得到明显的提升。
- 子查询、连接操作，XSQL配给Executor的内存多少，对执行时间也有影响。
- 子查询、连接操作，XSQL配给Executor的内存多少，对执行时间也有影响。
1 change: 0 additions & 1 deletion sql/xsql/docs/docs/performance_report/mongo.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,6 @@ MongoDB的性能测试报告分为基于TPCDS的性能测试报告和基于业
​
**虚拟机版本**： Java HotSpot(TM) 64-Bit Server VM (build 25.60-b23, mixed mode)
​
**测试机器**： client01v.qss.zzzc.qihoo.net、clientadmin.dfs.shbt.qihoo.net

**XSQL配置**：

Expand Down
3 changes: 1 addition & 2 deletions sql/xsql/docs/docs/performance_report/multi_datasource.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,6 @@
​
**虚拟机版本**： Java HotSpot(TM) 64-Bit Server VM (build 25.60-b23, mixed mode)
​
**测试机器**： client01v.qss.zzzc.qihoo.net、clientadmin.dfs.shbt.qihoo.net

**XSQL配置**：

Expand Down Expand Up @@ -116,4 +115,4 @@

### 结论

上图分析可知，执行ElasticSearch与MySQL的混合查询时，两个数据源下推的执行效率要高于非下推的执行效率；尤其对于ES的执行，非下推时执行很慢，并且有时子查询执行会出现超时错误。
上图分析可知，执行ElasticSearch与MySQL的混合查询时，两个数据源下推的执行效率要高于非下推的执行效率；尤其对于ES的执行，非下推时执行很慢，并且有时子查询执行会出现超时错误。
4 changes: 1 addition & 3 deletions sql/xsql/docs/docs/performance_report/mysql.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,6 @@ MySQL性能测试报告是基于业务数据的性能测试报告。

**虚拟机版本**： Java HotSpot(TM) 64-Bit Server VM (build 25.60-b23, mixed mode)

**测试机器**： client01v.qss.zzzc.qihoo.net

**xsql配置**：

- Driver Memory： 5G
Expand Down Expand Up @@ -125,4 +123,4 @@ MySQL性能测试报告是基于业务数据的性能测试报告。

**Note**

以上结论的得出受数据量、SQL语句及测试环境的影响，仅供参考。
以上结论的得出受数据量、SQL语句及测试环境的影响，仅供参考。
3 changes: 1 addition & 2 deletions sql/xsql/docs/docs/performance_report/redis.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,6 @@ Redis的性能测试报告主要是基于TPCDS的性能测试报告。
​
**虚拟机版本**： Java HotSpot(TM) 64-Bit Server VM (build 25.60-b23, mixed mode)
​
**测试机器**： client01v.qss.zzzc.qihoo.net、clientadmin.dfs.shbt.qihoo.net

**XSQL配置**：

Expand Down Expand Up @@ -51,4 +50,4 @@ Redis的性能测试报告主要是基于TPCDS的性能测试报告。

- 对于点查询，使用jedis api查询用时为0.003s，使用xsql查询用时约为0.05s。
- 对于scan类型的查询，使用jedis api和xsql的查询性能均不理想，用时估算方法为：每1万/延迟3s。
- 更新：采用pipeline方法后，用时估算方法为第1万/延迟0.3s。
- 更新：采用pipeline方法后，用时估算方法为第1万/延迟0.3s。
8 changes: 4 additions & 4 deletions sql/xsql/docs/docs/troubleshooting/common.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -21,12 +21,12 @@
queue: root.test
start time: 1544780544655
final status: UNDEFINED
tracking URL: http://test.qihoo.net:8888/proxy/application_1543893582405_838478/
tracking URL: http://testhostname:8888/proxy/application_1543893582405_838478/
user: test
18/12/14 17:42:32 INFO Client: Application report for application_1543893582405_838478 (state: ACCEPTED)
```

其中的tracking URL为http://test.qihoo.net:8888/proxy/application_1543893582405_838478/，从浏览器打开页面将看到类似信息：
其中的tracking URL为http://testhostname:8888/proxy/application_1543893582405_838478/，从浏览器打开页面将看到类似信息：

```properties
User: test
Expand All @@ -46,7 +46,7 @@ Diagnostics:

可以看到状态也是ACCEPTED。并且队列是root.test。

打开http://test.qihoo.net:8888/cluster/scheduler?openQueues=root.test，找到root.test队列的资源，将看到如下信息：
打开http://testhostname:8888/cluster/scheduler?openQueues=root.test，找到root.test队列的资源，将看到如下信息：

```properties
Used Resources: <memory:799232, vCores:224, gCores:0>
Expand Down Expand Up @@ -168,7 +168,7 @@ ERROR executor.CoarseGrainedExecutorBackend: RECEIVED SIGNAL 15: SIGTERM

```
Job aborted due to stage failure: Task 2 in stage 3.0 failed 4 times, most recent failure:
Lost task 2.3 in stage 3.0 (TID 28, hpc152.sys.lycc.qihoo.net, executor 11):
Lost task 2.3 in stage 3.0 (TID 28, hpchostnam, executor 11):
org.apache.spark.SparkException: Kryo serialization failed: Buffer overflow. Available: 2, required: 8
```
合理设置spark.kryoserializer.buffer.max,spark.kryoserializer.buffer
Expand Down
2 changes: 1 addition & 1 deletion sql/xsql/docs/docs/tutorial/configuration.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -174,7 +174,7 @@ yarn-cluster0.conf文件中的配置信息可能为：
```properties
spark.yarn.stagingDir hdfs://namenode.dfs.cluster0.yahoo.com:9000/home/spark/cache
spark.hadoop.yarn.resourcemanager.cluster-id cluster0-yarn
spark.hadoop.yarn.resourcemanager.zk-state-store.address m2.dfs.cluster0.qihoo.net:2181,m3.dfs.cluster0.yahoo.com:2181,m4.dfs.cluster0.yahoo.com:2181,m5.dfs.cluster0.yahoo.com:2181,m6.dfs.cluster0.yahoo.com:2181
spark.hadoop.yarn.resourcemanager.zk-state-store.address m3.dfs.cluster0.yahoo.com:2181,m4.dfs.cluster0.yahoo.com:2181,m5.dfs.cluster0.yahoo.com:2181,m6.dfs.cluster0.yahoo.com:2181
spark.hadoop.yarn.resourcemanager.zk-address m2.dfs.cluster0.yahoo.com:2181,m3.dfs.cluster0.yahoo.com:2181,m4.dfs.cluster0.yahoo.com:2181,m5.dfs.cluster0.yahoo.com:2181,m6.dfs.cluster0.yahoo.com:2181
spark.hadoop.yarn.resourcemanager.zk-state-store.parent-path /cluster0/yarn/rmstore
spark.hadoop.yarn.resourcemanager.hostname.rm1 m7.dfs.cluster0.yahoo.com
Expand Down

0 comments on commit 1f474c0

Please sign in to comment.