Flight client is too slow in Monolith OSS & PRO

[MaduMitha-Ravi]

Problem Statement

On Monolith OSS, while doing Performance runs we have observed that the Comparison between CURL and Flight query performance raises the doubt whether there could be an underlying bug with the Flight implementation in Monolith as the Flight latency is way too slow.

Expected Results

Flight naturally will be slower than Curl but 2x times and in some cases 10x times are not expected.

Metrics to support the identification of the problem

Below are the metrics captured for various queries with curl and flight,

• Most of the queries irrespective of the query ranges report being about 2x times faster with curl
• last value cache shows nearly 10x better performance with curl

Query load (concurrency = 1)	OSS - 5m query range(P95 latency in ms)	OSS - 5m query range(P95 latency in ms)	OSS - 10m query range(P95 latency in ms)	OSS - 10m query range(P95 latency in ms)	OSS - 60m query range(P95 latency in ms)	OSS - 60m query range(P95 latency in ms)
Client	flight	curl	flight	curl	flight	curl
c13	125.4	46.69	127.62	47.46	129.48	48.76
c14	89.5	45.01	90.28	58.48	122.19	47.38
f1	78.24	44.65	94.27	60.41	98.48	63.21
lastcachevalue_1	89.45	8.62	90.23	9	92.23	10.5
lastcachevalue_10	94.34	10.62	90.34	8.62	90.06	8.43

Reach out to @MaduMitha-Ravi for more details/metrics.

[hiltontj]

@MaduMitha-Ravi any chance the code that uses the Flight client can be shared?

[MaduMitha-Ravi]

@adrian-thurston Can you share please?

[adrian-thurston]

Hi @hiltontj, code is here:

https://github.com/influxdata/perf-tools/blob/b460bba302257635a8724c9eb60cebb6bc9d556f/ic/query/query.go#L1154

[MaduMitha-Ravi]

fyi - Updated the title, the slowness is observed in OSS and PRO.

[praveen-influx]

@MaduMitha-Ravi - I've created an experimental branch to identify the bottle neck. You can see the actual changes in this draft PR. We have a mechanism in monolith to use the same port for both HTTP/GRPC (flightsql), but in my branch I've made them listen on separate ports (8181 for HTTP by default and 8183 for GRPC by default). I'd like to know if the difference in performance between flightsql client and curl can still be observed. Thanks