View Key Metrics

[dblooman]

It would be great to at a glance see the total number of partitions, topics, size of disk used, out of sync replications etc.

[tchiotludo]

Clearly, it's a must have I think.
Just never have the time to do it.

PR are welcome are !
Maybe also some wireframe or specifications can be defined to help ?

[theesen]

Hello,
I found this issue interesting and wanted to suggest a solution. Created a draft PR.

There are a few things I would still like to change myself. F.e. summing up kafka metrics smoother in the API.

I am recycling the table list for the cluster view (with just one row). I am not sure if you think this is a good idea though.

[tchiotludo]

hey @theesen ! Thanks for this !
Good idea & good start.

To be honest, I was thinking about a true homepage with all the metrics (topics / consumer groups / ...)
The main goals will be to have async call on this page (to displayed information as soon as possible) since the topic can be longer to load than other one.
Maybe you can continue with that ?

What do you think about that ?

[theesen]

Yeah, I think that would be better as well. It felt a bit off, not to have a "landing page" for clusters.

Do you have a sketch of what you have in mind roughly?

I think it would be nice to have a "tiled" view. Each tile being a cluster with the basic metrics summed up. If you click on the metrics it forwards you to the detail page (topics, connectors, etc..).

[tchiotludo]

mostly for now, I think about a home per cluster (not cross cluster) for the first one (multiple cluster can be a big deal with acls).
I don't have a sketch but I will try to explain 😄 A tiled view is exactly what is needed !

Each tile is a group of metrics, here is thought about that:

• Cluster :

  • Number of node

• Topic:

  • Number of topics
  • Number of partitions
  • Number of replica total
  • Number of out of sync partition

• Consumer Group

  • Number of consumer group
  • Number of rebalancing group
  • Number of empty group

• Connect

  • Number of connector
  • Number of tasks
  • Number of running tasks
  • Number of failed tasks

• Schema

  • Number of schema

Maybe this metrics can be also on the top of detail page ?

What do you think about that ?

[theesen]

Sounds good to me. I have started to make the adjustments for the backend. Hopefully be done with that by Monday. Before I start adjusting the frontend, write unit tests etc., I think it'd be good if you could evaluate the backend part. Would save me some refactoring later on.
Edit: I am not very experienced with JavaScript, so I will try to make a suggestion but it'll probably require some fine tuning :)

[tchiotludo]

No worry, there is no emergency !
Also, you are not obliged to implement all the metrics, do as you can.

The backend seems to be good on my side, just avoid the log.info("Cluster Stats retrieved: {}", stats); on final one.
From the frontend, If you have some issue I can complete this part by myself, no worry again

[theesen]

I have updated the PR with a initial draft of the backend metrics for the Cluster view.
Issues to tackle next:

• Remove unecessary Logging
• Didn't understand what @tchiotludo meant by rebalancing group and empty group. Left it empty until clarification
• Refactor loops over lists a bit further.
• Write tests
• Create Cluster View Page
• Split up routes per metric block to enable async calls for each

Hope you like the structure, feedback is very welcome 👍

[tchiotludo]

Overall, seems to be good, but take, you will need to have a route for every type of stats.
We really want async behavior here, if the topic stats is ready display it as soon as possible and don't wait for others one.
We have very large cluster implementation with for example few topics and many consumer groups.
If we need to compute sync all of these, it will never load on large cluster, so the react app will call both of them.

Do you see what I mean ?

[theesen]

Yup, good point. Will split up the routes for each metric block.

[tchiotludo]

@dblooman don't want to work on this anymore?

[dblooman]

Moved onto other projects, so no unfortunately

[tchiotludo]

No problems, which one for my curiosity?