Traces: default LIMIT leads to unusable trace if limited

[jwhitaker-gridcog]

For context, we are tracing a long-running computation task - it would be common to have day-long traces with 100k spans.

When using the Trace view, the default limit is 1000. If the results get limited, the resulting Trace tree is broken as you almost always end up with some child spans missing parents.

Before writing this up, I thought this was maybe tricky to do: the correct way is probably to ORDER BY "Trace Depth", but TraceDepth is not something we have and not something that can be queried without turning the thing into a giant top-down recursive CTE. Maybe you guys had thoughts on if this was doable in some other way?

Now I write this up, I think default ORDER BY Timestamp might be a good start - it will chop off spans at the end but this is quite understandable from user perspective, and at least the trace view isn't broken.

Ordering by depth would still be much better if possible though - Grafana UI gets really clunky if there are 10s of thousands of spans, so having a LIMIT is desirable. Limiting by depth would mean a LIMITed trace tree would still be quite usable.

[SpencerTorres]

I think default ORDER BY Timestamp might be a good start

I believe this is what it does by default. Also yes, transferring 100k rows of anything is going to be a lot for the frontend to load and render. Perhaps you could have some kind of pagination? Perhaps a dashboard with a variable for parentSpanId, then each span can have a link that would update the variable and refresh the trace view? In this case it could be a table and trace view side by side.

If you wanted to test this though, you could always write the query with raw SQL and your own LIMIT to see how it performs. Make sure to set the Query Type to Trace even in SQL mode. Let me know how this performs