Creating flame graphs for OpenDaylight

#Intro

CPU Flame-graphs (link1, link2, link3) have been a particularly valuable tool for us in visualizing OpenDaylight execution profiles from different test scenarios. They allow holistic, interactive and accurate visualization of hot code paths, which helps in quickly identifying and understanding performance bottlenecks.

Their key difference from classic profilers lies in the conceptual and condensed way they present the most common stack traces, i.e. the list of function calls that show the code-path ancestry:

• a stack trace is represented as a stack of boxes, where each box represents a function (a stack frame).
• the y-axis shows the stack depth, ordered from root at the bottom to leaf at the top. The top box shows the function that was on-CPU when the stack trace was collected, and everything beneath that is its ancestry. The function beneath a function is its parent.
• the x-axis spans the stack trace collection. It does not show the passage of time, so the left-to-right ordering has no special meaning. The left-to-right ordering of stack traces is performed alphabetically on the function names, from the root to the leaf of each stack. This maximizes box merging: when identical function boxes are horizontally adjacent, they are merged.
• the width of each function box shows the frequency at which that function was present in the stack traces, or part of a stack trace ancestry. Functions with wide boxes were more present in the stack traces than those with narrow boxes, in proportion to their widths.
• the profile visualized may span a single thread, multiple threads, multiple applications, or multiple hosts. Separate flame graphs can be generated if desired, especially for studying individual threads.

In order to generate flame graphs, you need a profiler that can sample stack traces. For Java applications, there have historically been two types of profilers:

1. System profilers: such as Linux perf_events, which can profile system code paths, including libjvm internals, GC, and the kernel, but not Java methods.
2. JVM profilers: such as hprof, Lightweight Java Profiler (LJP), and commercial profilers. These show Java methods, but not system code paths.

The ideal would be to have one flame graph that shows it all: system and Java code together. With recent fixes in the JVM (—XX:+PreserveFramePointer), and using the perf-map-agent Java agent, one can use Linux perf_events to capture full stack traces, containing both system code paths and JVM/Java methods.

In the next sections, we will follow option 2 above and use the Java Mission Control (JMC) and Java Flight Recording (JFR) tools to create flame-graphs from OpenDaylight.

#Requirements We will need the following:

• Oracle JDK (for JMC/JFR)

#Steps

#Demo