Creating flame graphs for OpenDaylight

#Intro

CPU Flame-graphs (link1, link2, link3) have been a particularly valuable tool for us in visualizing OpenDaylight execution profiles from different test scenarios. They allow holistic, interactive and accurate visualization of hot code paths, which helps in quickly identifying and understanding performance bottlenecks.

Their key difference from classic profilers lies in the conceptual and condensed way they present the most common stack traces, i.e. the list of function calls that show the code-path ancestry:

• a stack trace is represented as a stack of boxes, where each box represents a function (a stack frame).
• the y-axis shows the stack depth, ordered from root at the bottom to leaf at the top. The top box shows the function that was on-CPU when the stack trace was collected, and everything beneath that is its ancestry. The function beneath a function is its parent.
• the x-axis spans the stack trace collection. It does not show the passage of time, so the left-to-right ordering has no special meaning. The left-to-right ordering of stack traces is performed alphabetically on the function names, from the root to the leaf of each stack. This maximizes box merging: when identical function boxes are horizontally adjacent, they are merged.
• the width of each function box shows the frequency at which that function was present in the stack traces, or part of a stack trace ancestry. Functions with wide boxes were more present in the stack traces than those with narrow boxes, in proportion to their widths.
• the profile visualized may span a single thread, multiple threads, multiple applications, or multiple hosts. Separate flame graphs can be generated if desired, especially for studying individual threads.

In order to generate flame graphs, you need a profiler that can sample stack traces. For Java applications, there have historically been two types of profilers:

1. System profilers: such as Linux perf_events, which can profile system code paths, including libjvm internals, GC, and the kernel, but not Java methods.
2. JVM profilers: such as hprof, Lightweight Java Profiler (LJP), and commercial profilers. These show Java methods, but not system code paths.

The ideal would be to have one flame graph that shows it all: system and Java code together. With recent fixes in the JVM (—XX:+PreserveFramePointer), and using the perf-map-agent Java agent, one can use Linux perf_events to capture full stack traces, containing both system code paths and JVM/Java methods.

In the next sections, we will follow option 2 above and use the Java Mission Control (JMC) and Java Flight Recording (JFR) tools to create flame-graphs from OpenDaylight.

#Requirements We will need the following packages:

• Oracle JDK (for JMC/JFR)
• maven

#Steps

Download Oracle JDK

export JAVA_HOME=/home/anastop/Apps/jdk1.8.0_91/
export PATH=$JAVA_HOME/bin/:$PATH
export JAVA_OPTS="$JAVA_OPTS -XX:+UnlockCommercialFeatures"

Start ODL controller

anastop@oregon:~$ export JAVA_OPTS="$JAVA_OPTS -XX:+UnlockCommercialFeatures"
anastop@oregon:~$ jcmd
4132 sun.tools.jcmd.JCmd
2894 org.apache.karaf.main.Main
anastop@oregon:~$ jcmd 2894
Error parsing arguments: No command specified

Usage: jcmd <pid | main class> <command ...|PerfCounter.print|-f file>
   or: jcmd -l                                                    
   or: jcmd -h                                                    
                                                                  
  command must be a valid jcmd command for the selected jvm.      
  Use the command "help" to see which commands are available.   
  If the pid is 0, commands will be sent to all Java processes.   
  The main class argument will be used to match (either partially 
  or fully) the class used to start Java.                         
  If no options are given, lists Java processes (same as -p).     
                                                                  
  PerfCounter.print display the counters exposed by this process  
  -f  read and execute commands from the file                     
  -l  list JVM processes on the local machine                     
  -h  this help                                                   
anastop@oregon:~$ jcmd 2894 JFR.start name=ODL_recording settings=profile
2894:
Started recording 1. No limit (duration/maxsize/maxage) in use.

Use JFR.dump name=ODL_recording filename=FILEPATH to copy recording data to file.
anastop@oregon:~$ ls
Apps  Desktop  Downloads  mtcbench  nstat  opendaylight
anastop@oregon:~$ jcmd 2894 JFR.check
2894:
Recording: recording=1 name="ODL_recording" (running)
anastop@oregon:~$ jcmd 2894 JFR.dump name=ODL_recording filename=ODL_recording.jfr
2894:
Dumped recording "ODL_recording", 3.5 MB written to:

/home/anastop/opendaylight/distribution-karaf-0.4.1-Beryllium-SR1/ODL_recording.jfr
anastop@oregon:~$ jcmd 2894 JFR.stop name=ODL_recording
2894:
Stopped recording "ODL_recording".

git clone https://github.com/chrishantha/jfr-flame-graph.git 
git clone https://github.com/brendangregg/FlameGraph.git
cd jfr-flame-graph/
./install-mc-jars.sh 
mvn clean install -U
./run.sh -f ~/opendaylight/distribution-karaf-0.4.1-Beryllium-SR1/ODL_recording.jfr -o /tmp/output.txt
anastop@oregon:~/flamegraphs/FlameGraph$ cat /tmp/output.txt | ./flamegraph.pl > ~/odl_flamegraph.svg
anastop@oregon:~/flamegraphs/FlameGraph$ firefox !$

#Demo