Skip to content

On Profiling

Jump to bottom
Andre Merzky edited this page Mar 5, 2016 · 10 revisions

RP is a research tool first, and user tool second. To support basic CS research and experiments, RP is instrumented and tunable in ways not strictly necessary in an end user context.

This page describes 2 aspects: profiler instrumentation and profile analysis.

Profiler Instrumentation

RP Utils provides a profiler class (https://github.com/radical-cybertools/radical.pilot/blob/devel/src/radical/pilot/utils/prof_utils.py#L16). The class' main purpose is to take fine grained time events at certain points during the RP execution. An example usage is below:

import radical.pilot as rp

prof = rp.utils.Profiler('test')

prof.prof('event 1', msg='test')
prof.prof('event 2', msg='test', uid=123)
prof.prof('event 3', msg='test')
prof.close()

Running that script results in the following profile ./test.prof:

#time,name,uid,state,event,msg
0.0000,test:,,,sync abs,1457141272.44:1457141272.44:1457141272.44:sys
0.0001,test:MainThread,,,event 1,test
0.0001,test:MainThread,123,,event 2,test
0.0002,test:MainThread,123,ACTIVE,event 3,test
0.0002,test:MainThread,,,QED,

Times are relative to start of profiling. The first entry contains a reference point to the global clock (retrieved via an NTP callout to ensure global synchronization).

Each RP component creates one or more profiles. Those can be combined into a single profile in post-mortem operations. In that process, the sync timestamps are used to compute the correct time offset between events from different profiles, no matter from what resource they originate (system clock errors are corrected).

Profiler Precision

The nominal precision of the timings is 0.0001 seconds. The global syncing during recombination introduces an additional error, since NTP has a potential systematic bias for assymetric network routes. That error should not exceed a couple of milliseconds, at most. Further uncertainties arise from the nature of the implementation, ie. from python call stack overhead -- which though should be below the nominal precision by orders of magnitude.

Profiler Probes

RP has profiler probes throughout its code base, and many of them are subject to change w/o notice. Those usually inform the development process. Others are considered to be stable, and aim to support experiments. Among the latter class are specifically:

  • all unit and pilot state transitions
  • all API level object creations and destructions
  • all component creations and destructions
  • all invocations of callbacks
  • all event communications

Profile Analysis

Profiles are written in CSV format. A number of utility methods exist to retrieve, combine and convert profiles. The example in https://github.com/radical-cybertools/radical.pilot/blob/devel/examples/misc/profile_analysis.py demonstrates how profiles can be fetched, recombined, transformed into Pandas data frames, and evaluated.

Clone this wiki locally