Skip to content

Latest commit

 

History

History
101 lines (64 loc) · 2.36 KB

o11y.md

File metadata and controls

101 lines (64 loc) · 2.36 KB

Observability: Instrumentation, Tracing, Monitoring

Profile your application

$ node --prof program.<js|mjs>

$ node --prof-process isolate-*.log

Trace events in nodejs

  • [tools: chrome://tracing , ui.perfetto.dev]
$ node --trace-event-categories=v8,node.async_hooks program.<js|mjs>
$ node --trace-event-categories=v8,node.async_hooks,node,node.bootstrap,node.perf,node.threadpoolwork.sync,node.threadpoolwork.async program.<js|mjs>

Post Mortem Debugging ( Core Dump )

  • [tools: lldb, gdb, mdb, llnode]
$ node --abort-on-uncaught-exception program.<js|mjs>

$ gcore -o <filename> <pid>

Expose garbage collector for debugging memory issues

$ node --expose-gc --inspect program.<js|mjs>

Garbage Collector Traces

$ node --trace-gc program.<js|mjs>

Process Monitoring to restart the process

  • OS: systemd, upstart
  • Docker: restart
  • k8s: restartPolicy

Tools

  • Flamegraph: https://github.com/brendangregg/FlameGraph

  • perf

    • records lots of events into a file and passes that file to user space application for analysis
    • disadvantage: requires lot of disk space, post analysis will not be scalable
$ node --perf-basic-prof program.<js|mjs> # maps JIT to JS function names in perf report

$ perf record -p `pgrep -n node` -F 97 -g
# pgrep -n node: process filter
# -F 97 : frequency with which perf is going to grab samples
# -g : also include stack traces

$ perf report -f --stdio

  • eBPF ( Extended Berkeley Packet Filter ) Tracing Tools

    • Does not store or pass every events instead it samples the data
    • BPF program in kernel does the analysis in real time, samples data and outputs summary data

    tools: https://github.com/iovisor/bcc

    • offcputime: summarize off-cpu time by kernel stack trace
    $ offcputime -p `pgrep -n node` -f > perf.stacks

$ cat cpu.stacks | FlameGraph/flamegraph.pl > cpu.svg
    • fileslower: trace slow synchronous file reads and writes
    $ fileslower -p `pgrep -n node` 1 # traces reads/writes slower than 1 ms

  • tplist: lists available USDT probes for that process

$ tplist -p `pgrep -n node` # gives list probes that exists in node process
  • funclatency: Attaches itself to a particular function and gives summary
$ funclatency "`which node`:*PerformGarbageCollection*" -m