DTrace

DTrace is a performance analysis and troubleshooting tool included by default with various operating systems, including illumos, Solaris, Mac OS X and FreeBSD. A Linux port is in development.

DTrace instruments all software.

Not just user-level software, including applications, databases and webservers, but also the operating system kernel and device drivers. The name is short for Dynamic Tracing: an instrumentation technique pioneered by DTrace which dynamically patches live running instructions with instrumentation code. The DTrace facility also supports Static Tracing: where user-friendly trace points are added to code and compiled-in before deployment.

DTrace provides a language, 'D', for writing DTrace scripts and one-liners. The language is like C and awk, and provides powerful ways to filter and summarize data in-kernel before passing to user-land. This is an important feature that enables DTrace to be used in performance-sensitive production environments, as it can greatly reduce the overhead of gathering and presenting data. ...more

There is also a pony.

DTrace for Beginners

• Tutorial: DTrace by Example
• Using DTrace to Analyze Your Webstack (short video)
• Top 10 DTrace scripts for Mac OS X
• Introducción a DTrace (in Spanish)

More DTrace

• DTrace FAQs
• Advanced DTrace Tips, Tricks and Gotchas
• DTrace: printf debugging for seventh-level wizards

Books

• DTrace Guide - now ported and updated for illumos!
• Dynamic Tracing Guide: the original reference manual, freely available as a 408 page PDF (2008)
• Solaris Performance and Tools: takes Solaris perf analysis further with DTrace, 440 pages (2006)
• DTrace: the DTrace book of scripts and strategy, 1100 pages (2011). Read a sample chapter on Brendan's blog

• • Introduction to the DTrace Book
  • What's in the DTrace Book
  • Brendan's talk at BayLISA about DTrace and the DTrace book
  • Updated DTrace book intro, DTrace book and Solaris 11

Blogs

DTrace.org:

• Adam Leventhal
• Bill Pijewski
• Brendan Gregg
• Bryan Cantrill
• Dave Pacheco
• Eric Schrock
• Robert Mustacchi

Videos on DTrace

• Bryan Cantrill's Google Tech Talk
• Breaking Down MySQL/Percona Query Latency With DTrace
• Little Shop of Performance Horrors
• DTrace BoF at LISA10
• Brendan Gregg on DTrace at the Kernel Conference Australia, 2009
• How to Build Better Applications with Oracle Solaris DTrace
• Observing Your App and Everything Else it Runs on Using DTrace
• DTracing Your Website
• The Problems Solaris Solves: Diagnosing Live Systems with DTrace