Sometime last week I came across a reference to DTrace; since then I have spent a not insignificant amount of time reading up on DTrace, culminating in downloading the VMware image of Nexenta, the Ubuntu of OpenSolaris, just to play with it. DTrace is quite exciting to me—it is an extension of tools like truss/strace/ktrace and ltrace that I have long-used to trace system activity and debug otherwise mysterious problems. (If you’re not familiar with strace, Self-Service Linux is a book that gives a nice introduction to system debugging using strace and other such tools; you can buy it or download a PDF for free.) It also has the ability to do basic profiling (much like pct) and is also programmable (like Linux’s inotify).

Like ktrace and friends, DTrace allows you to observe operating system user/kernel behavior but it has at least two major benefits: it has a simple programming language that lets you specify (perhaps speculatively!) and aggregate information exactly how you want it, and its extensible architecture allows events to be defined for kernel internals up through user-space middleware frameworks and high-level languages.

After reading some documentation and articles describing the cool things you could do with DTrace, I really wanted to try it: I had visions of hooking it into SFS and Chord to track down the sources of our disk and memory load with great precision. Running Solaris turned out to be easy: VMware’s Server product is “free” and Nexenta offers a pre-installed VMware image. Naturally, the problem turned out to be in getting SFS to compile on Solaris, which I have not yet done successfully.

In the mean time, I’ve just worked through some bits of a DTrace tutorial. The basic language is easy to learn; my only concern so far is that the /self->follow/ idiom for tracing “between” actions seems somewhat onerous. The hard part is internalizing the capabilities of providers and the common D idioms. The authors themselves point out other DTrace tips and caveats. However, these downsides are minor points given what you can do with the architecture.

The power DTrace offers is quite appealing. It is exciting enough that it makes me consider using Solaris, at least on a VMware Server. And perhaps more exciting, in the long-term, DTrace will get incorporated into more operating systems as evidenced by efforts to port DTrace to FreeBSD. Combined with distributed tools like Pip, debugging complex systems might just be getting easier.