VMware Workstation 6.5: Reverse and Replay Debugging is Here!

I'm proud to announce that VMware Workstation 6.5 includes new experimental features that provide replay debugging for C/C++ developers using Microsoft Visual Studio. Replay debugging allows developers to debug recordings of programs running in virtual machines, and it is valuable for finding, diagnosing, and fixing bugs that are not easily reproduced, a particularly challenging class of bugs. Once the manifestation of a bug has been recorded, it can be replayed (and debugged) over and over again, and it is guaranteed to have instruction-by-instruction identical behavior each time. In addition, Workstation includes a feature that simulates reverse execution of the program, making it easier to pin point the origin of a bug.

Question: How can replay debugging help me?

Answer: Replay debugging helps find, diagnose, and fix bugs that are not easily reproduced. Consider the following examples:

Non-deterministic bugs. Some bugs only rear their ugly heads when a particular interleaving of concurrent entities (e.g., threads, the operating system, network activity, and user input) is present. Given that the programmer cannot normally control (or even reason about) this interleaving, these bugs can be very difficult to reproduce. But if such a bug is recorded, we can examine it in the debugger as often as we like. Once a bug is recorded, it is no longer non-deterministic.

Pseudo non-deterministic bugs. Some bugs are deterministic, but it's not always clear exactly what is necessary to reproduce the bug. Bug reports indicating that a program crashed "...after clicking on a bunch of different UI elements," are common but usually useless. If the bug is recorded, it is unnecessary to describe the steps for reproduction, because they are an implicit part of the recording itself.

Bugs that can only be reproduced with a complex environment. If a bug appears in an application that communicates with dozens of other local and remote processes that need to be setup and configured, reproducing the bug may be possible but prohibitively expensive. If the the bug is recorded, the developer need not setup the other components of the system (e.g., other processes or remote clients). The effect of the other components is recorded, allowing the program of interest to be debugged in isolation.

Heisenbugs. A Heisenbug (a play on the Heisenberg uncertainly principle) is a bug that does not manifest itself when it is examined in the debugger. Such bugs are common because debugging is an invasive process that dramatically changes the timing characteristics of the program. If a bug is recorded, it can be debugged without impacting the behavior of the program at all.

Memory corruption bugs. Memory corruption bugs are difficult to diagnose because almost any pointer dereference in the program can be responsible for the corruption. The replay debugging facility in VMware Workstation 6.5 includes a reverse execution feature that (among other things) greatly helps track down memory corruption bugs. When a corrupted data structure is identified, the developer simply sets a data breakpoint (i.e., a watchpoint) on the corrupted data and executes backward. This will take the developer to the last point in time that the corrupted data structure was modified.

Question: Do I have to learn a new debugging environment?

Answer: No, our replay debugging features are tightly integrated with the Visual Studio development environment. Aside from the replay-specific features, your debugging experience will be the same as a traditional debugging experience.

Question: How do I create recordings?

Answer: Our extension to Visual Studio provides a facility for creating recordings from within Visual Studio. Alternatively, recordings can be created from the VMware Workstation user interface. The later is useful when someone other than the developer (e.g., a tester) is creating recordings.

Question: How can I get it?

Answer: A release candidate for VMware Workstation 6.5 is currently available here. The release candidate is available to members of the beta program (free signup). The release version of the product will be available soon. Workstation 6.5 is a free upgrade for all Workstation 6.0 license holders.

Question: Where can I learn more?

Answer: We have created a number of resources to get you up and running with replay debugging.

Introductory video. This video introduces the replay debugging feature.

Tutorial screen cast. [Coming Soon!] This screencast shows the configuration and use of replay debugging.

Replay debugging forum. This form is a place for users and VMware engineers to get together to discuss replay debugging experiences, problems, and requests.

This blog. This blog will be updated with topics related to replay debugging. I expect to add articles on use cases, best practices, pitfalls, and so on.

VMware Workstation 6.5 Manual. The Workstation manual includes an appendix describing the Integrate Virtual Debugger for Visual Studio, which includes the replay debugging features. This is a useful reference but it does not include much more than the basic facts.