Preparing to Replay Debug. . .

This blog post provides some basic info about how to set up your host/guest/VS for replay debugging. Please note that things work a bit differently in WS 7 versus 6.5. Failure to follow these steps will likely result in an unproductive replay debugging experience. But it'll be worth it! Here we summarize each section.

• Preparing the Host and Guest. This section describes how you must prepare your host and guest systems for replay debugging. It is critical that you carefully read and obey this section.
• Configuring Visual Studio. This section describes how Visual Studio must be configured to use replay debugging. Note that this section contains only enough information to get started. Please see the Manual for a more complete treatment of the many ways Visual Studio can be configured for replay debugging.
• Creating a Recording. Before you can replay and debug a recording, you must create a recording. This section describes several ways that this can be achieved.
• Starting Replay Debugging. Once you have a recording and Visual Studio has been configured to be aware of it, you can debug programs running in this recording. This section describes how to kick off a replay debugging session.
• Why All the Preparing? This section helps you understand why the host and guest must be prepared for replay debugging.

Preparing the Host and Guest

You cannot simply use replay debugging out of the box. You must prepare your host and guest systems first. Fortunately, none of the preparation steps are difficult. Failure to carefully complete these steps will likely result in an unacceptable and unproductive replay debugging experience. I'm serious here!

Preparing the Host

You can't use just any old host for replay debugging. It must have the appropriate hardware and software.

Requirements.

1. The host must have a replay-capable CPU. Use VMware Workstation to create and replay a recording to confirm that you have a replay-capable processor.
2. Windows XP, Vista, or 7 must be installed on the host.
3. Visual Studio 2005 or 2008 must be installed on the host.
4. VMware Workstation 7.0 must be installed on the host. Please note that Workstation must be installed after Visual Studio so that the VMware Integrated Virtual Debugger Plugin for Visual Studio is properly installed.

Ensure guest DLLs are available on the host. All DLLs used by the programs you debug in the guest must be accessible on the host (please see "Why All the Preparing?" below for an explanation of why this is so). The DLLs that are built as part of your Visual Studio project will automatically be available on both the host and guest, but all other DLLs (including system DLLs) must be copied from the guest to the host. In particular, you will at least want to copy c:\windows\system32 on the guest to a folder on the host (e.g., c:\guestdlls\system32).

Preparing the Guest

You also need to get the guest in order.

Requirements.

1. The virtual machine must have exactly one virtual CPU. We inherit this limitation from the record/replay technology on which replay debugging is built.
2. 32-bit Windows XP, Server 2003, Vista, or 7 must be installed on the guest.
3. Guest tools corresponding to the version of VMware Workstation on the host must be installed in the guest.

Disable paging of kernel-mode stacks. Paging of kernel-mode stacks is disabled by changing a registry setting. This is most simply performed with the Global Flags utility distributed with the Debugging Tools for Windows package (Microsoft Debugging Tools for Windows Page (http://www.microsoft.com/whdc/devtools/debugging/default.mspx)). You need not install this package in the guest; instead, install it in the host and copy the Global Flags utility (you'll find it at C:\Program Files\Debugging Tools for Windows\gflags.exe) to the guest. Run the Global Flags utility in the guest, select Disable paging of kernel stacks under the System Registry tab, and click "OK". You must reboot the guest for this change to take effect.

Copy necessary DLLs from the host to the guest. You must ensure that any DLLs used by the program you intend to debug are available in the guest (that's where your program will be running after all!). System DLLs are already in the guest and DLLs built by your Visual Studio project will automatically be available in the guest, but any other DLLs must be manually copied into the guest; drag and drop from the host to the guest works well for this. For simple programs you won't need to do this.

Take a snapshot. This will serve as the starting point for recordings you create and will obviate the need to power on the virtual machine every time we want to create a recording. Let's call this snapshot "BaseSnapShot".

Configuring Visual Studio

Now it's time to get Visual Studio ready for replay debugging. I'm assuming you've already created a project, built it, and run it locally.

Use Microsoft symbol server. Visual Studio interacts with the replay debugging infrastructure much better when it has access to Microsoft symbols. In Visual Studio go to Tools > Options... An options window will appear. In the left pane, select Debugging > Symbols. Add a new symbol file location by clicking on the "folder" icon. Enter http://msdl.microsoft.com/download/symbols (type it carefully!). For Cache symbols from the symbol server to this directory: enter some directory, like C:\mysyms.

Don't use debugging DLLs. By default Visual Studio generates programs that use debugging DLLs that are not available on most machines. In Visual Studio go to Project > Properties... The Property Pages window appears. In the left pane, select Configuration Properties > C/C++ > Code Generation. In the right pane, change Runtime Library to Multi-threaded Debug (/MTd). Make sure you rebuild your project and run it locally to confirm that all is well.

Configuring the Integrated Virtual Debugger (IVD)

We're almost there. Now we need to configure the VMware Visual Studio plugin (the Integrated Virtual Debugger or IVD). Note that we'll only consider the essential configuration options here. A more complete treatment can be found in the Manual. Here's the recipe...

1. Open the IVD options (VMware > Options...).
2. In the left pane, click on Replay Debugging in VM > General.
3. Ensure that Local or Remote (in the right pane) is set to Local (we'll talk about remote debugging later).
4. In the right pane, enter the path to your virtual machine's .vmx file in the Virtual Machine field.
5. In the left pane, click on Replay debugging in VM > Pre-Record Event.
6. In the right pane, enter the name of your base snapshot ("BaseSnapShot") in the Base Snapshot for Recording field.
7. Click "OK".

If you were paying attention you probably saw configuration options for recording name, guest login credentials, etc. These are important, but you'll be prompted for these as you go, and they'll automatically be filled in for us.

Creating a Recording

Let's create a recording. Sure, you could always manually power on your virtual machine, copy your program from the host to the guest, start a recording, run your program, wait for it to finish, stop the recording, and update the Integrated Virtual Debugger options to refer to this new recording. Sure, you could do that! But the IVD automates the process for us.

Creating a Recording via Visual Studio. Ensure that your project is built and go to VMware > Create Recording for Replay. That's it. You'll be prompted for the guest login credentials and then you just wait for your program to run as the recording is created.

Trouble shooting. If your program never appears to start in the guest, it may be because the program can't run in the guest (e.g., due to a missing DLL). To diagnose this you may want to copy your program from the host to the guest and try to run it.

Starting Replay Debugging

The time has arrived! Let's start debugging. Set a breakpoint in the first line of main() and go to VMware > Start Replay Debugging in VM. Replay will start. You will be prompted to enter the folder containing various DLLs (e.g., ntdll.dll). Enter the name of the host folder in which you placed guest DLLs (in this tutorial it is c:\guestdlls\system32). Don't worry, the configuration options will be updated with this folder so you won't be asked again (unless a different DLL can't be found in this folder). Eventually your breakpoint will be hit. That's it, you're debugging. You can do almost anything you could do in traditional debugging. Play around. Have some fun. Don't forget to check out VMware > Reverse Continue to simulate executing in reverse.

Why All the Preparing?

Above we asked you (among other things) to ensure that Visual Studio is configured to use the Microsoft symbol server, DLLs used by your program are available in both the guest and the host, and paging of kernel-mode stacks is disabled. Why? It mostly comes down to one thing... accessing memory in the guest. From the virtualization layer we only have access to guest physical memory. When you or Visual Studio wants access to memory that is not present in guest physical memory (e.g., because it hase been paged), we can't get that memory directly.

What do about this problem? Well, in some cases, we simply try to avoid the problem. If paging of kernel-mode stacks is disabled, we are guaranteed it will be in guest physical memory. Similarly, we ask you to configure Visual Studio to use the Microsoft symbol server because when Visual Studio has access to symbols it is much more accurate in the memory it chooses to read (i.e., it reads memory that is more likely to be in guest physical memory).

Otherwise, we try to get the desired data in some other way. One technique is to read code memory out of the file from which it was mapped. This is why we need access to the same DLLs on the host and the guest. Finally, as a last resort, we take a snapshot, go live (i.e., diverge from the replay process), ask the guest tools to read the memory, and restore the snapshot. This technique is effective but slow, so it is critical that all the preparatory steps are taken to avoid these costs!

Note that the content of this post is derived from a document entitled "Guerrilla Replay Debugging Manual for Workstation" (published on the VMware Communities web site by ecl100).