Extracting Windows VM crash dumps

This post aims to provides an overview of the many options for extracting Windows crash dump files and their equivalents out of Windows virtual machines. The assumption is that a Windows instance has just crashed with a Blue Screen Of Death (BSOD) and a dump is deemed helpful to diagnose the issue. Note that most of the tips below apply to physical machines as well but the examples here focus on a QEMU/KVM virtual machine.

1. Copy MEMORY.DMP after reboot

This is straightforward as long as MEMORY.DMP has actually been generated and there is a simple way to copy it out of the VM. Note that when Windows is showing the BSOD screen and “dumping memory to disk”, it is not creating a crash dump file yet. To increase the chances that the memory dump file will be persisted, Windows writes it in the page file first. Then, on first reboot, it tries to extract it into a separate file, usually %SystemRoot%\MEMORY.DMP.

A few things can go wrong here. First, if the system got super messed up by whatever caused the crash, the VM may not be able to boot again. No boot means no MEMORY.DMP and we’re out of luck. There may also not be enough free disk space to create the crash dump. This would, again, mean no MEMORY.DMP for us. Note that freeing disk space after the fact won’t help as the dump in the page file is already partially overwritten by the time the user gets to do anything about the disk space situation. Unless of course the disk can be accessed offline from the host or another VM. If you’re willing to do this, it is not necessary to reboot though (see below). Here’s an older blog post with a detailed description of the Windows behavior and useful tips.

2. Extract the page file

Little known is the fact that the page file itself can be fed into windbg and used in lieu of MEMORY.DMP (thanks to Vadim Rozenfeld for enlightening me!). The page file will be larger than MEMORY.DMP but it doesn’t require the reboot mentioned above. A Windows VM crashed and doesn’t boot back up? No problem, just extract the page file:

$ guestmount -a Windows10.qcow2 -i --ro /mnt/windows
$ cp /mnt/windows/pagefile.sys ~

And then on a Windows machine:

C:\>windbg -z path\to\pagefile.sys

3. Process QEMU guest memory dump with Volatility

This is not guaranteed to succeed but is worth the try as a last ditch effort or if the VM is in production and a “crash” dump is required without actually crashing it.

Side note: The easiest way to trigger a Windows VM crash is raising a non-maskable interrupt. This can be done by issuing the inject-nmi QMP command or its equivalent:

Welcome to the QMP low-level shell!
Connected to QEMU 2.8.50

(QEMU) inject-nmi

Back to the non-crashing scenario. First, get the guest memory dump:

Welcome to the QMP low-level shell!
Connected to QEMU 2.8.50

(QEMU) dump-guest-memory paging=false protocol=file:/tmp/vmdump

The file is a complete image of the guest physical memory in ELF format. It contains most of the data found in MEMORY.DMP but the structure is different. Use Volatility to convert it:

python vol.py -f /tmp/vmdump --profile=Win10x64 raw2dmp -O /tmp/MEMORY.DMP
Volatility Foundation Volatility Framework 2.5
Writing data (5.00 MB chunks): 
|..........................................................................
...........................................................................
...........................................................................
...........................................................................
...........................................................................
..............................|

Note that you have to know or guess (or let Volatility guess) the version of Windows running in the VM, sometimes down to the Service Pack / build level. If all works well, the resulting MEMORY.DMP can be opened in windbg as usual. You just won’t see any bug check code and BSOD parameters (duh!) and probably no context either.

4. Push dump data out of the VM programmatically

This is advanced and more of a note to self, as I hope to turn it into a tool at some point. Windows offers two kernel APIs of interest.

KeRegisterBugCheckReasonCallback with KbCallbackDumpIo
In theory, this should allow a kernel driver to be invoked by Windows as the BSOD is happening and memory is being dumped to the page file. The driver is given a pointer to the memory blocks being dumped and can push this data out of the VM using, for instance, a virtual serial port. Quite understandably, the callback code running on BSOD is very limited in what it can do. Memory allocations are obviously out of the question as is everything else that might end up allocating. What’s worse is that it’s not clear if the addresses passed to the callback are virtual or physical. The first part of the dump (header) is indicated with virtual addresses. The second part (body) starts with a few virtual pages and is then followed by physical pages indicated with physical addresses. Secondary data is again virtual. I might be missing something but apparently I’m not the only one observing this. As it stands now, this API is not usable without ugly and fragile heuristics.

KeInitializeCrashDumpHeader
This call produces the crash dump header, a 4096 byte data structure at offset 0 of MEMORY.DMP. And it works, as long as the physical memory layout that Windows works with is identical to that of whoever is producing the actual physical memory data to be appended to the header. Which sadly doesn’t seem to be the case with QEMU’s dump-guest-memory. So on one hand, having a valid and correct header saves us from the guesswork that Volatility has to do (no need to mess with Volatility “profiles”). But it’s still necessary to at least understand, or better patch the header to adapt it to the guest memory dump. Here’s a partial description of the dump header layout: computer.forensikblog.de/en/2006/03/dmp-file-structure.html.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s