Sheetrock's Journal: Sheetback: exciting, tweaking, lengthy

I've been about to write a journal entry maybe three or four times since my last one, consequently there might be enough here to be of some substance.

First, some exciting news

For a limited time my two most popular signatures are being offered as a combination. Can't decide? Why should you have to! Now you can correct movie trivia and grammar in the same offtopic post. Valid while supplies last.

Firefox HTML/CSS tweaking

I don't know when the Slashdot contest for developing alternative stylesheets is going to get off the ground, but if you're looking to get a head start on the action I've discovered a free tool for Firefox users that helps greatly with the debugging process: Firefox Web Developer Extension. I've had the misfortune of working with CSS/HTML lately and while I tend to avoid WYSIWYG HTML editors and the like this utility is now indispensable.

Aardvark also deserves a mention. This tool is quite nice for "cleaning up" a web page for printing -- for example, you can move the mouse over a CSS block, press "E", and it will remove the block from the page. Also for Firefox.

Lengthy Windows rambling

I'm no master of the Windows Debugger (WinDbg, freely available from Microsoft), but it's nice to have around for diagnosing system crashes. If you don't know about it and you're troubleshooting any 2000/XP machines you should get acquainted, and I'll explain why.

For many people, the diagnosis stops at the STOP screen. You get a cryptic and mostly useless message about IRQL_LESS_THAN_EQUAL or some similar bullshit with a list of hexadecimal numbers. Savvy individuals write down the message, the numbers, and any other information (ntfs.sys?), walk over to a functioning Internet-connected system, and punch something like "STOP 0x0000000a" into Google. Which gets you a Microsoft support article explaining that 0x0000000a is a code for IRQL_LESS_THAN_EQUAL. Fantastic.

If the crashing system is configured to give a meaningful crash dump, you can go farther with WinDbg. I typically configure my systems to do a Kernel Memory Dump, which writes out whatever memory Windows thinks is in use, but the Small Memory Dump (which only writes 64K each crash) will write a new file with each crash whereas the Kernel Memory Dump will overwrite its storage file each time. The setting is somewhere under Control Panel -> System -> Advanced on Windows XP.

Basically, you need a debugger and a copy of the symbol files for your Windows installation (also freely available from Microsoft), although if you are using a faster-than-56kbps connection you can also tell the debugger to request symbols as needed via an Internet connection by following the details in the help file provided with the debugger.

Anyway, to set up a system I'll unpack the symbol files to C:\WINDOWS\SYMBOLS. Then I go into WinDbg and configure the symbol file path to the same location, then Save Workspace (both options under the File menu, IIRC) so I don't have to keep setting this option. Then File -> Open Crash Dump. The crash dump will be in Minidump under the Windows directory (for example, C:\WINDOWS\MINIDUMP) for 64K dumps or in the file C:\WINDOWS\MEMORY.DMP for a Kernel Memory Dump.

This gets a window, Command, which is a subwindow of the debugger (and can be dragged-and-dropped into its frame, which I do.) If I punch in "!analyze" at the prompt and hit Enter I get this:

Use !analyze -v to get detailed debugging information.

BugCheck E2, {0, 0, 0, 0}

Probably caused by : i8042prt.sys ( i8042prt!I8xProcessCrashDump+237 )

Followup: MachineOwner

Then, the command "!analyze -v" gets me this:

MANUALLY_INITIATED_CRASH (e2)
The user manually initiated this crash dump.
Arguments:
Arg1: 00000000
Arg2: 00000000
Arg3: 00000000
Arg4: 00000000

Debugging Details:
------------------

BUGCHECK_STR: MANUALLY_INITIATED_CRASH

DEFAULT_BUCKET_ID: DRIVER_FAULT

LAST_CONTROL_TRANSFER: from f77817fa to 805339ae

STACK_TEXT:
805507dc f77817fa 000000e2 00000000 00000000 nt!KeBugCheckEx+0x1b
805507f8 f7781032 00887598 01da58c6 00000000
i8042prt!I8xProcessCrashDump+0x237
80550840 804dad9f 83595948 838874e0 00010008
i8042prt!I8042KeyboardInterruptServ
80550840 804dc0d9 83595948 838874e0 00010008 nt!KiInterruptDispatch+0x3d
805508d4 00000000 0000000e 00000000 00000000 nt!KiIdleLoop+0x12

FOLLOWUP_IP:
i8042prt!I8xProcessCrashDump+237
f77817fa 5d pop ebp

SYMBOL_STACK_INDEX: 1

FOLLOWUP_NAME: MachineOwner

SYMBOL_NAME: i8042prt!I8xProcessCrashDump+237

MODULE_NAME: i8042prt

IMAGE_NAME: i8042prt.sys

DEBUG_FLR_IMAGE_TIMESTAMP: 41107ecc

STACK_COMMAND: kb

FAILURE_BUCKET_ID: MANUALLY_INITIATED_CRASH_i8042prt!I8xProcessCrashDump+237

BUCKET_ID: MANUALLY_INITIATED_CRASH_i8042prt!I8xProcessCrashDump+237

Followup: MachineOwner

If desired, one can then dump a list of drivers that were loaded at the time. "lm t n" gets me a huge list of the following form:

start end module name
804d7000 806eb780 nt ntoskrnl.exe Wed Aug 04 01:19:48 2004 (41108004)
806ec000 8070c380 hal hal.dll Wed Aug 04 00:59:05 2004 (41107B29)
b91a4000 b91cdf00 kmixer kmixer.sys Wed Aug 04 01:07:46 2004 (41107D32)
ba06f000 ba0c1180 srv srv.sys Wed Aug 04 01:14:44 2004 (41107ED4)
[...about a hundred entries]
f777f000 f778be00 i8042prt i8042prt.sys Wed Aug 04 01:14:36 2004 (41107ECC)
[...fifty or so more]

If the debugger has difficulty determining where the crash occurred, it's worth examining the STACK_TEXT portion of "!analyze -v" or opening the call stack window (View -> Call Stack or Alt-6). As it's a stack, the most recent call (or the one occuring most closely to the generation of this dump) is on top.

Here, the call stack ends with "nt!KeBugCheckEx+0x1b". "nt" refers to the module, which you'll happily note appears in the driver list above. The "KeBugCheckEx" is a symbol referring to a particular spot in the module, and gives you something you can use Google to look up in addition to giving you some idea about what this chunk of code is about from the name. "+0x1b" means it's jumping 0x1b bytes after "KeBugCheckEx". Technically speaking, you could issue the command "u nt!KeBugCheckEx+0x1b" to show a disassembly, or "u nt!KeBugCheckEx" if you want to see what happens in this module leading up to the call, but it's unlikely to help you out.

In this case, the symbol names tell the story:

nt!KeBugCheckEx+0x1b
i8042prt!I8xProcessCrashDump+0x237
i8042prt!I8042KeyboardInterruptService+0x21c
nt!KiInterruptDispatch+0x3d
nt!KiIdleLoop+0x12

There's a registry key you can set to permit you to crash the system by holding down the right Ctrl key and pressing Scroll Lock twice, and that's how I generated this dump. But this is the same routine I use when I suspect a driver is causing a problem on the system (a common cause of crashes.) There's another command "!process" that helps working out software-created lockups, but this scenario's complex enough as is.

Anyway, I actually bring it up because after the unfortunate Sony DRM flap I went to figure out what sort of interesting API hooks might have been made into my system. If you're still with me, I thought I'd use what I'd learned from Mark's Sysinternals Blog to do it by hand. With a Kernel Memory Dump, I could check the results of the memory dump at kiservicetable (it's a window you can open in WinDbg) against my list of drivers from the command "lm t n" -- the deviants tend to show up because on my system the kernel API calls are in the 8xxxxxxx region and the driver API hooks are in the 4xxxxxxx region. Compare the 4xxxxxxx hooks against the driver list, and voila. Well, you can see what's intercepting system API calls, but to decipher which API calls are being intercepted is a pain (I just checked them in sequence against a list of API calls I found somewhere on the Internet.)

Interestingly, I do have something on my system that does this: my firewall. Although it's worth noting that because of the way API hooking works, there could be more than one program doing it -- program A inserts its hook that calls the system API after doing what it wants to do, then program B inserts a hook that calls program A after doing what it wants to do, etc. You only get to see the final hook, although I'd imagine disassembling the code at the address shown for the hook would allow you to dig further. But only do so if the code doing the hook is actually malware and not a program you paid for that has an EULA forbidding you from examining your computer.

Anyway, I won't bother checking my system this way again, as in the process of looking up the API functions I found a tool that automatically did everything I did. The dire warnings on the website prevent me from making this a part of my diagnostic kit, but I'm keeping an eye on them to see if they get to a stable release.

Christmas

Merry Christmas. I know it's late, but whatever. I dig Civilization IV but it eats memory like it's free. What's doing that, a Python instance for every frigging unit, town, and scrap of land?