Emulating & Debugging

Developing directly against physical hardware makes even a routine mistake expensive: a crash means a physical reboot, a corrupted disk means reimaging it, and there is no built-in way to pause execution and inspect what the CPU is doing at a given instant. An emulator removes all three costs at once, and pairing one with a source-level debugger turns kernel development from a cycle of “boot, observe a black screen, guess” into something closer to debugging any other piece of software.

QEMU

QEMU is the emulator most commonly used for kernel development, both because it boots quickly and resets instantly, and because it exposes a debugging interface purpose-built for exactly this use case. A kernel image is typically run either directly, if it is Multiboot-compliant and QEMU is invoked with -kernel, or via a bootable ISO built with a real bootloader such as GRUB:

qemu-system-x86_64 -kernel mykernel.elf -serial stdio

-serial stdio redirects the guest’s serial port output directly to the host terminal, turning a working serial driver into visible output with no physical wiring or terminal emulator configuration required: one of the reasons a serial driver is frequently among the very first pieces of working kernel code, well before any graphical output exists. -no-reboot and -no-shutdown, commonly added alongside it during early development, stop QEMU from silently resetting the virtual machine on a triple fault the way real hardware would; without them, a kernel bug severe enough to triple-fault produces nothing more than a flicker as QEMU reboots and re-runs the same broken code, rather than the frozen, inspectable state a working -s -S debug session depends on. -d int,cpu_reset goes a step further, logging every CPU exception and reset QEMU processes to the terminal (or a file, via -D logfile), which is frequently the fastest way to identify why a triple fault happened at all: a page fault while already handling a general protection fault, for instance, escalates to a double fault, and a second fault while handling that one is what triggers the triple fault and reset, a sequence -d int prints out plainly, in contrast to the reset itself, which carries no explanation of its own.

QEMU exposes a separate monitor console, distinct from the guest’s own serial output, for inspecting and controlling the emulated machine directly (info registers to dump the full register file, info mem to walk the current page tables, x/10i $pc to disassemble instructions at the program counter), reachable via Ctrl-A C when running with -serial stdio and -nographic together, or through its own dedicated socket when given a separate -monitor flag. By default, QEMU runs the guest under TCG (Tiny Code Generator), a dynamic binary translator that compiles guest instructions to host instructions on the fly entirely in software; the -enable-kvm flag instead hands guest execution off to the Linux kernel’s hardware virtualization support for near-native speed, but at some cost to debugging predictability: instruction-level single-stepping and some of QEMU’s own instrumentation behave less consistently under KVM’s hardware-virtualized execution than under TCG’s software one, which is why most kernel developers leave KVM disabled specifically during active debugging sessions and only enable it once a kernel is stable enough that raw execution speed matters more than step-by-step visibility.

Bochs

Bochs is a slower, software-only x86 emulator (it interprets each instruction directly rather than translating blocks of guest code to host code the way QEMU’s TCG does, which is the main reason for the speed difference), but one that models hardware behavior, including some undefined or edge-case behavior real silicon exhibits, with a level of fidelity QEMU’s faster, more abstracted implementation does not always match; Bochs’ unusually precise emulation of real-mode segmentation and the A20 gate in particular made it a mainstay of the hobbyist OS development community well before QEMU’s own debugging facilities matured. It is configured through a plain-text bochsrc file rather than command-line flags (naming the disk image, memory size, and boot device), and includes its own built-in low-level debugger, invoked by building or installing a debug-enabled Bochs binary and reached through its own command syntax rather than GDB’s: b to set a breakpoint, s to step, c to continue, and info gdtr/info idtr/sreg/creg to inspect descriptor tables and control registers directly, without needing a separate tool or remote connection at all. It is generally used as a complement to QEMU rather than a full replacement: QEMU for the bulk of fast, iterative development, Bochs when a bug is suspected to depend on hardware-level timing or corner-case behavior QEMU’s emulation might not reproduce faithfully: real-mode BIOS interrupt quirks and boot-sector edge cases are the areas where this distinction comes up most often in practice.

Attaching GDB

QEMU’s -s flag opens a GDB-compatible remote debugging stub on TCP port 1234 (shorthand for -gdb tcp::1234), and -S additionally pauses the CPU immediately at startup rather than letting it begin executing before a debugger has had a chance to attach:

qemu-system-x86_64 -kernel mykernel.elf -s -S

In a separate terminal, GDB connects to this stub as though it were debugging a remote process rather than one running locally:

(gdb) target remote localhost:1234
(gdb) symbol-file mykernel.elf
(gdb) break kernel_main
(gdb) continue

Loading the kernel’s own ELF file with symbol-file is what lets GDB display source-level information (function names, line numbers, local variable values) rather than raw addresses; QEMU’s stub provides the running CPU and memory state, but has no independent knowledge of what symbols correspond to what addresses without being told explicitly. A host’s stock gdb binary is frequently built only for that host’s own architecture and refuses to load a foreign-architecture symbol file at all; gdb-multiarch, a separate package on most Linux distributions, or a gdb built as part of the same cross-compiler toolchain, supports arbitrary target architectures from a single binary and is the more reliable choice for kernel work, particularly when targeting anything other than the host’s own architecture.

Breakpoints, single-stepping (stepi/nexti for instruction-level stepping, step/next for source-level stepping once debug symbols are present), and memory inspection (x/10xg $rsp, for instance, examining ten 64-bit hex values starting at the current stack pointer) all work exactly as they would debugging an ordinary userspace program, despite the “program” in this case being an entire operating system kernel running inside an emulated CPU. Ordinary break sets a software breakpoint by temporarily overwriting the target instruction with an int3 opcode, which requires the underlying page to be writable; hbreak instead uses a hardware breakpoint backed by one of the CPU’s debug registers (DR0–DR3), which QEMU emulates faithfully, and is the only option that works against code mapped read-only, such as a ROM image or a page explicitly marked non-writable by the kernel’s own page tables. watch some_variable sets a matching hardware watchpoint, halting execution the moment the named memory location changes, regardless of which instruction anywhere in the kernel happens to write it, considerably faster than manually single-stepping through code hunting for the write, and often the only practical way to track down memory corruption whose write site isn’t obvious from the corrupted value alone. GDB can also pass commands straight through to the QEMU monitor described above without leaving the debugger, via monitor <command>: monitor info registers, for instance, reaches QEMU’s own register dump from inside an ordinary GDB session.

Debugging mode transitions

A particular strength of this setup, with no easy equivalent on real hardware, is stepping directly through the real-to-protected-mode, or protected-to-long-mode, transition instruction by instruction, watching register and control-register state change on each step: code that would otherwise fail with nothing more informative than a silent reboot on real hardware instead pauses exactly where the fault occurs, with the emulator’s own diagnostic output frequently identifying precisely which step of the transition sequence went wrong. GDB’s own disassembler, however, needs help across exactly this transition: it infers the CPU’s instruction width from the loaded ELF file’s declared architecture, so it disassembles correctly once a 32- or 64-bit kernel is running but misreads every instruction executed earlier, while the CPU is still in 16-bit real mode: set architecture i8086 before single-stepping through that portion of boot corrects this, and needs to be reversed with set architecture i386 or set architecture i386:x86-64 again once the mode switch actually completes, or subsequent disassembly output becomes just as wrong in the opposite direction.

Automating the debug workflow

Typing the same handful of GDB commands after every relaunch grows tedious quickly, and most kernel projects collect them into a .gdbinit file read automatically on startup:

target remote localhost:1234
symbol-file mykernel.elf
break kernel_main

GDB refuses to auto-load a .gdbinit file outside a small set of trusted directories by default, as a security measure against a malicious file silently running arbitrary commands when GDB happens to be launched from an untrusted directory; running GDB with -x .gdbinit explicitly, or adding the project directory to ~/.gdbinit via add-auto-load-safe-path, are the two common ways around this restriction for a kernel project’s own script. A small shell script or Makefile target launching QEMU in the background with -s -S and then immediately starting gdb -x .gdbinit in the foreground reduces the entire cycle to a single command, which matters more than it might seem for a workflow repeated dozens of times over the course of debugging a single boot-sequence bug.

Implementation notes

Optimizations a compiler applies by default can make source-level debugging considerably harder to follow (variables optimized into registers rather than stack slots, code reordered relative to the source it came from), so kernel code intended to be actively debugged is commonly built with optimizations disabled (-O0) and debug information enabled (-g) at least for development builds, even though a kernel’s actual release build might reasonably use more aggressive optimization once it is working correctly. It’s also worth remembering that QEMU is a software emulation of the CPU, not identical in every timing and edge-case detail to a real processor: code that works flawlessly under QEMU is not automatically guaranteed to behave identically on real hardware, which is why testing on physical machines eventually matters even for a kernel that has been extensively validated under emulation first. Finally, a debugging session left attached across a triple fault without -no-reboot produces a particularly confusing symptom: GDB’s continue appears to succeed, execution resumes, and moments later every previously set breakpoint and watchpoint has silently stopped triggering, not because they were removed, but because QEMU quietly rebooted the entire virtual machine underneath the still-connected debugger, and the running code is now the very beginning of boot again rather than wherever the session had previously reached.

References

^ QEMU Documentation, “GDB usage”: the reference for QEMU’s -s/-S remote debugging stub used above.
^ QEMU Documentation, “QEMU Monitor”: documents the monitor console and commands referenced above, including info registers and info mem.
^ Bochs User Manual, “The Internal Debugger”: documents Bochs’ own breakpoint and register-inspection commands.

Emulating & Debugging

QEMU

Bochs

Attaching GDB

Debugging mode transitions

Automating the debug workflow

Implementation notes

References

See also