Kernel Panics and Stack Backtraces

A kernel panic is what a kernel does when it decides, deliberately, that there is no safe way to continue: an assertion that should never fail did, a data structure the kernel relies on is provably corrupted, or an exception arrived that has no sane recovery (a page fault inside the page fault handler itself, for instance). Reserved exception vectors explains what triggers many of the underlying faults; this article covers what happens once the kernel has decided one of them, or an internal check, warrants stopping entirely rather than attempting to press on.

What a panic routine reports

The general-purpose registers at the moment of failure, captured either from the interrupt frame if the panic originates from an exception handler or from the calling context otherwise, are the first thing worth preserving, since whatever corrupted a pointer or miscalculated an index is often still visible in register state moments after the fact even when it’s long gone from memory by the time anyone looks. A short, specific message identifying what went wrong (which assertion failed, which exception fired and at what address) matters more than it might seem: a panic routine that only prints “kernel panic” and nothing else turns every future crash into a fresh investigation from scratch, where one that names the failing condition often points straight at the responsible code. The third element, a backtrace, is what the rest of this article covers in detail, since reconstructing it correctly is the least obvious part of the three.

Reconstructing a backtrace from frame pointers

A backtrace lists the chain of function calls active at the moment of failure, and the cheapest way to reconstruct one needs no debug information embedded in the binary at all: it walks a linked list of saved frame pointers already sitting on the stack, left there by the standard function-entry prologue every non-leaf function executes.

; standard x86-64 function prologue
push rbp
mov  rbp, rsp

Because this prologue pushes the caller’s RBP before establishing its own, RBP at any point during a non-leaf function’s execution points at a location on the stack holding the previous frame’s RBP, forming a singly-linked list threaded backward through every currently active call, one link per stack frame, all the way to the very first frame.

void print_backtrace(uint64_t rbp) {
    while (rbp) {
        uint64_t return_addr = *(uint64_t *)(rbp + 8);  // return address sits just above saved RBP
        printf("  at %#lx\n", return_addr);
        rbp = *(uint64_t *)rbp;                          // follow the chain to the caller's frame
    }
}

Each frame also has the calling function’s return address sitting immediately above the saved RBP on the stack (pushed there by the call instruction itself, before the callee’s own prologue runs), so walking the chain and reading that fixed offset at each link produces a full list of addresses, one per active call, that a kernel can either print raw or resolve against its own symbol table if one is embedded in the binary. This entire technique depends on the compiler actually maintaining the RBP chain: -fomit-frame-pointer, a default at higher optimization levels on some toolchains, repurposes RBP as an ordinary general-purpose register instead, silently breaking the walk, which is why kernel code intending to support this kind of backtrace generally builds with -fno-omit-frame-pointer explicitly rather than trusting the compiler’s own default.

Halting every CPU, not just the one that panicked

On a single-core system, halting execution after reporting is straightforward: disable interrupts and loop forever, or execute hlt in a loop, since nothing else on the machine could still be running regardless. On a multiprocessor system this is not enough: every other CPU keeps executing kernel code, potentially the exact code path that produced the corruption the panic is reporting on, right through and past the moment one CPU decides to stop, which both makes the reported state stale within moments and risks a second, unrelated-looking panic on another core triggered by the same underlying corruption. A panic routine on SMP therefore has to send an inter-processor interrupt to every other core specifically to force it to halt too, before the system as a whole is in a state actually safe to inspect (over serial or under a debugger): only once every core has stopped touching memory does the snapshot a panic reports actually describe a system that has stopped changing underneath the person reading it.

Implementation notes

A panic routine is, almost by definition, running in the least trustworthy state a kernel is ever in, which puts real limits on what it can safely do: allocating memory, taking a lock another CPU might already be holding (the very corruption being reported might be exactly what’s preventing that lock from ever being released), or calling back into normal kernel code paths all risk turning a diagnosable panic into a second, less informative failure before the first one’s report is even fully written out. The safest panic implementations write directly to a framebuffer or serial port using the smallest, most self-contained code path available, deliberately avoiding the kernel’s own general-purpose formatting or memory-management routines, precisely because those routines are exactly what might already be broken.

References

^ Linux kernel documentation, Kernel Oops: describes the analogous non-fatal report format the panic mechanism this article covers is closely related to.

Kernel Panics and Stack Backtraces

What a panic routine reports

Reconstructing a backtrace from frame pointers

Halting every CPU, not just the one that panicked

Implementation notes

References

See also