Skip to content
Kernelpedia

Security Mitigations (ASLR, Stack Canaries, NX/SMEP/SMAP)

Paging, the GDT, and system calls are all mechanisms a kernel needs simply to function; the mitigations this article covers are different, restrictions a kernel imposes on itself that the hardware doesn’t strictly require for correct operation, adopted specifically to make a memory-safety bug harder to turn into a working exploit rather than to make anything work at all.

ASLR

Section titled “ASLR”

Address Space Layout Randomization loads a binary’s code, its libraries, its stack, and its heap at addresses chosen freshly, at random, on every run, rather than at the same fixed addresses every time. This directly targets ELF Loading’s own PT_LOAD segment mapping: an exploit that depends on jumping to a specific, known address (a particular library function, or a piece of injected shellcode sitting at a predictable stack offset) has to first either guess that address correctly or leak it through some other bug, rather than simply hardcoding a value that works identically across every vulnerable machine running the same binary. ASLR’s actual protection is only as strong as the entropy backing it: a randomization scheme seeded from a weak or predictable source can be brute-forced or guessed in practice even though the mechanism itself is sound, which is exactly why hardware random number generation matters directly here rather than being a tangential concern.

Stack canaries

Section titled “Stack canaries”

A stack canary is a known sentinel value the compiler places on the stack, between a function’s local buffers and its saved return address, whenever -fstack-protector (or one of its stricter variants, -fstack-protector-strong/-all) is enabled. The function’s epilogue checks that value against the original before returning, and a classic stack buffer overflow, one long enough to overwrite the saved return address and redirect execution, has to overwrite the canary first to reach it, corrupting a value the epilogue is specifically watching for.

void vulnerable(char *input) {
    char buf[64];
    strcpy(buf, input);   // an overflow here overwrites the canary before the return address
    return;                // epilogue checks the canary; a mismatch aborts rather than returning
}

A mismatched canary at return time means the program aborts immediately rather than continuing to execute with a corrupted return address, turning what would otherwise be a silent, exploitable control-flow hijack into a loud, diagnosable crash instead. The canary’s own value is deliberately unpredictable (commonly containing a null byte specifically to also break naive string-based overflows that rely on functions like strcpy stopping at one) and regenerated per process from the kernel’s own random source, so an attacker can’t simply hardcode a known constant the way an unprotected exploit against a fixed stack layout otherwise could.

NX, SMEP, and SMAP

Section titled “NX, SMEP, and SMAP”

NX (No-eXecute) is a single bit in a page table entry that, when set, makes the CPU refuse to execute any instruction fetched from that page regardless of what data happens to be sitting there, closing off an entire historical class of exploit that worked by injecting executable shellcode into a writable buffer (the stack, or the heap) and jumping directly into it. Marking every writable page non-executable and every executable page non-writable (a discipline commonly called W^X, “write xor execute”) is what actually realizes NX’s protection in practice: the bit itself does nothing unless a kernel’s own memory-mapping policy consistently avoids ever creating a page that is simultaneously writable and executable, since NX only blocks execution where the kernel has actually set the bit, not by any default the hardware imposes on its own.

SMEP (Supervisor Mode Execution Prevention) and SMAP (Supervisor Mode Access Prevention) extend the same idea across the ring 3/ring 0 boundary rather than within a single address space: SMEP makes the CPU fault if ring 0 code ever attempts to execute an instruction from a page marked user-accessible, and SMAP makes it fault on an ordinary ring 0 read or write to such a page as well (a kernel that genuinely needs to touch user memory, copying a syscall argument buffer, for instance, has to use a dedicated instruction sequence that temporarily lifts the restriction, rather than accessing it through a plain pointer dereference). Both close off a related exploit pattern where corrupted kernel control flow is redirected into attacker-controlled code or data that was never placed in kernel memory at all, only in memory the exploit already controls entirely from ring 3, which NX alone, scoped to a single address space’s own page permissions, doesn’t prevent.

Implementation notes

Section titled “Implementation notes”

None of these mitigations is complete in isolation, and none is meant to be: ASLR without NX still permits code injection into predictable-once-leaked addresses, NX without ASLR still permits return-oriented programming built entirely from existing, already-executable code the attacker never needed to inject in the first place, and stack canaries protect only the specific overflow pattern of overwriting a saved return address through a contiguous buffer overrun, not other memory-safety bugs such as a use-after-free or an out-of-bounds heap write. A kernel enabling all of them together is deliberately raising the cost and complexity an exploit needs to succeed at each individual stage, not eliminating memory-safety bugs at their source, which remains a correctness problem no mitigation here actually fixes.

References

Section titled “References”
  1. ^ PaX Team, PaX address space layout randomization (ASLR): the original design document for ASLR as later adopted broadly across mainstream kernels.
  2. ^ Intel, Intel 64 and IA-32 Architectures Software Developer’s Manual, Volume 3A, Chapter 4.6: defines the NX bit, and CR4’s SMEP/SMAP enable bits.

See also

Section titled “See also”