The System V ABI and Calling Convention

System V AMD64 ABI

Integer args (in order)

RDI, RSI, RDX, RCX, R8, R9

Return value

RAX (RDX:RAX if 128-bit)

Stack alignment before call

16 bytes

Red zone

128 bytes below RSP

The System V AMD64 ABI is the calling convention nearly every x86-64 Unix-like system, Linux included, compiles native code against, and several articles on this wiki already lean on pieces of it without spelling the whole convention out: System Calls reuses its register assignments for syscall arguments, and Build Systems explains why -mno-red-zone matters in kernel code without defining what the red zone actually is. A calling convention is simply an agreement, not enforced by the CPU at all, about where arguments go, where the return value comes back, and which registers a callee must leave unchanged; nothing prevents code that ignores it entirely, except that it then can’t correctly call, or be called by, any code compiled to expect it.

Passing arguments

The first six integer or pointer arguments to a function are passed in fixed registers, in order: RDI, RSI, RDX, RCX, R8, R9. A seventh integer argument, and any argument beyond what fits in registers, is passed on the stack instead, pushed in reverse order so the first stack argument ends up closest to the return address. Floating-point arguments follow an entirely separate sequence of XMM0 through XMM7, independent of the integer registers, so a function taking one integer and one floating-point argument passes the first in RDI and the second in XMM0, not RDI and RSI.

long example(long a, long b, double c);
// a -> RDI, b -> RSI, c -> XMM0

Return values and register ownership

A function’s return value comes back in RAX, or, for a value too large for one register (a 128-bit integer, for instance), split across RDX:RAX the same way a 64-bit MSR value splits across EDX:EAX. Registers themselves are split into two categories the ABI fixes by convention: caller-saved registers (RAX, RCX, RDX, RSI, RDI, R8–R11) may be freely overwritten by a called function, so anything the caller still needs from them has to be saved before the call; callee-saved registers (RBX, RBP, R12–R15) must come back holding whatever value they held at entry, so a function using them for its own scratch work has to save and restore them itself. This is a second, ABI-level reason (beyond the one already covered when CPUID itself clobbers it) that RBX specifically needs explicit saving around inline assembly that touches it: the compiler assumes, correctly for any properly-written function, that RBX survives a call unless told otherwise.

Stack alignment

RSP must be 16-byte aligned at the point a call instruction executes, meaning it is aligned to 16 bytes plus 8 immediately after the call (the return address call pushes is 8 bytes, breaking what was 16-byte alignment into a consistent, ABI-defined offset from it) inside the called function’s own prologue. Code that allocates stack space in units not aligned to this boundary, or that skips restoring alignment after pushing an odd number of 8-byte values, produces a misaligned stack that most ordinary integer code tolerates silently but that faults immediately the moment an aligned SSE instruction (MOVAPS, among others) executes against it, since those specific instructions require 16-byte alignment as a hard precondition rather than merely preferring it.

The red zone

A leaf function, one that calls nothing else itself, is permitted to use up to 128 bytes below RSP as scratch space without first subtracting anything from RSP to formally reserve it, since nothing else can be relying on that memory if the function calls out to nothing that might also want it. This red zone is what -mno-red-zone, already covered from the build-flag side, disables: the assumption breaks down specifically inside an interrupt or exception handler, which can be invoked asynchronously at any point atop whatever stack the interrupted code happened to be using, including a leaf function’s own red zone still holding live data the handler would silently overwrite if it used the same 128 bytes for its own scratch space, corrupting the interrupted function’s state in a way that only manifests once execution returns to it.

; leaf function using the red zone, no stack adjustment needed
compute:
    mov [rsp - 8], rax   ; scratch write into the red zone
    mov [rsp - 16], rbx
    ; ... use the scratch values ...
    ret

Implementation notes

The ABI’s rules apply uniformly to both userspace and kernel-mode code (the CPU enforces no distinction), but a kernel’s own build has to deliberately opt out of two default assumptions the compiler otherwise makes on the ABI’s behalf: the red zone, unsafe the moment interrupts can land atop arbitrary code, and unrestricted SSE/floating-point codegen, unsafe until the kernel has committed to saving and restoring that register state across context switches, which is why kernel build flags routinely include both -mno-red-zone and -mno-sse/-mno-mmx together rather than either alone.

References

1. ^ M. Matz, J. Hubička, A. Jaeger, M. Mitchell, System V Application Binary Interface: AMD64 Architecture Processor Supplement: the full specification, including register assignment, stack alignment, and the red zone.

See also

• System Calls: reuses this article’s integer-argument registers, with a kernel-defined convention rather than the compiler’s.
• Build Systems: the -mno-red-zone and -mno-sse flags that opt a kernel build out of two ABI assumptions unsafe in kernel code.
• CPUID: a concrete example of an instruction that clobbers the callee-saved RBX register and needs it saved explicitly.