System Calls

A system call is the controlled mechanism by which code running in ring 3 requests a service (reading a file, allocating memory, sending data over a network) that only ring 0 is permitted to perform directly. Unlike an ordinary function call, a system call has to cross a privilege boundary the CPU actively enforces, which means it cannot be a plain call instruction; it requires one of a small number of instructions specifically designed to raise privilege in a controlled, kernel-defined way.

The software interrupt approach

The original mechanism, still supported for backward compatibility, uses an ordinary software interrupt: user-mode code executes int 0x80 (the specific vector is a convention, not an architectural requirement; Linux settled on 0x80, other systems have used others), which the CPU treats identically to a hardware interrupt arriving on that vector, consulting the IDT and transitioning to whatever handler is registered there, at ring 0, exactly as it would for any other interrupt. The system call number and its arguments are conventionally passed in general-purpose registers before executing int 0x80, since there is no dedicated argument-passing mechanism built into the interrupt itself.

This approach works correctly but is comparatively slow: a software interrupt goes through the full interrupt-dispatch machinery (consulting the IDT, checking the gate’s privilege requirements, saving a complete interrupt frame), machinery designed to handle arbitrary, unpredictable hardware events rather than optimized for the specific, well-known case of “user code voluntarily requesting kernel entry.”

The fast path

x86-64 defines a dedicated pair of instructions, syscall and sysret, specifically for this common case, bypassing the IDT entirely. Rather than looking up a handler through a table, syscall jumps directly to an address the kernel configured in advance in a model-specific register (LSTAR), and simultaneously loads a new code and stack segment selector from another MSR (STAR) and masks a configurable set of flags (via FMASK), accomplishing in one instruction what the interrupt path spreads across several stages of table lookups and checks. sysret reverses the transition, returning to ring 3 at the address left in a register by convention.

// Configuring the fast syscall path (executed once, during kernel init)
void init_syscalls(void) {
    wrmsr(MSR_STAR, ((uint64_t)USER_CS_BASE << 48) | ((uint64_t)KERNEL_CS << 32));
    wrmsr(MSR_LSTAR, (uint64_t)syscall_entry); // handler address
    wrmsr(MSR_FMASK, 0x200); // clear IF on entry, among other flags
}

The 32-bit predecessor pair, sysenter and sysexit, follows the identical philosophy (a fixed, MSR-configured entry point instead of an IDT lookup) for 32-bit code, and long mode’s syscall/sysret is best understood as the same idea carried forward and adapted for 64-bit operation, rather than an unrelated mechanism.

Calling convention

Because neither syscall nor int 0x80 behaves like an ordinary function call, the argument-passing convention for a system call is defined by the kernel, not by the CPU or by the platform’s regular C calling convention, though kernels commonly choose something close to the System V calling convention for familiarity, typically passing the system call number in one fixed register and arguments in a small set of others, with the return value coming back in whatever register the regular calling convention already uses for that purpose. This convention has to be agreed upon identically by both the kernel’s handler and whatever code (a userspace C library, most commonly) actually issues the syscall instruction, since neither side can infer it from the instruction itself.

Implementation notes

A syscall handler runs with interrupts typically still masked by the FMASK configuration until it explicitly re-enables them, and critically, runs on whatever stack pointer was active in user mode at the moment of the call: syscall does not automatically switch to a kernel stack the way an interrupt gate can. A kernel using the fast path therefore has to swap to a known-good kernel stack as the very first thing its handler does, before it is safe to do much of anything that might itself fault or need a reliable stack, commonly by saving the user stack pointer to a per-CPU location and loading a pre-established kernel stack pointer in its place. Mixing the two mechanisms carelessly is also a common source of confusion during early development: a userspace library built expecting int 0x80 will not work against a kernel that only implements the syscall path, and vice versa, since from the calling code’s perspective these are simply two different, mutually incompatible ways of asking to enter the kernel.

References

^ Intel, Intel 64 and IA-32 Architectures Software Developer’s Manual, Volume 2B: the reference for the SYSCALL/SYSRET instruction pair and their MSR configuration.
^ AMD, AMD64 Architecture Programmer’s Manual, Volume 2, Chapter 6: describes the STAR/LSTAR/FMASK model-specific registers referenced above.

System Calls

The software interrupt approach

The fast path

Calling convention

Implementation notes

References

See also