FXSAVE/XSAVE and FPU/SIMD State

Context Switching already introduces lazy FPU switching, deferring the cost of saving and restoring floating-point and SIMD register state until a task actually touches it, without naming the actual instructions involved or the difference between the older fixed-size save format and the newer extensible one. This article stays on that same mechanism and goes further into FXSAVE/FXRSTOR, XSAVE/XRSTOR, and the #NM exception that lazy switching depends on to detect first use.

FXSAVE and the legacy 512-byte area

FXSAVE writes a fixed 512-byte block covering the x87 FPU’s eight stack registers, its control and status words, and the XMM0–XMM15 registers SSE added; FXRSTOR reads that same layout back. Because the block’s size and internal layout are both fixed by the instruction itself, no metadata inside the block needs to describe what it contains: a kernel implementing lazy switching only needs one 512-byte buffer per task and can save or restore it unconditionally, without checking which specific registers a given task actually used.

struct fpu_state {
    uint8_t data[512];
} __attribute__((aligned(16)));

void save_fpu(struct fpu_state *s)  { __asm__ volatile("fxsave %0" : "=m"(*s)); }
void restore_fpu(struct fpu_state *s) { __asm__ volatile("fxrstor %0" : : "m"(*s)); }

XSAVE and variable-size state

AVX, and later extensions such as AVX-512, added register state (YMM’s upper halves, then ZMM, then the AVX-512 mask registers) that FXSAVE’s fixed 512-byte layout has no room for and cannot be extended to cover without breaking every existing consumer of that fixed format. XSAVE replaces it with a self-describing, variable-size layout: the XCR0 register (read and written with XGETBV/XSETBV, not a normal MSR since it’s specific to extended state rather than general CPU configuration) has one bit per state component, x87, SSE, the upper AVX bits, and so on, and only the components whose XCR0 bit is set are actually saved or restored on a given XSAVE/XRSTOR call. CPUID leaf 0x0D reports the actual save area size a particular combination of enabled components requires, which a kernel queries once at boot to size its per-task save buffers correctly rather than assuming a fixed size the way FXSAVE allowed.

xor ecx, ecx
xgetbv                  ; EDX:EAX = current XCR0
or eax, (1 << 2)        ; enable the AVX state component (bit 2)
xsetbv

XSAVE is itself a family of related instructions rather than one fixed behavior: XSAVEOPT additionally skips writing any component that hasn’t been modified since the last restore, and XSAVEC/XSAVES add compaction, omitting disabled or unused components from the saved layout entirely rather than reserving space for them, both refinements aimed at the same goal FXSAVE’s fixed format couldn’t reach: not paying to save state a task never touched.

The #NM exception and lazy switching

Lazy FPU switching, as Context Switching covers from the scheduling side, works by clearing CR0.TS for whichever task currently owns the live FPU/SIMD register state, and setting it for every other task: an attempt by a task without ownership to execute any floating-point, MMX, or SSE/AVX instruction with TS set raises #NM (Device Not Available, vector 7 in the IDT’s reserved range), rather than executing the instruction against whatever stale register contents happen to be sitting there. The #NM handler is where the actual save/restore this article describes happens: save the previous owner’s state with FXSAVE or XSAVE, restore the faulting task’s own previously-saved state with FXRSTOR/XRSTOR, clear TS, and return, letting the faulting instruction re-execute successfully against now-correct register contents.

void nm_handler(void) {
    clts();                                  // clear CR0.TS
    if (fpu_owner) save_fpu(&fpu_owner->fpu_state);
    restore_fpu(&current_task->fpu_state);
    fpu_owner = current_task;
}

Implementation notes

A task’s saved FPU/SIMD state has to be initialized to a defined value the first time it’s ever restored, not left as whatever garbage happened to occupy the allocated buffer: FXRSTOR/XRSTOR faithfully reproduce whatever bit pattern they’re given, including an invalid one, and a freshly created task’s buffer needs to hold a valid reset-equivalent state (commonly produced once with FXSAVE/XSAVE immediately after the FPU is first reset, then copied as a template) rather than uninitialized memory. XSAVE’s save area additionally requires 64-byte alignment, stricter than FXSAVE’s 16-byte requirement, which needs deliberate attention when a kernel simply switches from one instruction family to the other without re-checking the alignment of buffers already allocated for the older, less strict layout.

References

^ Intel, Intel 64 and IA-32 Architectures Software Developer’s Manual, Volume 1, Chapter 13: the complete XSAVE feature set, XCR0, and the legacy FXSAVE area layout.

FXSAVE/XSAVE and FPU/SIMD State

FXSAVE and the legacy 512-byte area

XSAVE and variable-size state

The #NM exception and lazy switching

Implementation notes

References

See also