Multiprocessor Bring-Up (SMP)

Every article on processes and scheduling on this wiki, Multitasking, Context Switching, Schedulers, Synchronization, already assumes more than one CPU can be running at once; none of them explain how the CPUs beyond the first, the Application Processors (APs), actually start executing code in the first place. A kernel boots on exactly one core, the Bootstrap Processor (BSP), chosen by firmware before the kernel ever runs, and every other core sits idle, waiting in a startup state, until the BSP deliberately wakes it.

The INIT-SIPI-SIPI sequence

Waking an AP means the BSP sending it a specific sequence of inter-processor interrupts through the Local APIC: an INIT IPI resets the target AP’s internal state, followed by two Startup IPIs (SIPI), sent a short delay apart, each one directing the AP to begin executing at a specific physical address encoded in the SIPI itself.

void start_ap(uint8_t apic_id, uint8_t trampoline_page) {
    send_ipi(apic_id, ICR_INIT, 0);
    delay_ms(10);

    send_ipi(apic_id, ICR_STARTUP, trampoline_page);
    delay_us(200);
    send_ipi(apic_id, ICR_STARTUP, trampoline_page);   // sent twice per spec
}

Sending the SIPI twice is a deliberate part of the original specification, a defensive measure against an AP that missed the first one (older hardware in particular was not always guaranteed to catch a single SIPI reliably); a well-behaved AP that already processed the first SIPI simply ignores the second, so sending both unconditionally is safe on every implementation rather than a workaround needed only on some.

The 16-bit trampoline

The address a SIPI encodes is a page number, not a full address, which restricts where the AP’s very first instruction can live to somewhere in the lowest 1 MB of physical memory, 4 KB-aligned: an AP starts execution in 16-bit real mode, identical to how the BSP itself began at power-on, regardless of what mode the BSP is currently running in. This means a kernel needs a small, self-contained trampoline, a block of 16-bit real-mode code assembled to run at a fixed, low physical address, that takes the AP from real mode through the same protected mode (and, for a 64-bit kernel, long mode) transition the BSP itself already performed once during boot.

[bits 16]
[org 0x8000]                ; trampoline loaded at a fixed low physical address
trampoline_start:
    cli
    lgdt [gdt_descriptor]    ; same GDT the BSP already set up
    mov eax, cr0
    or  eax, 1
    mov cr0, eax
    jmp CODE_SEG:protected_entry

The trampoline is typically assembled as a separate small binary blob and copied into place at a fixed low physical address by the BSP before any SIPI is sent, since the AP has no way to load anything from disk or fetch code from wherever the kernel’s main image happens to be mapped at this early point; every byte it executes has to already be sitting at the exact address the SIPI told it to jump to.

Reaching common kernel code

Once past the trampoline’s own mode transitions, each AP still needs its own private resources before it can safely run the same kernel code the BSP does: its own stack, since sharing one stack across multiple concurrently executing cores would corrupt it almost immediately, and its own per-CPU data structure (holding, among other things, that core’s own TSS, already covered as a hard SMP requirement, and whatever scheduler state is tracked per-core rather than globally).

    mov esp, [ap_stack_ptr]   ; each AP loads its own, pre-allocated stack
    mov eax, [ap_id]
    call kernel_ap_entry      ; jumps into common, mode-agnostic kernel code

The BSP conventionally pre-allocates one stack per AP before sending any SIPI at all, and communicates each one’s address to the corresponding AP through a fixed, agreed-upon location the trampoline knows to read (a small table indexed by APIC ID, or a value patched into the trampoline blob itself before it’s copied into place), since the AP has no allocator of its own available yet to request memory dynamically at this point. Only once an AP has its own stack and per-CPU structure fully set up does it enter the kernel’s ordinary, mode-agnostic code path, the same scheduler and synchronization primitives Synchronization already covers, indistinguishable from that point on from code running on the BSP.

Implementation notes

The BSP has to wait for confirmation an AP actually started (a flag the AP sets in shared memory once it reaches common kernel code, polled with a timeout) before either sending the next AP’s SIPI or assuming bring-up is complete, since nothing about sending a SIPI itself guarantees the target core successfully executed the trampoline. A trampoline built to be position-independent within its own fixed load address, rather than assuming it will always be copied to that exact same address on every boot, avoids having to reassemble it for a different physical layout if the chosen low-memory location is ever unavailable (already reserved by firmware, for instance) on some specific machine.

References

^ Intel, Intel 64 and IA-32 Architectures Software Developer’s Manual, Volume 3A, Chapter 9.4: the MP initialization protocol, including the INIT-SIPI-SIPI sequence described above.

Multiprocessor Bring-Up (SMP)

The INIT-SIPI-SIPI sequence

The 16-bit trampoline

Reaching common kernel code

Implementation notes

References

See also