Threads and Thread-Local Storage

Multitasking already draws the basic line between a process and a thread in a short section; this article stays on that same distinction and goes further into what a thread actually is underneath, what exactly is shared between threads of the same process versus what each one keeps private, and how thread-local storage gives each thread its own copy of what looks, from source code, like an ordinary global variable.

What’s shared and what isn’t

Every thread belonging to the same process shares that process’s virtual address space in full: the same code, the same heap, the same memory-mapped files, and the same open file descriptor table, so a file one thread opens is immediately visible to every sibling thread without any explicit sharing mechanism required. What each thread keeps private is exactly what a context switch actually has to save and restore on its behalf: its own stack, its own saved general-purpose registers, and its own thread-local storage area, described below. A thread’s stack is itself just an ordinary region within the shared address space, not a separate address space of its own, meaning nothing at the hardware level stops one thread from reading or corrupting another’s stack through a stray pointer; the isolation between threads’ stacks is a convention the code enforces on itself, not a protection boundary the CPU or page tables are enforcing the way they would between two separate processes.

Thread-local storage

Ordinary global variables are unsuitable for anything a thread needs its own copy of, an error-reporting variable such as C’s errno being the textbook case, since every thread sharing the same address space would otherwise be reading and writing the exact same memory location and stepping on each other’s values. Thread-local storage (TLS) solves this by giving each thread its own private copy of specially-marked variables, reached through an indirection rather than a fixed address: on x86-64, the segment registers FS or GS (their base, specifically, set through the FS_BASE/GS_BASE MSRs rather than through an actual segment descriptor the way real or protected mode would use them) point at a small, per-thread memory area, and TLS variable accesses compile down to a load or store at a fixed offset from that segment base rather than from a fixed absolute address.

__thread int error_code;   // one instance per thread, not shared

void set_error(int code) {
    error_code = code;     // compiles to a FS- or GS-relative store
}

mov eax, fs:[error_code@tpoff]   ; load this thread's own copy

Every thread’s TLS area is allocated separately when that thread is created, and a context switch between two threads of different processes (or even, in some designs, sibling threads) has to reload the relevant segment base MSR as part of the switch, so that the same TLS-relative instruction resolves to a different physical location depending on which thread is currently running, without the compiled code itself needing to know or care which thread that is.

What actually changes on a thread switch

Because sibling threads already share one address space, switching between them can skip the step that dominates the cost of switching between two unrelated processes: reloading CR3, which as Context Switching already covers, flushes the TLB and produces a burst of otherwise-avoidable cache misses on the next several memory accesses. A kernel scheduling threads from the same process back to back benefits directly from this: the general-purpose register save and restore, and the segment-base MSR reload for TLS, still both have to happen, but the address-space switch, the most expensive single component of an inter-process switch, doesn’t, which is a meaningful part of why thread-heavy designs are often faster in practice than an equivalent design built from many small separate processes.

Implementation notes

A thread’s stack, unlike a process’s, is not automatically placed and sized by whatever mechanism sets up the initial process image; it has to be explicitly allocated (commonly from the heap, or a dedicated mapped region) by whatever thread-creation call spawns it, with a fixed size chosen up front, since unlike a process’s original stack there is no separate address space boundary naturally limiting how far it could otherwise grow. Choosing that size too small produces a stack overflow that, without a guard page mapped immediately past the stack’s end to catch it, quietly corrupts whatever memory happens to sit adjacent (frequently another thread’s stack or the heap) rather than faulting cleanly at the moment the overflow actually happens.

References

^ Intel, Intel 64 and IA-32 Architectures Software Developer’s Manual, Volume 3A, Chapter 3.4.4: describes FS/GS segment base usage for thread-local data on x86-64.
^ D. Butenhof, Programming with POSIX Threads, Chapter 5: covers thread-local storage and per-thread stack allocation from the POSIX threading API’s side.

Threads and Thread-Local Storage

What’s shared and what isn’t

Thread-local storage

What actually changes on a thread switch

Implementation notes

References

See also