Operating Systems Through the Go Runtime

A mini course that teaches operating systems concepts through the Go runtime.

Why the Go Runtime?

The Go runtime is essentially a user-space operating system. It:

• Schedules thousands of goroutines onto a small number of OS threads
• Manages memory with its own allocator and garbage collector
• Multiplexes I/O using epoll/kqueue so blocking calls don’t block threads
• Handles signals for preemption and OS integration
• Manages stacks that grow and shrink dynamically

All of this is written in readable Go (with some assembly for the lowest-level operations), making it far more approachable than a kernel like Linux while covering the same fundamental concepts.

Course Structure

Optional Modules

Materials

Each module includes:

• Lecture notes — Detailed reading material with Go runtime code walkthroughs
• Comprehension checks — Short questions to verify understanding

Additional resources:

• Programming Assignments — Three hands-on projects
• Final Exam — Comprehensive assessment
• Glossary — Key terms defined
• Code Walkthrough — Trace a goroutine’s full lifecycle
• Experiments — Hands-on runtime exploration
• Further Reading — Books, papers, and resources

Getting Started

1. Read the syllabus for the full course overview and learning outcomes
2. Clone the Go source tree to follow along with code references
3. Start with Module 1

All code references use the format file.go:line and point to files under src/runtime/ in the Go source tree unless otherwise noted.

Operating Systems Through the Go Runtime

Course Overview

This mini course teaches core operating systems concepts using the Go runtime as a living, readable case study. Rather than studying OS theory in isolation, students will read and analyze production code that implements scheduling, concurrency, memory management, and I/O — the same code that runs every Go program.

The Go runtime is an ideal subject because it is essentially a user-space operating system: it multiplexes thousands of goroutines onto OS threads, manages its own memory allocator and garbage collector, handles signals, and implements its own I/O scheduler. All of this is written in readable Go (with some assembly), making it far more approachable than a kernel like Linux.

Prerequisites: An introductory systems programming or computer organization course (or equivalent). Students should be comfortable reading C-like code and have basic familiarity with concepts like pointers, memory addresses, and CPU registers. Go syntax is introduced as needed — prior Go experience is helpful but not required.

Source code: All code references point to the Go source tree. Students should have a copy available (the course references files under src/runtime/ and src/os/).

Learning Outcomes

By the end of this course, students will be able to:

1. Explain the role of an operating system and identify which OS responsibilities the Go runtime handles in user space.
2. Distinguish processes, OS threads, and goroutines, and explain why user-level threading is useful.
3. Describe the GMP scheduling model (goroutines, machines, processors) and trace a goroutine through its lifecycle states.
4. Explain work stealing as a distributed scheduling strategy and analyze its tradeoffs versus centralized scheduling.
5. Analyze low-level synchronization primitives (futexes, semaphores, spin locks) and explain how they build on hardware atomics and OS support.
6. Read and explain the channel implementation, including the blocking/waking mechanism and the select algorithm.
7. Trace an I/O operation from os.File.Read() through the internal poller to epoll/kqueue, and explain how the runtime makes blocking I/O non-blocking.

Optional learning outcomes:

8. Trace how a system call works from a Go function call through the syscall package down to the hardware instruction, and explain the user/kernel boundary.
9. Describe a multi-level memory allocator and explain how per-CPU caches, size classes, and spans reduce fragmentation and contention.
10. Explain how goroutine stacks grow and shrink dynamically, contrasting this with fixed-size OS thread stacks.

Module Schedule

Total: 7 hours (allows flexibility for questions and discussion)

Optional Modules

Assessments

Narrative Arc

The course tells a coherent story, building from the bottom up:

1. Module 1 establishes the foundation: what an OS does and how the Go runtime acts as a user-space operating system.
2. Modules 3–5 cover the scheduler — the heart of the runtime. Students learn why goroutines exist, how the GMP model works, and how work stealing keeps all CPUs busy.
3. Modules 6–7 explore concurrency from the programmer’s perspective: how locks and channels are implemented on top of the scheduler primitives (gopark/goready) introduced in Modules 4–5.
4. Module 10 ties everything together by showing how I/O integrates with the scheduler, completing the picture of the runtime as a full OS.

The optional modules — System Calls (Module 2), Memory Management (Module 8), and Goroutine Stacks (Module 9) — provide deeper dives into specific subsystems and can be covered as time permits.

How to Use These Materials

• Notes provide detailed reading material with code walkthroughs. Read these before or after the corresponding lecture.
• Comprehension checks are short questions to verify understanding of each module.
• Assignments are hands-on programming exercises that reinforce key concepts.
• The exam tests conceptual understanding across all modules.

All code references use the format file.go:line and point to the Go source tree under src/runtime/ unless otherwise noted. Students are encouraged to open the actual source files and read the surrounding context.

Module 1: Introduction - The Runtime as an OS

Duration: 45 minutes

Background: What Makes an Operating System?

At its core, an operating system is the layer of software that stands between application code and raw hardware, solving three fundamental problems. First, it manages resources: a machine has a finite number of CPU cores, a fixed amount of RAM, and a handful of I/O devices, yet dozens or hundreds of programs need to share them simultaneously. Second, it provides abstraction: rather than forcing every programmer to understand the registers of a particular NIC or the geometry of a disk, the OS exposes uniform interfaces – files, sockets, virtual address spaces – that hide hardware specifics. Third, it enforces protection and isolation: one process’s bug should not be able to overwrite another’s memory or starve the rest of the system of CPU time. These three pillars – multiplexing, abstraction, and isolation – appear in every OS textbook from Tanenbaum to OSTEP, and they have driven the design of operating systems for over half a century.

Linux is the most widely deployed kernel in the world, running on everything from Android phones to supercomputers. It is a monolithic kernel: the process scheduler, virtual memory subsystem, file system layer (VFS), device drivers, and networking stack all execute in a single, shared address space in kernel mode. The kernel has grown from roughly 10,000 lines of C in Linus Torvalds’s 1991 original to well over 30 million lines of code today, with contributions from thousands of developers. Linux on x86-64 exposes over 450 system calls – the narrow interface through which user-space programs request kernel services. When a process calls read(), the CPU executes a SYSCALL instruction that traps into kernel mode, where the kernel validates arguments, performs the I/O (perhaps scheduling the process to sleep while waiting for a disk), and eventually returns a result. Processes get the illusion of private address spaces through page tables managed by the memory subsystem. The Completely Fair Scheduler (CFS) distributes CPU time across runnable threads using a red-black tree keyed by virtual runtime. All of this machinery exists to solve the same three problems: sharing hardware, presenting clean interfaces, and keeping programs from interfering with each other.

What is less commonly appreciated is that language runtimes have been reinventing these same mechanisms in user space for decades. The Erlang BEAM virtual machine is perhaps the most striking example: it implements its own lightweight processes (a single BEAM instance can run millions of them), its own preemptive scheduler that interrupts a process after a fixed number of “reductions” (abstract units of work), per-process garbage-collected heaps, and its own I/O multiplexing layer. An Erlang application running on BEAM looks structurally more like a small operating system than a typical user-space program. The Java Virtual Machine (JVM) similarly manages its own heap with sophisticated garbage collectors (G1, ZGC, Shenandoah), maps application threads onto OS threads with its own scheduling hints, and provides a bytecode-level abstraction that decouples programs from the underlying hardware and OS. This pattern is not entirely new – Smalltalk images in the 1970s managed their own object memory and green threads, and the Lisp machines of the 1980s ran language runtimes directly on custom hardware, blurring the line between OS and runtime entirely. Even in the world of OS research, projects like MIT’s Exokernel (1995) argued that the kernel should do as little as possible, pushing resource management into application-level “library operating systems” – a philosophy that rhymes with what modern language runtimes actually do, albeit above a conventional kernel rather than a minimal one.

The Go runtime sits squarely in this tradition. It is a substantial piece of systems software – roughly 150,000 lines of Go and assembly – that ships embedded in every compiled Go binary. It implements its own user-space thread scheduler (goroutines and the G-M-P model), its own memory allocator and garbage collector, its own I/O multiplexer (the “netpoller,” built on top of epoll or kqueue), and its own mechanisms for preemption and synchronization. Unlike the BEAM or JVM, Go compiles to native machine code and interacts with the kernel through direct system calls rather than an interpreter or JIT, which makes its runtime a particularly transparent case study: you can trace the path from a goroutine sending on a channel all the way down to a futex system call, reading real production code at every layer. This module introduces the Go runtime through the lens of operating systems – showing how the same problems that motivate kernel design (scheduling, memory management, I/O multiplexing, synchronization) reappear in the runtime, and how studying the runtime gives you a concrete, readable codebase in which to explore them.

1. What is an Operating System? (10 min)

An operating system performs three fundamental jobs:

1. Resource management – multiplexing scarce hardware (CPUs, memory, disks, network) across many competing programs.
2. Abstraction – hiding hardware details behind clean interfaces (files, processes, virtual memory) so programs don’t need to know whether they’re talking to an NVMe drive or a spinning disk.
3. Protection – isolating programs from each other so one buggy process can’t corrupt another’s memory or monopolize the CPU.

Every OS textbook opens with these three pillars. What’s less commonly taught is that a language runtime solves the same problems, just one level up the stack.

2. The Go Runtime as a User-Space OS (10 min)

The Go runtime is a substantial piece of systems software (~150k lines of Go and assembly) that ships inside every Go binary. It implements:

The analogy is not perfect – the Go runtime runs in user space, relies on the real kernel for actual hardware access, and doesn’t enforce hard protection boundaries. But the design problems it faces are remarkably similar to those of an OS kernel, and studying the runtime gives students a way to read real, production-quality systems code that solves the same problems covered in an OS course.

Why study the Go runtime instead of (or alongside) a real kernel?

• Readable: The runtime is written mostly in Go, with targeted assembly only where necessary. Compare this to the Linux kernel’s mix of C, macros, and architecture-specific assembly.
• Self-contained: A single src/runtime/ directory (~250 files) contains the scheduler, memory allocator, garbage collector, and platform abstraction layer.
• Runnable: Students can instrument the runtime, rebuild it with GOROOT, and immediately run programs against their modified runtime.
• Production-quality: Unlike teaching kernels (xv6, PintOS), this is code that runs every Go program in production at Google, Cloudflare, and thousands of other companies.

3. Tour of the Runtime Source Tree (10 min)

The Go runtime lives at src/runtime/ within the Go source tree. Here are the key files and their roles:

Core Scheduling

Memory Management

Platform Abstraction (System Calls)

I/O and Networking

Synchronization

4. The Key Abstractions: G, M, P (10 min)

The Go scheduler is built around three core data structures, defined in src/runtime/runtime2.go. The comment at the top of src/runtime/proc.go, lines 24-34 introduces them:

// src/runtime/proc.go, lines 24-34

// Goroutine scheduler
// The scheduler's job is to distribute ready-to-run goroutines over worker threads.
//
// The main concepts are:
// G - goroutine.
// M - worker thread, or machine.
// P - processor, a resource that is required to execute Go code.
//     M must have an associated P to execute Go code, however it can be
//     blocked or in a syscall w/o an associated P.
//
// Design doc at https://golang.org/s/go11sched.

G – Goroutine

A g represents a goroutine. It holds the goroutine’s stack, saved registers (program counter, stack pointer), status, and a pointer to the m currently running it (src/runtime/runtime2.go, lines 473-509):

// src/runtime/runtime2.go, lines 473-509

type g struct {
    // Stack parameters.
    // stack describes the actual stack memory: [stack.lo, stack.hi).
    // stackguard0 is the stack pointer compared in the Go stack growth prologue.
    // It is stack.lo+StackGuard normally, but can be StackPreempt to trigger a preemption.
    stack       stack   // offset known to runtime/cgo
    stackguard0 uintptr // offset known to liblink
    stackguard1 uintptr // offset known to liblink

    _panic    *_panic // innermost panic - offset known to liblink
    _defer    *_defer // innermost defer
    m         *m      // current m; offset known to arm liblink
    sched     gobuf
    syscallsp uintptr // if status==Gsyscall, syscallsp = sched.sp to use during gc
    syscallpc uintptr // if status==Gsyscall, syscallpc = sched.pc to use during gc
    syscallbp uintptr // if status==Gsyscall, syscallbp = sched.bp to use in fpTraceback
    ...
    atomicstatus atomic.Uint32
    goid         uint64

A goroutine moves through well-defined states (also in runtime2.go, lines 35-77):

• _Gidle (0) – just allocated, not yet initialized
• _Grunnable (1) – on a run queue, ready to execute
• _Grunning (2) – currently executing on an M with a P
• _Gsyscall (3) – executing a system call, not running Go code
• _Gwaiting (4) – blocked (channel, mutex, sleep, etc.)
• _Gdead (6) – finished execution or on a free list

M – Machine (OS Thread)

An m represents an OS thread. It points to the goroutine it’s currently running (curg), its associated P, and a special g0 goroutine used for scheduling stack frames (src/runtime/runtime2.go, lines 618-653):

// src/runtime/runtime2.go, lines 618-653

type m struct {
    g0      *g     // goroutine with scheduling stack
    ...
    curg         *g       // current running goroutine
    ...
    p puintptr            // currently attached P
    nextp           puintptr
    oldp            puintptr // The P that was attached before executing a syscall.
    id              int64
    ...
    spinning        bool // m is out of work and is actively looking for work
    blocked         bool // m is blocked on a note

Key insight: m.g0 is a special goroutine whose stack is used whenever the scheduler itself needs to run. Scheduling decisions happen on g0’s stack, not on any user goroutine’s stack. This is analogous to how a kernel switches to a kernel stack to handle interrupts.

P – Processor (Logical CPU)

A p represents a “logical processor” – a resource token that an M must hold to execute Go code. The number of Ps equals GOMAXPROCS (default: number of CPU cores) (src/runtime/runtime2.go, lines 772-818):

// src/runtime/runtime2.go, lines 772-818

type p struct {
    id          int32
    status      uint32 // one of pidle/prunning/...
    link        puintptr
    schedtick   uint32     // incremented on every scheduler call
    syscalltick uint32     // incremented on every system call
    sysmontick  sysmontick // last tick observed by sysmon
    m           muintptr   // back-link to associated m (nil if idle)
    mcache      *mcache
    ...
    // Queue of runnable goroutines. Accessed without lock.
    runqhead uint32
    runqtail uint32
    runq     [256]guintptr
    // runnext, if non-nil, is a runnable G that was ready'd by
    // the current G and should be run next instead of what's in
    // runq if there's time remaining in the running G's time slice.
    runnext guintptr

The P is the key innovation in Go’s scheduler design (introduced in Go 1.1). Each P has:

• A local run queue (runq – a circular buffer of 256 goroutines) that can be accessed without locks
• An mcache for fast, lock-free memory allocation
• A runnext slot for cache-friendly goroutine handoff

Without Ps, every goroutine operation would require locking a global run queue, creating a scalability bottleneck. The P design enables the work-stealing algorithm: when an M’s P runs out of goroutines, it steals from other Ps’ run queues.

How G, M, P Relate

    ┌─────┐     ┌─────┐     ┌─────┐
    │  P  │     │  P  │     │  P  │   (GOMAXPROCS = 3)
    │runq │     │runq │     │runq │
    └──┬──┘     └──┬──┘     └─────┘
       │           │          (idle)
    ┌──┴──┐     ┌──┴──┐
    │  M  │     │  M  │     ┌─────┐
    │(thr)│     │(thr)│     │  M  │   (blocked in syscall,
    └──┬──┘     └──┬──┘     │     │    no P attached)
       │           │        └──┬──┘
    ┌──┴──┐     ┌──┴──┐     ┌──┴──┐
    │  G  │     │  G  │     │  G  │
    │(run)│     │(run)│     │(sys)│
    └─────┘     └─────┘     └─────┘

An M must acquire a P before it can run Go code. When a goroutine enters a blocking system call, the runtime can detach the P from that M and hand it to a different M, keeping the CPU utilized. This is covered in the Processes, Threads, and Goroutines module.

5. Course Roadmap Preview (5 min)

This course explores OS concepts through the lens of the Go runtime. The seven core modules tell a bottom-up story:

1. Introduction (this module) – The runtime as a user-space OS; G, M, P overview
2. Processes, Threads, and Goroutines – OS threads vs. goroutines; the G and M structs; goroutine states
3. The Go Scheduler – The scheduling loop; schedule(), findRunnable(), execute(); run queues
4. Work Stealing and Preemption – Distributed scheduling; cooperative and asynchronous preemption via SIGURG
5. Synchronization Primitives – Futexes, semaphores, spin locks; sync.Mutex internals
6. Channels and Select – Channel implementation; hchan, chansend/chanrecv; the selectgo algorithm
7. File Systems, I/O, and the Network Poller – epoll/kqueue; the integrated netpoller; non-blocking I/O

Three optional modules provide deeper dives into specific subsystems:

2. System Calls – How Go crosses the user/kernel boundary; SYSCALL instruction; entersyscall/exitsyscall
3. Memory Management – Virtual memory concepts; Go’s allocator hierarchy; size classes; garbage collection
4. Goroutine Stacks – Growable stacks; stack growth and copying; contrast with fixed OS thread stacks

Suggested Exercises

1. Explore the source: Clone the Go repository. Open src/runtime/proc.go and read the top-level comment (lines 24-80). Identify where schedule() is defined.
2. Count goroutines: Write a Go program that spawns 100,000 goroutines, each sleeping for 1 second. Use runtime.NumGoroutine() to observe the count. How much memory does each goroutine consume?
3. GOMAXPROCS experiment: Run a CPU-bound benchmark with GOMAXPROCS=1, 2, 4, and your machine’s core count. Observe the speedup and diminishing returns.
4. Build the runtime: Modify src/runtime/proc.go to add a println("schedule called") at the top of schedule(). Rebuild the toolchain with ./make.bash and run a simple program. How often is schedule() called?

Key Definitions

• Goroutine (G): A lightweight, user-space thread of execution managed by the Go runtime. Initial stack size is 2 KB (vs. ~1 MB for OS threads).
• Machine (M): An OS thread. The runtime creates Ms as needed and parks them when idle.
• Processor (P): A logical scheduling context. An M needs a P to run Go code. The number of Ps is set by GOMAXPROCS.
• Work stealing: When a P’s run queue is empty, its M steals goroutines from other Ps’ run queues, balancing load without central coordination.
• sysmon: A special background thread (M without a P) that monitors for stuck goroutines, retakes Ps from long syscalls, and triggers GC.

Further Reading

• The Go scheduler design document (Dmitry Vyukov, 2012)
• Go source: src/runtime/proc.go – The scheduler
• Go source: src/runtime/runtime2.go – Core data structures
• GopherCon 2018: Kavya Joshi - The Scheduler Saga

Module 1: Introduction (The Runtime as an Operating System)

Comprehension Check

Question 1 (Short Answer)

List at least four responsibilities that a traditional operating system kernel handles, and for each one, identify the corresponding component or subsystem in the Go runtime.

Question 2 (True/False with Explanation)

True or False: The Go runtime runs entirely in user space and never makes system calls to the kernel.

Question 3 (Code Reading)

Consider this comment from the top of runtime/proc.go:

// Goroutine scheduler
// The scheduler's job is to distribute ready-to-run goroutines over worker threads.
//
// The main concepts are:
// G - goroutine.
// M - worker thread, or machine.
// P - processor, a resource that is required to execute Go code.
//     M must have an associated P to execute Go code, however it can be
//     blocked or in a syscall w/o an associated P.

Why does the design separate M (machine/thread) from P (processor)? What problem would arise if every M always had exclusive access to a P?

Question 4 (What Would Happen If…)

What would happen if the Go runtime used a single global run queue instead of per-P local run queues? Consider both correctness and performance.

Question 5 (Short Answer)

Explain what GOMAXPROCS controls and why its default value equals the number of available CPU cores. What does it not control?

Question 6 (Short Answer)

In what sense is the Go runtime’s relationship to goroutines analogous to a hypervisor’s relationship to virtual machines? In what ways does the analogy break down?

Module 3: Processes, Threads, and Goroutines

Duration: 60 minutes

Background: From Processes to Lightweight Threads

The history of concurrent execution in operating systems is a story of progressively reducing overhead. Early computers in the 1950s and 1960s ran one program at a time in batch processing mode: a job was loaded, it ran to completion, and the next job was loaded. The insight that CPUs spent most of their time waiting for I/O led to multiprogramming in systems like IBM OS/360 and Multics, where multiple programs could be resident in memory simultaneously, and the OS would switch between them when one blocked on I/O. Each program ran as a process with its own protected address space, file descriptors, and kernel metadata. This isolation was essential for safety, but it came at a cost: creating a process meant duplicating page tables, allocating kernel data structures, and flushing the TLB. By the 1980s, researchers recognized that many concurrent activities within the same application did not need full isolation. This led to the development of threads—multiple execution contexts sharing a single address space. The POSIX threads (pthreads) standard, formalized in 1995 as IEEE 1003.1c, gave Unix systems a portable API for kernel-managed threads, each with its own stack (typically 1–8 MB), register set, and scheduling state.

Linux took a distinctive approach to this problem. Rather than implementing threads as a fundamentally different abstraction from processes, Linux unified them under a single kernel data structure: task_struct. The clone() system call, introduced in Linux 2.0 (1996), accepts a bitmask of flags that specify exactly which resources the new task should share with its parent—address space (CLONE_VM), file descriptors (CLONE_FILES), signal handlers (CLONE_SIGHAND), and more. A “process” is simply a clone() call that shares nothing; a “thread” is a clone() call that shares (almost) everything. The kernel scheduler treats both identically. This design is elegant and flexible, and it is why the Go runtime can call clone() directly on Linux to create OS threads, bypassing the pthreads library entirely and retaining full control over stack placement and signal masks.

Even with threads being cheaper than processes, OS threads remain expensive at scale. Each thread requires a kernel scheduling entity, a fixed-size stack (8 MB of virtual address space on Linux by default, 512 KB on macOS), and context switches that cost 1–10 microseconds due to kernel entry/exit, register save/restore, and cache/TLB disruption. Dan Kegel’s influential “C10K problem” essay (1999) crystallized the challenge: how do you handle 10,000 simultaneous network connections on a single server? A thread-per-connection model buckles under the weight of memory consumption and scheduling overhead at that scale. The problem has only intensified in the era of microservices and cloud computing, where C10M (ten million connections) is the new frontier.

This tension between the convenience of threads (a sequential programming model for each task) and their cost has driven a decades-long search for lightweight alternatives. Solaris 2 introduced green threads in the early 1990s, multiplexing user-level threads onto a smaller number of kernel threads. The GNU Portable Threads library (Pth) offered cooperative user-level threading on any Unix system. Erlang, designed for telecom switches, built its entire runtime around ultra-lightweight processes (not OS processes) that could number in the millions. More recently, Java’s Project Loom (finalized in Java 21, 2023) introduced virtual threads that are scheduled by the JVM rather than the OS. Rust’s async/await model compiles asynchronous tasks into state machines that are driven by a user-space executor. Each of these approaches embodies the same core idea: decouple the unit of concurrent work from the OS thread, so that the programmer can think in terms of tasks while the runtime manages the mapping to hardware.

Go’s answer to this problem is the goroutine: a user-level thread that starts with a 2 KB stack, is scheduled cooperatively (with asynchronous preemption added in Go 1.14), and can be created in about a microsecond. The Go runtime multiplexes potentially millions of goroutines onto a small pool of OS threads, handling all the complexity of stack growth, blocking system calls, and work stealing behind the scenes. This module examines how that machinery works, starting from the OS primitives (processes and threads) that form the foundation, then diving into Go’s runtime representations of OS threads (the m struct) and goroutines (the g struct), and finally tracing the lifecycle of a goroutine through its state transitions.

Learning Objectives

By the end of this module, students will be able to:

1. Explain the difference between OS processes, OS threads, and goroutines
2. Describe the cost tradeoffs between OS threads and goroutines
3. Read and understand the m struct (OS thread) and g struct (goroutine) in the Go runtime
4. Trace how Go creates OS threads on Linux (clone) and Darwin (pthread_create)
5. Enumerate goroutine states and explain their transitions

Part 1: OS Processes (10 min)

What is a Process?

A process is the OS’s unit of resource ownership and protection. Each process has:

• Address space: A virtual memory mapping (text, data, heap, stack segments) isolated from other processes via page tables.
• File descriptor table: An array of open file handles (files, sockets, pipes) private to the process.
• Process table entry: Kernel metadata including PID, parent PID, credentials (uid/gid), signal handlers, resource limits, and scheduling state.
• One or more threads of execution.

Process Creation

On Unix, processes are created with fork(), which duplicates the entire address space (copy-on-write in practice), followed by exec() to load a new program image. This is expensive:

• Page table duplication
• File descriptor table duplication
• Kernel data structure allocation
• TLB flush on the new process

Why Processes Are Too Heavy for Concurrency

If you want 10,000 concurrent activities, creating 10,000 processes is impractical:

• Each gets its own address space (page tables, memory mappings)
• Communication requires IPC (pipes, shared memory, sockets) – all involve kernel transitions
• Context switching between processes requires TLB flushes

This motivates threads: multiple execution contexts sharing a single address space.

Part 2: OS Threads (10 min)

What is a Thread?

A thread (sometimes called a “kernel thread” or “OS thread”) is an independent execution context within a process. Threads within the same process share:

• Address space (code, heap, global data)
• File descriptor table
• Signal handlers
• Process ID

Each thread has its own:

• Stack (typically 1-8 MB, allocated by the OS)
• Register set (saved/restored on context switch)
• Thread ID
• Signal mask
• errno (on Linux, thread-local storage)

Why OS Threads Are Expensive

There are three major costs:

1. Stack size (1-8 MB per thread)

The OS must allocate a fixed-size stack for each thread at creation time. The default on Linux is typically 8 MB (though only pages actually touched are physically allocated). On macOS, the default pthread stack is 512 KB for secondary threads. Even with lazy allocation, the virtual address space is reserved upfront. With 10,000 threads, that is 10-80 GB of virtual address space for stacks alone.

2. Context switch cost (~1-10 microseconds)

Switching between OS threads requires:

• Saving all CPU registers to kernel memory
• Switching kernel stack pointers
• Potentially flushing CPU caches and TLB entries
• Restoring registers for the new thread
• Returning from kernel mode to user mode

This takes roughly 1-10 microseconds depending on hardware and how much cache state is disrupted.

3. Kernel involvement in every operation

Creating, destroying, blocking, and waking threads all require system calls. The kernel must maintain scheduling data structures, and the scheduler itself runs with O(1) or O(log n) complexity per operation, but the constant factors include kernel entry/exit overhead.

Thread Creation on Linux and Darwin

Go creates OS threads through platform-specific mechanisms. On Linux, it uses the clone() system call directly. On Darwin (macOS), it uses pthread_create().

Linux: newosproc calls clone()

From src/runtime/os_linux.go, lines 170-201:

func newosproc(mp *m) {
    stk := unsafe.Pointer(mp.g0.stack.hi)
    // Disable signals during clone, so that the new thread starts
    // with signals disabled. It will enable them in minit.
    var oset sigset
    sigprocmask(_SIG_SETMASK, &sigset_all, &oset)
    ret := retryOnEAGAIN(func() int32 {
        r := clone(cloneFlags, stk, unsafe.Pointer(mp),
            unsafe.Pointer(mp.g0), unsafe.Pointer(abi.FuncPCABI0(mstart)))
        if r >= 0 {
            return 0
        }
        return -r
    })
    sigprocmask(_SIG_SETMASK, &oset, nil)
    // ...error handling...
}

The actual clone system call is issued from assembly in src/runtime/sys_linux_amd64.s, lines 574-623:

TEXT runtime·clone(SB),NOSPLIT|NOFRAME,$0
    MOVL    flags+0(FP), DI
    MOVQ    stk+8(FP), SI
    MOVQ    $0, DX
    MOVQ    $0, R10
    MOVQ    $0, R8
    // Copy mp, gp, fn off parent stack for use by child.
    MOVQ    mp+16(FP), R13
    MOVQ    gp+24(FP), R9
    MOVQ    fn+32(FP), R12
    // ...TLS setup...
    MOVL    $SYS_clone, AX
    SYSCALL

Key observation: the clone() call receives the flags, the new stack pointer, and pointers to the m and g0 structs. The child thread starts executing at mstart, which enters the scheduler loop.

Darwin: newosproc calls pthread_create()

From src/runtime/os_darwin.go, lines 224-258:

func newosproc(mp *m) {
    stk := unsafe.Pointer(mp.g0.stack.hi)
    // Initialize an attribute object.
    var attr pthreadattr
    var err int32
    err = pthread_attr_init(&attr)
    // ...
    // Find out OS stack size for our own stack guard.
    var stacksize uintptr
    if pthread_attr_getstacksize(&attr, &stacksize) != 0 { ... }
    mp.g0.stack.hi = stacksize // for mstart

    // Tell the pthread library we won't join with this thread.
    if pthread_attr_setdetachstate(&attr, _PTHREAD_CREATE_DETACHED) != 0 { ... }

    // Finally, create the thread.
    var oset sigset
    sigprocmask(_SIG_SETMASK, &sigset_all, &oset)
    err = retryOnEAGAIN(func() int32 {
        return pthread_create(&attr, abi.FuncPCABI0(mstart_stub), unsafe.Pointer(mp))
    })
    // ...
}

On Darwin, Go cannot use clone() (that is Linux-specific), so it goes through the POSIX threads API. The thread starts at mstart_stub, which does low-level setup and then calls mstart.

Part 3: The M Struct – Go’s Representation of an OS Thread (10 min)

In Go’s runtime, every OS thread is represented by an m struct (M for “machine”). This is defined in src/runtime/runtime2.go, lines 618-727.

Key Fields

// src/runtime/runtime2.go, lines 618-727 (https://cs.opensource.google/go/go/+/refs/tags/go1.26.1:src/runtime/runtime2.go;l=618)
type m struct {
    g0      *g     // goroutine with scheduling stack
    morebuf gobuf  // gobuf arg to morestack

    procid       uint64            // for debuggers, but offset not hard-coded
    gsignal      *g                // signal-handling g
    sigmask      sigset            // storage for saved signal mask
    tls          [tlsSlots]uintptr // thread-local storage
    mstartfn     func()
    curg         *g       // current running goroutine

    p       puintptr       // currently attached P (nil if not executing Go code)
    nextp   puintptr       // the next P to install before executing
    oldp    puintptr       // the P that was attached before executing a syscall
    id      int64
    spinning bool          // m is out of work and is actively looking for work
    blocked  bool          // m is blocked on a note
    park     note
    alllink  *m            // on allm
    schedlink muintptr
    lockedg  guintptr
    // ...
}

Important Concepts

g0: The scheduling stack. Every M has a special goroutine called g0 that runs scheduler code. When the scheduler needs to make decisions (find a runnable goroutine, handle a syscall return, etc.), it switches to g0’s stack. This is necessary because the scheduler cannot run on a user goroutine’s stack – that stack might be tiny (2 KB) and might need to be moved.

curg: The current goroutine. This points to the user goroutine currently executing on this thread. When curg is nil, the M is running scheduler code on g0.

p: The attached P. An M must have an associated P (processor) to execute Go code. The M-P binding is how the runtime controls parallelism: GOMAXPROCS sets the number of Ps, which limits how many Ms can execute Go code simultaneously.

spinning: When an M has no work, it may enter a “spinning” state where it actively looks for work to steal from other Ps. This avoids the latency of parking and unparking threads.

park: A note (futex-based synchronization primitive) used to park the M when there is no work. The M sleeps on this note and is woken when new work appears.

Part 4: Goroutines – The G Struct (15 min)

A goroutine is Go’s lightweight, user-level thread. Each goroutine is represented by a g struct, defined in src/runtime/runtime2.go, lines 473-598.

Key Fields

// src/runtime/runtime2.go, lines 473-598 (https://cs.opensource.google/go/go/+/refs/tags/go1.26.1:src/runtime/runtime2.go;l=473)
type g struct {
    // Stack parameters.
    // stack describes the actual stack memory: [stack.lo, stack.hi).
    // stackguard0 is the stack pointer compared in the Go stack growth
    // prologue. It is stack.lo+StackGuard normally, but can be
    // StackPreempt to trigger a preemption.
    stack       stack   // offset known to runtime/cgo
    stackguard0 uintptr // offset known to liblink
    stackguard1 uintptr // offset known to liblink

    _panic    *_panic // innermost panic
    _defer    *_defer // innermost defer
    m         *m      // current m; offset known to arm liblink
    sched     gobuf   // saved context (sp, pc, etc.) for scheduling
    syscallsp uintptr // if status==Gsyscall, syscallsp = sched.sp
    syscallpc uintptr // if status==Gsyscall, syscallpc = sched.pc

    param        unsafe.Pointer
    atomicstatus atomic.Uint32
    goid         uint64
    schedlink    guintptr
    waitsince    int64      // approx time when the g became blocked
    waitreason   waitReason // if status==Gwaiting

    preempt       bool // preemption signal
    preemptStop   bool // transition to _Gpreempted on preemption

    lockedm   muintptr       // locked to this m
    parentGoid uint64         // goid of goroutine that created this goroutine
    gopc       uintptr        // pc of go statement that created this goroutine
    startpc    uintptr        // pc of goroutine function

    waiting    *sudog         // sudog structures this g is waiting on
    timer      *timer         // cached timer for time.Sleep
    // ...
}

The Stack

// src/runtime/runtime2.go, lines 462-465 (https://cs.opensource.google/go/go/+/refs/tags/go1.26.1:src/runtime/runtime2.go;l=462)
type stack struct {
    lo uintptr
    hi uintptr
}

Goroutine stacks start small. From src/runtime/stack.go, line 78:

// The minimum size of stack used by Go code
stackMin = 2048

That is 2 KB – compare this to the 1-8 MB default for OS thread stacks. Goroutine stacks grow dynamically: every function prologue checks if the stack needs to grow. If it does, the runtime allocates a new, larger stack (typically 2x), copies the old stack contents, and updates all pointers. This is called a copyable stack or segmented stack (Go used segmented stacks historically but switched to contiguous, copyable stacks in Go 1.4).

The gobuf: Saved Execution Context

// src/runtime/runtime2.go, lines 303-322 (https://cs.opensource.google/go/go/+/refs/tags/go1.26.1:src/runtime/runtime2.go;l=303)
type gobuf struct {
    sp   uintptr
    pc   uintptr
    g    guintptr
    ctxt unsafe.Pointer
    lr   uintptr
    bp   uintptr // for framepointer-enabled architectures
}

When a goroutine is descheduled, its execution state (stack pointer, program counter, base pointer) is saved into g.sched (a gobuf). When it is scheduled again, these registers are restored. This is the core of user-level context switching – no kernel involvement needed.

Contrast with an OS context switch, which must save/restore:

• All general-purpose registers (16 on amd64)
• Floating-point / SSE / AVX state (potentially hundreds of bytes)
• Kernel stack pointer
• Various control registers

The goroutine context switch saves only 6 values. The Go compiler ensures that no other registers hold live values at safe points where goroutines can be descheduled.

Part 5: Goroutine States (10 min)

Goroutine states are defined as constants in src/runtime/runtime2.go, lines 17-120. The primary states are:

State Definitions

// src/runtime/runtime2.go, lines 17-120 (https://cs.opensource.google/go/go/+/refs/tags/go1.26.1:src/runtime/runtime2.go;l=17)
const (
    // _Gidle means this goroutine was just allocated and has not
    // yet been initialized.
    _Gidle = iota // 0

    // _Grunnable means this goroutine is on a run queue. It is
    // not currently executing user code. The stack is not owned.
    _Grunnable // 1

    // _Grunning means this goroutine may execute user code. The
    // stack is owned by this goroutine. It is assigned an M and
    // usually has a P.
    _Grunning // 2

    // _Gsyscall means this goroutine is executing a system call.
    // It is not executing user code. The stack is owned by this
    // goroutine. It is assigned an M.
    _Gsyscall // 3

    // _Gwaiting means this goroutine is blocked in the runtime.
    // It is not executing user code. It is not on a run queue,
    // but should be recorded somewhere (e.g., a channel wait
    // queue) so it can be ready()d when necessary.
    _Gwaiting // 4

    // _Gdead means this goroutine is currently unused. It may be
    // just exited, on a free list, or just being initialized.
    _Gdead // 6

    // _Gcopystack means this goroutine's stack is being moved.
    _Gcopystack // 8

    // _Gpreempted means this goroutine stopped itself for a
    // suspendG preemption.
    _Gpreempted // 9
)

State Transition Diagram

                    newproc1()
    _Gidle ──────────────────────► _Gdead ─────► _Grunnable
      │                              ▲               │
      │                              │               │
      │                         goexit0()            │ execute()
      │                              │               │
      │                              │               ▼
      │                         _Grunning ◄──── _Grunning
      │                              │               │
      │                              │               │
      │                         gopark()          entersyscall()
      │                              │               │
      │                              ▼               ▼
      │                         _Gwaiting        _Gsyscall
      │                              │               │
      │                              │               │
      │                         goready()        exitsyscall()
      │                              │               │
      │                              ▼               │
      │                         _Grunnable ◄─────────┘
      │
      └──────────────────────────────────────────────┘

Key State Transitions

Stack Ownership

A critical concept: the goroutine state controls who owns the stack.

• _Grunning: The goroutine owns its stack. Only it can read/write the stack.
• _Grunnable: The stack is not owned. The scheduler may inspect it.
• _Gwaiting: The stack is not owned, except that a channel operation may read or write parts of it under the channel lock.
• _Gsyscall: The goroutine owns its stack, but the P may be detached.
• _Gcopystack: The stack is being moved by the goroutine that initiated the copy.

This ownership model is essential for garbage collection: the GC needs to scan goroutine stacks, but can only do so when it owns the stack (i.e., when the goroutine is not in _Grunning).

Part 6: Cost Comparison – Goroutines vs. OS Threads (5 min)

Why This Matters

A typical web server might have 10,000 concurrent connections. With the thread-per-connection model:

• 10,000 threads * 8 MB stack = 80 GB of virtual address space
• 10,000 kernel scheduling entities
• Context switches hit kernel on every transition

With goroutine-per-connection:

• 10,000 goroutines * 2 KB stack = 20 MB of memory
• ~4 kernel threads (GOMAXPROCS) for scheduling
• Context switches stay in user space

This is the fundamental insight behind Go’s concurrency model: goroutines provide the programming convenience of threads at a fraction of the cost.

Key Takeaways

1. Processes provide isolation but are too expensive for fine-grained concurrency.
2. OS threads share an address space but still carry significant kernel overhead (large stacks, expensive context switches).
3. Goroutines are user-level threads managed entirely by the Go runtime, with 2 KB stacks and sub-microsecond context switches.
4. The m struct represents an OS thread; the g struct represents a goroutine. The runtime multiplexes many Gs onto fewer Ms.
5. Goroutine states (_Gidle, _Grunnable, _Grunning, _Gsyscall, _Gwaiting, _Gdead) control stack ownership and are central to correctness of both the scheduler and the garbage collector.
6. On Linux, Go creates OS threads with the clone() system call; on Darwin, with pthread_create(). Both paths ultimately start the new thread at mstart, entering the scheduler loop.

Discussion Questions

1. Why does Go use clone() directly on Linux instead of pthread_create()? What control does this give the runtime?
2. If goroutines are so cheap, why does Go still need OS threads at all? Why not implement everything in user space?
3. What would happen if goroutine stacks were fixed-size (say, 1 MB) instead of dynamically growing from 2 KB? How would this affect the design of Go programs?
4. The g struct stores stackguard0, which can be set to StackPreempt to trigger preemption. How is this different from how the OS preempts threads? (We will explore this more in Module 5.)

Further Reading

• Go source: src/runtime/runtime2.go (struct definitions)
• Go source: src/runtime/os_linux.go and src/runtime/os_darwin.go (thread creation)
• Go source: src/runtime/sys_linux_amd64.s (clone syscall wrapper)
• Go source: src/runtime/stack.go (stack size constants)
• Scalable Go Scheduler Design Doc: https://golang.org/s/go11sched

Module 3: Processes, Threads, and Goroutines (G, M structs)

Comprehension Check

Question 1 (Code Reading)

Consider these fields from the g struct in runtime/runtime2.go:

type g struct {
	stack       stack   // offset known to runtime/cgo
	stackguard0 uintptr // offset known to liblink
	stackguard1 uintptr // offset known to liblink

	_panic    *_panic // innermost panic - offset known to liblink
	_defer    *_defer // innermost defer
	m         *m      // current m; offset known to arm liblink
	sched     gobuf
	syscallsp uintptr // if status==Gsyscall, syscallsp = sched.sp to use during gc
	syscallpc uintptr // if status==Gsyscall, syscallpc = sched.pc to use during gc
	...
}

Why does the g struct store both sched.sp/sched.pc and separate syscallsp/syscallpc fields? Under what circumstances are the syscall variants used?

Question 2 (Short Answer)

Compare and contrast goroutines with: (a) OS threads (pthreads) (b) Green threads (as in early Java or Erlang processes)

What specific design choices make goroutines different from both?

Question 3 (True/False with Explanation)

True or False: A goroutine is always executed by the same OS thread throughout its lifetime.

Question 4 (Code Reading)

Consider these fields from the m struct:

type m struct {
	g0      *g     // goroutine with scheduling stack
	...
	curg    *g     // current running goroutine
	p       puintptr // attached p for executing go code (nil if not executing go code)
	nextp   puintptr
	oldp    puintptr // the p that was attached before executing a syscall
	...
	spinning bool
}

What is g0 and why does each M need a separate goroutine for scheduling? Why can’t the scheduler code run on the current goroutine’s stack?

Question 5 (Short Answer)

What are the possible states a goroutine (g) can be in? Describe the transitions between _Grunnable, _Grunning, _Gwaiting, and _Gsyscall.

Module 4: The Go Scheduler

Duration: 75 minutes

Background: CPU Scheduling From Hardware to User Space

CPU scheduling is one of the oldest and most fundamental problems in operating systems. At its core, the question is deceptively simple: given more work than processors to run it on, which task should run next? But behind that simplicity lies a web of competing goals. We want fairness, so that no process starves while others monopolize the CPU. We want high throughput, so the system completes as much useful work as possible. And we want low latency, so interactive tasks feel responsive to users. These goals are often in tension – optimizing for one can hurt another – and the history of OS scheduling is largely the story of navigating those trade-offs with increasingly clever designs. Early systems used simple first-come, first-served (FCFS) scheduling, then introduced preemptive round-robin to bound response times, and eventually developed multilevel feedback queues (MLFQ) that adapt scheduling priority based on observed process behavior.

Linux’s scheduling evolution illustrates how real systems have tackled these problems at scale. The 2.6 kernel introduced the O(1) scheduler, which used active and expired priority arrays to achieve constant-time scheduling decisions – a big improvement over the earlier O(n) scheduler that scanned every process on each context switch. But the O(1) scheduler’s heuristics for distinguishing interactive and batch processes were fragile and hard to tune. In 2007, Ingo Molnar replaced it with the Completely Fair Scheduler (CFS), which took a radically different approach: instead of priority arrays and heuristics, CFS tracks each task’s virtual runtime – a measure of how much CPU time it has received, weighted by its priority (nice value). Tasks are stored in a red-black tree ordered by virtual runtime, and the scheduler always picks the task with the smallest virtual runtime, achieving O(log n) scheduling with an elegant fairness guarantee. CFS served Linux well for over 15 years, but in 2023 (kernel 6.6), it was replaced by the EEVDF (Earliest Eligible Virtual Deadline First) scheduler. EEVDF builds on the same virtual-runtime foundation but adds a virtual deadline concept, allowing the scheduler to make better decisions about latency-sensitive tasks without the ad hoc “sleeper bonus” heuristics that CFS had accumulated over the years. The result is more predictable latency behavior with a cleaner theoretical basis.

Modern multicore hardware introduces another dimension to the scheduling problem: a single global run queue protected by one lock becomes a severe bottleneck when dozens or hundreds of cores contend for it. Linux solves this with per-CPU run queues – each processor core maintains its own queue of runnable tasks, and scheduling decisions are made locally without cross-core locking in the common case. This dramatically reduces contention, but it creates a new challenge: load balancing. If one core’s queue drains while another is overloaded, the system wastes hardware parallelism. Linux uses periodic load balancing and idle-core work stealing to redistribute tasks, with a topology-aware approach that accounts for the cache-sharing relationships between cores (preferring to migrate tasks within the same NUMA node or LLC domain before moving them farther away).

The kernel scheduler works at the level of OS threads, but many applications need concurrency at a much finer granularity. This is the domain of user-level scheduling, where a runtime manages its own lightweight tasks on top of OS threads. The M:N threading model – multiplexing M user-level threads onto N kernel threads – has a long and checkered history. Sun’s Solaris operating system experimented with M:N threading in the 1990s, and the Linux community explored it with the NGPT (Next Generation POSIX Threads) project in the early 2000s. Both efforts were largely abandoned in favor of 1:1 threading (one user thread per kernel thread), which was simpler to implement correctly and avoided thorny problems like priority inversion between the user-level and kernel schedulers. But the 1:1 model has costs: OS threads are heavyweight (typically consuming 1-8 MB of stack), expensive to create, and context-switching between them requires entering the kernel. Runtimes like the Erlang BEAM VM sidestepped this by implementing their own preemptive schedulers for lightweight processes, and Java’s ForkJoinPool uses work-stealing to efficiently distribute fine-grained tasks. These user-level approaches trade generality for performance within their specific domains.

Go revived the M:N model with a design that learns from earlier failures. Rather than trying to be a general-purpose threading library, Go’s scheduler is tightly integrated with the language runtime, the compiler, and the garbage collector. It can insert preemption points at compile time, it knows when a goroutine is about to make a blocking system call, and it can grow goroutine stacks dynamically (starting at just a few kilobytes). The result is a system that supports millions of concurrent goroutines with efficient scheduling – and at its heart is the GMP model we will study in this module. Understanding the design of Go’s scheduler is not just about learning Go; it is a case study in how the classic ideas of per-CPU run queues, work stealing, and cooperative-preemptive hybrid scheduling come together in a modern, practical system.

Learning Objectives

By the end of this module, students will be able to:

1. Explain the GMP model and why the P (processor) abstraction exists
2. Describe the P struct’s role: local run queue, caching, logical processor
3. Read the scheduling loop (schedule -> findRunnable -> execute)
4. Trace a goroutine through its complete lifecycle: creation, execution, blocking, waking, exit
5. Explain the run queue data structures and the runnext optimization

Part 1: The GMP Model (10 min)

Overview

The Go scheduler is built around three core abstractions, described at the top of src/runtime/proc.go, lines 24-34:

// src/runtime/proc.go lines 24-34
// Goroutine scheduler
// The scheduler's job is to distribute ready-to-run goroutines over
// worker threads.
//
// The main concepts are:
// G - goroutine.
// M - worker thread, or machine.
// P - processor, a resource that is required to execute Go code.
//     M must have an associated P to execute Go code, however it can be
//     blocked or in a syscall w/o an associated P.
//
// Design doc at https://golang.org/s/go11sched.

Why Three Entities?

The original Go scheduler (before Go 1.1) only had G and M. Every M had a global mutex-protected run queue. This created severe contention: every time any goroutine was created, scheduled, or completed, it had to acquire the global lock. With 8+ cores, the lock became a bottleneck.

The solution was to introduce P (processor) – a logical processor that provides:

1. A local run queue per P (no lock needed for local operations)
2. Local caches (mcache, defer pool, sudog cache) to avoid global allocation locks
3. Bounded parallelism – GOMAXPROCS controls the number of Ps, which bounds how many Ms can execute Go code simultaneously

The relationship:

G (goroutine)  -- the work to be done
M (machine)    -- the OS thread that does the work
P (processor)  -- the "ticket" that allows an M to do Go work

Rule: an M must hold a P to execute Go code.
      An M may exist without a P (blocked in syscall, idle).
      A P always has a local run queue of Gs.

Worker Thread Parking/Unparking

The scheduler must balance two competing goals (from src/runtime/proc.go, lines 36-43):

1. Keep enough running worker threads to utilize available hardware parallelism
2. Park excessive running worker threads to conserve CPU resources and power

This is hard because scheduler state is distributed (per-P queues) and we cannot predict the future (a parked thread may be needed immediately).

Part 2: The P Struct (10 min)

The P struct is defined in src/runtime/runtime2.go, lines 772-928.

Key Fields

From src/runtime/runtime2.go, lines 772-818:

// src/runtime/runtime2.go lines 772-835
type p struct {
    id          int32
    status      uint32     // one of pidle/prunning/...
    link        puintptr
    schedtick   uint32     // incremented on every scheduler call
    syscalltick uint32     // incremented on every system call
    sysmontick  sysmontick // last tick observed by sysmon
    m           muintptr   // back-link to associated m (nil if idle)
    mcache      *mcache
    raceprocctx uintptr

    deferpool    []*_defer // pool of available defer structs
    deferpoolbuf [32]*_defer

    // Cache of goroutine ids, amortizes accesses to sched.goidgen.
    goidcache    uint64
    goidcacheend uint64

    // Queue of runnable goroutines. Accessed without lock.
    runqhead uint32
    runqtail uint32
    runq     [256]guintptr
    // runnext, if non-nil, is a runnable G that was ready'd by
    // the current G and should be run next instead of what's in
    // runq if there's time remaining in the running G's time
    // slice. It will inherit the time left in the current time
    // slice.
    runnext guintptr

    // Available G's (status == Gdead)
    gFree gList

    sudogcache []*sudog
    sudogbuf   [128]*sudog
    // ...
}

The Local Run Queue

The most important part of the P is its local run queue:

runqhead uint32
runqtail uint32
runq     [256]guintptr

This is a lock-free, fixed-size circular buffer of 256 goroutine pointers. Key properties:

• No lock needed for the owning P to push/pop
• Lock-free stealing by other Ps using atomic CAS operations
• Fixed size of 256 – if it overflows, half the queue is moved to the global run queue

The runnext Slot

From src/runtime/runtime2.go, lines 806-818:

// src/runtime/runtime2.go, lines 806-818
// runnext, if non-nil, is a runnable G that was ready'd by
// the current G and should be run next instead of what's in
// runq if there's time remaining in the running G's time
// slice. It will inherit the time left in the current time
// slice. If a set of goroutines is locked in a
// communicate-and-wait pattern, this schedules that set as a
// unit and eliminates the (potentially large) scheduling
// latency that otherwise arises from adding the ready'd
// goroutines to the end of the run queue.
//
// Note that while other P's may atomically CAS this to zero,
// only the owner P can CAS it to a valid G.
runnext guintptr

This is an optimization for the common producer-consumer pattern: when goroutine A sends on a channel and wakes goroutine B, B is placed in runnext so it runs immediately (inheriting A’s time slice). This avoids the latency of going to the back of the run queue.

P Local Caches

The P also holds various caches to reduce contention on global data structures:

• mcache – memory allocator cache (avoids locking the heap for small allocations)
• deferpool – pool of _defer structs
• sudogcache – pool of sudog structs (used for channel operations)
• gFree – pool of dead goroutines for reuse
• goidcache – batch of goroutine IDs (avoids atomically incrementing a global counter for every new goroutine)

Part 3: The schedt Struct – Global Scheduler State (5 min)

The global scheduler state is held in a single schedt struct (variable sched), defined in src/runtime/runtime2.go, lines 930-1053.

Key Fields

// src/runtime/runtime2.go, lines 930-976
type schedt struct {
    goidgen    atomic.Uint64
    lastpoll   atomic.Int64 // time of last network poll
    pollUntil  atomic.Int64 // time to which current poll is sleeping

    lock mutex

    midle        listHeadManual // idle m's waiting for work
    nmidle       int32          // number of idle m's
    nmidlelocked int32          // number of locked m's waiting for work
    mnext        int64          // number of m's that have been created
    maxmcount    int32          // maximum number of m's allowed (or die)

    ngsys        atomic.Int32   // number of system goroutines

    pidle        puintptr       // idle p's
    npidle       atomic.Int32
    nmspinning   atomic.Int32   // spinning M count

    // Global runnable queue.
    runq gQueue

    // Global cache of dead G's.
    gFree struct {
        lock    mutex
        stack   gList // Gs with stacks
        noStack gList // Gs without stacks
    }

    // freem is the list of m's waiting to be freed
    freem *m

    gcwaiting  atomic.Bool // gc is waiting to run
    // ...
}

Global vs. Local State

The design principle: fast paths use local, per-P state; slow paths fall back to global state under a lock.

Part 4: The gobuf Struct – Saved Register State (5 min)

When a goroutine is descheduled, its register state is saved into a gobuf. From src/runtime/runtime2.go, lines 303-322:

// src/runtime/runtime2.go, lines 303-322
type gobuf struct {
    // The offsets of sp, pc, and g are known to (hard-coded in) libmach.
    sp   uintptr
    pc   uintptr
    g    guintptr
    ctxt unsafe.Pointer
    lr   uintptr
    bp   uintptr // for framepointer-enabled architectures
}

This is stored in g.sched for each goroutine. The key operations:

• Save context: When a goroutine yields (via gopark, Gosched, or preemption), its SP, PC, and BP are written to g.sched.
• Restore context: When the scheduler picks a goroutine to run, gogo(&gp.sched) loads SP, PC, and BP from the gobuf and jumps to the saved PC.

Why only 6 fields? The Go compiler cooperates with the scheduler to ensure that at any point where a goroutine can be descheduled (a “safe point”), all live values are on the stack, not in registers. This means the context switch only needs to save the stack pointer and program counter – the values themselves are preserved on the goroutine’s stack.

Part 5: The Scheduling Loop (15 min)

schedule()

The scheduling loop is the heart of the runtime. Every M that is ready to do work calls schedule(), which finds a runnable goroutine and starts executing it. From src/runtime/proc.go, lines 4135-4231:

// src/runtime/proc.go, lines 4135-4231
func schedule() {
    mp := getg().m

    if mp.locks != 0 {
        throw("schedule: holding locks")
    }

    if mp.lockedg != 0 {
        stoplockedm()
        execute(mp.lockedg.ptr(), false) // Never returns.
    }

    // We should not schedule away from a g that is executing a cgo call,
    // since the cgo call is using the m's g0 stack.
    if mp.incgo {
        throw("schedule: in cgo")
    }

top:
    pp := mp.p.ptr()
    pp.preempt = false

    // Safety check: if we are spinning, the run queue should be empty.
    if mp.spinning && (pp.runnext != 0 || pp.runqhead != pp.runqtail) {
        throw("schedule: spinning with local work")
    }

    gp, inheritTime, tryWakeP := findRunnable() // blocks until work is available

    // This thread is going to run a goroutine and is not spinning anymore,
    // so if it was marked as spinning we need to reset it now and
    // potentially start a new spinning M.
    if mp.spinning {
        resetspinning()
    }

    // ...disabled scheduling, locked-g handling...

    execute(gp, inheritTime)
}

The Flow

schedule()
    |
    +--> findRunnable()     // find a G to run (may block)
    |       |
    |       +--> returns (gp, inheritTime, tryWakeP)
    |
    +--> resetspinning()    // if we were spinning, stop
    |
    +--> execute(gp, inheritTime)  // start running gp
            |
            +--> gogo(&gp.sched)   // restore context, jump to user code
                                    // NEVER RETURNS to schedule()

Important: execute() never returns. When the goroutine eventually yields, the M will re-enter schedule() from a different path (via mcall which switches to g0 and calls the appropriate handler).

execute()

From src/runtime/proc.go, lines 3331-3383:

// src/runtime/proc.go, lines 3331-3383
func execute(gp *g, inheritTime bool) {
    mp := getg().m

    // Assign gp.m before entering _Grunning so running Gs have an M.
    mp.curg = gp
    gp.m = mp
    casgstatus(gp, _Grunnable, _Grunning)
    gp.waitsince = 0
    gp.preempt = false
    gp.stackguard0 = gp.stack.lo + stackGuard
    if !inheritTime {
        mp.p.ptr().schedtick++
    }

    // Check whether the profiler needs to be turned on or off.
    hz := sched.profilehz
    if mp.profilehz != hz {
        setThreadCPUProfiler(hz)
    }

    trace := traceAcquire()
    if trace.ok() {
        trace.GoStart()
        traceRelease(trace)
    }

    gogo(&gp.sched)
}

Key steps:

1. Bind M and G together (mp.curg = gp; gp.m = mp)
2. Transition status: _Grunnable -> _Grunning
3. Reset preemption state
4. Increment schedtick if this is a new time slice (not inherited)
5. Call gogo() to restore the goroutine’s saved context and jump to user code

Part 6: findRunnable() – Finding Work (15 min)

findRunnable() is the most complex function in the scheduler. It implements a priority-ordered search for work. From src/runtime/proc.go, lines 3389-3658.

The Priority Order

// src/runtime/proc.go, lines 3389-3538
func findRunnable() (gp *g, inheritTime, tryWakeP bool) {
    mp := getg().m

top:
    pp := mp.p.ptr()
    if sched.gcwaiting.Load() {
        gcstopm()                           // 0. GC is running, stop
        goto top
    }

    // ...timer checks...

    // 1. Try to schedule the trace reader.
    if traceEnabled() || traceShuttingDown() {
        gp := traceReader()
        if gp != nil { return gp, false, true }
    }

    // 2. Try to schedule a GC worker.
    if gcBlackenEnabled != 0 {
        gp, tnow := gcController.findRunnableGCWorker(pp, now)
        if gp != nil { return gp, false, true }
    }

    // 3. Check global runnable queue once in a while (fairness).
    if pp.schedtick%61 == 0 && !sched.runq.empty() {
        lock(&sched.lock)
        gp := globrunqget()
        unlock(&sched.lock)
        if gp != nil { return gp, false, false }
    }

    // 4. Wake up finalizer/cleanup goroutines if needed.
    // ...

    // 5. Local run queue.
    if gp, inheritTime := runqget(pp); gp != nil {
        return gp, inheritTime, false
    }

    // 6. Global run queue (take a batch).
    if !sched.runq.empty() {
        lock(&sched.lock)
        gp, q := globrunqgetbatch(int32(len(pp.runq)) / 2)
        unlock(&sched.lock)
        if gp != nil {
            runqputbatch(pp, &q)
            return gp, false, false
        }
    }

    // 7. Poll network (non-blocking).
    if netpollinited() && netpollAnyWaiters() && ... {
        list, delta := netpoll(0)
        if !list.empty() {
            gp := list.pop()
            injectglist(&list)
            return gp, false, false
        }
    }

    // 8. Steal work from other Ps.
    if mp.spinning || 2*sched.nmspinning.Load() < gomaxprocs-sched.npidle.Load() {
        if !mp.spinning { mp.becomeSpinning() }
        gp, inheritTime, tnow, w, newWork := stealWork(now)
        if gp != nil { return gp, inheritTime, false }
    }

    // 9. Try idle-time GC marking.
    // ...

    // 10. Give up: release P, park the M.
    lock(&sched.lock)
    // ...re-check global queue, needspinning...
    releasep()
    pidleput(pp, now)
    unlock(&sched.lock)
    // ...park the M, wait for wakeup...
}

The Priority Order Summarized

The 1/61 Fairness Check

From src/runtime/proc.go, line 3443:

// src/runtime/proc.go, line 3443
if pp.schedtick%61 == 0 && !sched.runq.empty() {

Every 61st scheduling decision, the scheduler checks the global queue before the local queue. Without this, two goroutines on the same P that continuously spawn each other could starve goroutines on the global queue indefinitely. The number 61 is prime (to avoid resonance with other periodicities) and represents a balance between fairness and fast-path efficiency.

Work Stealing

When the local queue is empty and the global queue is empty, the scheduler attempts to steal work from other Ps. From src/runtime/proc.go, line 3522:

// src/runtime/proc.go, line 3522
gp, inheritTime, tnow, w, newWork := stealWork(now)

stealWork iterates over all Ps in a random order and attempts to steal half of another P’s local run queue. This is the key mechanism for load balancing across cores. The number of spinning Ms is limited to half the busy Ps to avoid excessive CPU consumption. From src/runtime/proc.go, line 3517:

// src/runtime/proc.go, line 3517
if mp.spinning || 2*sched.nmspinning.Load() < gomaxprocs-sched.npidle.Load() {

Part 7: Goroutine Lifecycle (15 min)

Creation: newproc / newproc1

When Go code executes go f(), the compiler emits a call to newproc. From src/runtime/proc.go, lines 5295-5308:

// src/runtime/proc.go, lines 5295-5308
func newproc(fn *funcval) {
    gp := getg()
    pc := sys.GetCallerPC()
    systemstack(func() {
        newg := newproc1(fn, gp, pc, false, waitReasonZero)

        pp := getg().m.p.ptr()
        runqput(pp, newg, true)

        if mainStarted {
            wakep()
        }
    })
}

Steps:

1. Switch to the system stack (g0)
2. Call newproc1 to allocate and initialize a new G
3. Put the new G on the current P’s local run queue via runqput, with next=true (so it goes into runnext)
4. If the runtime is past initialization, call wakep() to wake an idle M if one exists

newproc1 does the detailed initialization, from src/runtime/proc.go, lines 5313-5427:

// src/runtime/proc.go, lines 5313-5357
func newproc1(fn *funcval, callergp *g, callerpc uintptr,
              parked bool, waitreason waitReason) *g {
    if fn == nil {
        fatal("go of nil func value")
    }

    mp := acquirem() // disable preemption
    pp := mp.p.ptr()
    newg := gfget(pp)        // try to reuse a dead G from the free list
    if newg == nil {
        newg = malg(stackMin) // allocate new G with 2KB stack
        casgstatus(newg, _Gidle, _Gdead)
        allgadd(newg)
    }

    // Set up the stack frame
    memclrNoHeapPointers(unsafe.Pointer(&newg.sched), unsafe.Sizeof(newg.sched))
    newg.sched.sp = sp
    newg.stktopsp = sp
    newg.sched.pc = abi.FuncPCABI0(goexit) + sys.PCQuantum
    newg.sched.g = guintptr(unsafe.Pointer(newg))
    gostartcallfn(&newg.sched, fn)
    newg.parentGoid = callergp.goid
    newg.gopc = callerpc
    newg.startpc = fn.fn
    // ...assign goid, set status to _Grunnable...
}

Key detail: the new goroutine’s PC is set up so that when fn returns, it returns into goexit, which performs cleanup. The gostartcallfn call arranges the stack frame so that fn appears to have been called by goexit.

Another key detail: gfget(pp) tries to reuse a dead goroutine from the P’s local free list (p.gFree) before allocating a new one. This amortizes allocation cost for programs that create and destroy many goroutines.

Execution: execute

Covered above in Part 5. The key transition: _Grunnable -> _Grunning, then gogo(&gp.sched) restores context and begins executing user code.

Blocking: gopark / park_m

gopark and goready (below) are the runtime’s universal blocking and waking primitives. Channels (Module 7), mutexes (Module 6), and I/O (Module 10) all ultimately use gopark to suspend a goroutine and goready to wake it. Understanding this pair is key to understanding all blocking operations in Go.

When a goroutine needs to block (channel operation, mutex, sleep, etc.), it calls gopark. From src/runtime/proc.go, lines 445-463:

// src/runtime/proc.go, lines 445-463
func gopark(unlockf func(*g, unsafe.Pointer) bool, lock unsafe.Pointer,
            reason waitReason, traceReason traceBlockReason, traceskip int) {
    if reason != waitReasonSleep {
        checkTimeouts()
    }
    mp := acquirem()
    gp := mp.curg
    status := readgstatus(gp)
    if status != _Grunning && status != _Gscanrunning {
        throw("gopark: bad g status")
    }
    mp.waitlock = lock
    mp.waitunlockf = unlockf
    gp.waitreason = reason
    mp.waitTraceBlockReason = traceReason
    mp.waitTraceSkip = traceskip
    releasem(mp)
    // can't do anything that might move the G between Ms here.
    mcall(park_m)
}

gopark stores the wait reason and unlock function, then calls mcall(park_m). mcall switches from the user goroutine’s stack to the M’s g0 stack and calls park_m. From src/runtime/proc.go, lines 4253-4302:

// src/runtime/proc.go, lines 4253-4302
func park_m(gp *g) {
    mp := getg().m

    trace := traceAcquire()
    if trace.ok() {
        trace.GoPark(mp.waitTraceBlockReason, mp.waitTraceSkip)
    }
    casgstatus(gp, _Grunning, _Gwaiting)
    if trace.ok() {
        traceRelease(trace)
    }

    dropg()   // disconnect G from M

    if fn := mp.waitunlockf; fn != nil {
        ok := fn(gp, mp.waitlock)
        mp.waitunlockf = nil
        mp.waitlock = nil
        if !ok {
            // The unlock function said "don't park after all"
            casgstatus(gp, _Gwaiting, _Grunnable)
            execute(gp, true) // Schedule it back, never returns.
        }
    }

    // G is now parked. M calls schedule() to find new work.
    schedule()
}

Key steps:

1. Transition status: _Grunning -> _Gwaiting
2. dropg(): disconnect G from M (m.curg = nil; g.m = nil)
3. Call the unlock function (e.g., release a channel lock). If it returns false, the park is cancelled and the goroutine resumes.
4. Call schedule() to find the next goroutine to run

The unlock function pattern is important: it allows the runtime to atomically release a lock and park, avoiding a race where another goroutine could wake us before we are fully parked.

Waking: goready / ready

When a blocked goroutine should be woken (e.g., a value arrives on a channel), goready is called. From src/runtime/proc.go, lines 481-485:

// src/runtime/proc.go, lines 481-485
func goready(gp *g, traceskip int) {
    systemstack(func() {
        ready(gp, traceskip, true)
    })
}

Which calls ready. From src/runtime/proc.go, lines 1120-1140:

// src/runtime/proc.go, lines 1120-1140
func ready(gp *g, traceskip int, next bool) {
    status := readgstatus(gp)

    // Mark runnable.
    mp := acquirem()
    if status&^_Gscan != _Gwaiting {
        dumpgstatus(gp)
        throw("bad g->status in ready")
    }

    // status is Gwaiting or Gscanwaiting, make Grunnable and put on runq
    trace := traceAcquire()
    casgstatus(gp, _Gwaiting, _Grunnable)
    if trace.ok() {
        trace.GoUnpark(gp, traceskip)
        traceRelease(trace)
    }
    runqput(mp.p.ptr(), gp, next)
    wakep()
    releasem(mp)
}

Key steps:

1. Transition status: _Gwaiting -> _Grunnable
2. Put the goroutine on the current P’s local run queue via runqput with next=true, meaning it goes into runnext for immediate scheduling
3. Call wakep() to wake an idle M, since there is now new work available

Exit: goexit0 / gdestroy

When a goroutine’s function returns, it returns into goexit (which was set up during creation). The cleanup happens in goexit0. From src/runtime/proc.go, lines 4491-4501:

// src/runtime/proc.go, lines 4491-4501
func goexit0(gp *g) {
    if goexperiment.RuntimeSecret && gp.secret > 0 {
        memclrNoHeapPointers(unsafe.Pointer(gp.stack.lo),
            gp.stack.hi-gp.stack.lo)
    }
    gdestroy(gp)
    schedule()
}

And gdestroy does the actual cleanup, from src/runtime/proc.go, lines 4503-4539:

// src/runtime/proc.go, lines 4503-4511
func gdestroy(gp *g) {
    mp := getg().m
    pp := mp.p.ptr()

    casgstatus(gp, _Grunning, _Gdead)
    gcController.addScannableStack(pp, -int64(gp.stack.hi-gp.stack.lo))
    if isSystemGoroutine(gp, false) {
        sched.ngsys.Add(-1)
    }
    gp.m = nil
    locked := gp.lockedm != 0
    gp.lockedm = 0
    mp.lockedg = 0
    gp.preemptStop = false
    gp.paniconfault = false
    gp._defer = nil
    gp._panic = nil
    gp.writebuf = nil
    gp.waitreason = waitReasonZero
    gp.param = nil
    gp.labels = nil
    gp.timer = nil

    dropg()
    // ...put gp on free list for reuse (gfput)...
}

Key steps:

1. Transition status: _Grunning -> _Gdead
2. Clear all goroutine-specific state (defers, panics, labels, etc.)
3. dropg(): disconnect G from M
4. Put the G on the P’s free list (gfput) for reuse by newproc1
5. Call schedule() to find the next goroutine

Complete Lifecycle Diagram

                    newproc()
                       |
                   newproc1() -- allocate G, set up stack
                       |
                  runqput(pp, newg, true) -- place on runnext
                       |
                       v
              ┌── _Grunnable ◄────────────────────────┐
              |        |                               |
              |   schedule() -> findRunnable()         |
              |        |                               |
              |   execute()                        ready()
              |        |                          [goready]
              |   casgstatus -> _Grunning              |
              |   gogo(&gp.sched)                      |
              |        |                               |
              |   ┌────┴────┐                          |
              |   |         |                          |
              | gopark   function                      |
              |   |     returns                        |
              |   v         |                          |
              | _Gwaiting   |                          |
              |   |         |              ────────────┘
              |   |         v
              |   |     goexit0()
              |   |         |
              |   |    casgstatus -> _Gdead
              |   |         |
              |   |    gfput() -- place on free list
              |   |         |
              |   |    schedule() -- find next G
              |   |
              |   └─── (woken by another G) ───► _Grunnable
              |
              └─────────── (stolen by another P)

Part 8: Run Queue Operations (10 min)

runqput – Adding to the Local Queue

From src/runtime/proc.go, lines 7478-7520:

// src/runtime/proc.go, lines 7478-7520
func runqput(pp *p, gp *g, next bool) {
    if !haveSysmon && next {
        // Without sysmon, avoid runnext to prevent starvation
        next = false
    }
    if randomizeScheduler && next && randn(2) == 0 {
        next = false
    }

    if next {
    retryNext:
        oldnext := pp.runnext
        if !pp.runnext.cas(oldnext, guintptr(unsafe.Pointer(gp))) {
            goto retryNext
        }
        if oldnext == 0 {
            return
        }
        // Kick the old runnext out to the regular run queue.
        gp = oldnext.ptr()
    }

retry:
    h := atomic.LoadAcq(&pp.runqhead) // load-acquire
    t := pp.runqtail
    if t-h < uint32(len(pp.runq)) {
        pp.runq[t%uint32(len(pp.runq))].set(gp)
        atomic.StoreRel(&pp.runqtail, t+1) // store-release
        return
    }
    if runqputslow(pp, gp, h, t) {
        return
    }
    goto retry
}

The logic:

1. If next=true, place the goroutine in runnext (CAS loop). If there was already something in runnext, kick it to the regular queue.
2. Try to add to the circular buffer (runq[t % 256]). Uses load-acquire on head and store-release on tail for lock-free producer-consumer.
3. If the buffer is full (256 entries), call runqputslow which moves half the queue to the global run queue.

runqputslow – Overflow to Global Queue

From src/runtime/proc.go, lines 7524-7560:

// src/runtime/proc.go, lines 7524-7560
func runqputslow(pp *p, gp *g, h, t uint32) bool {
    var batch [len(pp.runq)/2 + 1]*g

    // Grab a batch from local queue.
    n := t - h
    n = n / 2
    for i := uint32(0); i < n; i++ {
        batch[i] = pp.runq[(h+i)%uint32(len(pp.runq))].ptr()
    }
    if !atomic.CasRel(&pp.runqhead, h, h+n) {
        return false
    }
    batch[n] = gp

    // Link the goroutines and put on global queue.
    for i := uint32(0); i < n; i++ {
        batch[i].schedlink.set(batch[i+1])
    }
    lock(&sched.lock)
    globrunqputbatch(&q)
    unlock(&sched.lock)
    return true
}

When the local queue is full, half of it (128 goroutines) plus the new goroutine are moved to the global queue. This serves dual purposes:

1. Makes room in the local queue
2. Makes work available for other Ps to find (load balancing)

runqget – Taking from the Local Queue

From src/runtime/proc.go, lines 7598-7619:

// src/runtime/proc.go, lines 7598-7619
func runqget(pp *p) (gp *g, inheritTime bool) {
    // If there's a runnext, it's the next G to run.
    next := pp.runnext
    // If the runnext is non-0 and the CAS fails, it could only have
    // been stolen by another P, because other Ps can race to set
    // runnext to 0, but only the current P can set it to non-0.
    if next != 0 && pp.runnext.cas(next, 0) {
        return next.ptr(), true
    }

    for {
        h := atomic.LoadAcq(&pp.runqhead)
        t := pp.runqtail
        if t == h {
            return nil, false
        }
        gp := pp.runq[h%uint32(len(pp.runq))].ptr()
        if atomic.CasRel(&pp.runqhead, h, h+1) {
            return gp, false
        }
    }
}

Key details:

1. Check runnext first. If set, take it and return inheritTime=true (the goroutine inherits the current time slice).
2. Then check the circular buffer. FIFO order (take from head). Uses CAS on head to handle concurrent stealers.
3. The inheritTime flag is important: a goroutine from runnext shares the time slice with the goroutine that readied it, while a goroutine from the regular queue gets a fresh time slice.

Part 9: The runnext Optimization (5 min)

The runnext optimization deserves special attention because it is central to Go’s performance for channel-based patterns.

The Problem

Consider a producer-consumer pattern:

func producer(ch chan int) {
    for i := 0; i < 1000000; i++ {
        ch <- i   // wake consumer, then block
    }
}

func consumer(ch chan int) {
    for v := range ch {
        process(v) // process, then block waiting for next
    }
}

Without runnext, when the producer sends:

1. Consumer is woken, placed at the back of the run queue
2. Producer blocks
3. Other goroutines in the queue run first
4. Eventually consumer runs, processes value, blocks again
5. Repeat with high scheduling latency

The Solution

With runnext:

1. Consumer is woken, placed in runnext (front of queue)
2. Producer blocks
3. Consumer runs immediately (inherits producer’s time slice)
4. Consumer processes value, sends result/blocks
5. Very low scheduling latency

From the comment in src/runtime/runtime2.go, lines 806-818:

If a set of goroutines is locked in a communicate-and-wait pattern, this schedules that set as a unit and eliminates the (potentially large) scheduling latency that otherwise arises from adding the ready’d goroutines to the end of the run queue.

The Starvation Guard

There is a subtle starvation risk: if two goroutines keep waking each other via runnext, they could monopolize the P forever. The sysmon thread (system monitor) guards against this by preempting goroutines that run too long. From src/runtime/proc.go, lines 7479-7489:

// src/runtime/proc.go, lines 7479-7489
if !haveSysmon && next {
    // A runnext goroutine shares the same time slice as the
    // current goroutine. To prevent a ping-pong pair from
    // starving all others, we depend on sysmon to preempt
    // "long-running goroutines".
    next = false
}

On platforms without sysmon (like Wasm), runnext is disabled entirely to prevent starvation.

Key Takeaways

1. The GMP model separates concerns: G is the work, M is the thread, P is the scheduling context. P limits parallelism and provides per-core caches.

2. The P struct holds a local run queue (256-slot lock-free ring buffer), a runnext slot for producer-consumer optimization, and various caches to avoid global locks.

3. The scheduling loop (schedule -> findRunnable -> execute) searches for work in priority order: GC work > global queue (1/61 fairness) > local queue > global queue (batch) > network poll > work stealing > idle GC > park.

4. Goroutine lifecycle: newproc1 creates and enqueues; execute transitions to _Grunning; gopark/park_m blocks with _Gwaiting; goready/ready wakes to _Grunnable; goexit0/gdestroy cleans up to _Gdead and recycles.

5. The runnext optimization enables efficient producer-consumer scheduling by giving the woken goroutine the current time slice.

6. Load balancing happens through work stealing (take half of another P’s queue) and overflow (when local queue is full, half goes to global).

Discussion Questions

1. Why is findRunnable’s priority order designed the way it is? Why does GC work come before user goroutines?

2. The global queue fairness check uses schedtick%61. What would happen if this were schedtick%2 (too frequent) or schedtick%10000 (too rare)?

3. When a goroutine makes a blocking system call, the P is detached from the M. How does the runtime know when the syscall completes? (Hint: look at exitsyscall and sysmon.)

4. Why does runqputslow move half the queue to the global queue instead of just the overflowing goroutine? What would happen with a different strategy?

5. The runnext optimization trades fairness for latency. In what scenarios might you want to disable it?

Further Reading

• Go source: src/runtime/proc.go (scheduler implementation)
• Go source: src/runtime/runtime2.go (data structures)
• Design document: https://golang.org/s/go11sched (Scalable Go Scheduler)
• src/runtime/proc.go, lines 36-97: detailed comment on worker thread parking/unparking strategy

Module 4: The Go Scheduler (GMP Model, schedule(), findRunnable())

Comprehension Check

Question 1 (Code Reading)

Study this excerpt from schedule() in runtime/proc.go:

func schedule() {
	mp := getg().m

	if mp.locks != 0 {
		throw("schedule: holding locks")
	}

	if mp.lockedg != 0 {
		stoplockedm()
		execute(mp.lockedg.ptr(), false) // Never returns.
	}

top:
	pp := mp.p.ptr()
	pp.preempt = false

	if mp.spinning && (pp.runnext != 0 || pp.runqhead != pp.runqtail) {
		throw("schedule: spinning with local work")
	}

	gp, inheritTime, tryWakeP := findRunnable() // blocks until work is available

(a) Why does schedule() panic if mp.locks != 0? (b) What does the mp.lockedg check accomplish? (c) Why is it an error to be spinning with local work?

Question 2 (Short Answer)

The findRunnable() function checks for work in a specific order. List at least five sources of work it checks, and explain why the ordering matters.

Question 3 (True/False with Explanation)

True or False: The global run queue is checked on every call to schedule().

if pp.schedtick%61 == 0 && sched.runqsize > 0 {
    ...
}

Question 4 (What Would Happen If…)

The p struct has a runnext field described as:

// runnext, if non-nil, is a runnable G that was ready'd by
// the current G and should be run next instead of what's in
// runq if there's time remaining in the running G's time
// slice. It will inherit the time left in the current time
// slice.

What would happen to the performance of a producer-consumer pattern (one goroutine sending values on a channel, another receiving) if runnext were removed?

Question 5 (Short Answer)

Explain the role of sysmon in the scheduler. What specific problems does it solve that the normal scheduling loop cannot?

Question 6 (Code Reading)

Consider the execute() function’s signature and a key operation it performs:

func execute(gp *g, inheritTime bool) {
	mp := getg().m
	...
	mp.curg = gp
	gp.m = mp
	casgstatus(gp, _Grunnable, _Grunning)
	gp.waitsince = 0
	gp.preempt = false
	gp.stackguard0 = gp.stack.lo + stackGuard
	...
	gogo(&gp.sched)
}

What is the purpose of the inheritTime parameter? Why does the function reset stackguard0 to gp.stack.lo + stackGuard?

Question 7 (Short Answer)

Why does the Go scheduler use a fixed-size local run queue (256 entries in p.runq) rather than a dynamically-sized slice?

Module 5: Work Stealing and Preemption

Background: Load Balancing and Preemption in Parallel Systems

A parallel system with P processors and a collection of tasks faces a fundamental scheduling question: which processor should run which task? The simplest answer – static partitioning, where tasks are divided evenly among processors at the start – works only when you know the tasks’ durations in advance and they are roughly equal. In practice, tasks vary wildly in cost. A web server might handle one request in 50 microseconds and the next in 500 milliseconds. A parallel divide-and-conquer algorithm generates subtasks whose sizes depend on the input data. Static partitioning leaves some processors idle while others are overwhelmed, and the overall completion time is dictated by the slowest processor. The problem gets worse as processor counts grow: on a 64-core machine, even modest imbalance means most of the hardware sits idle.

Work stealing is the dominant solution to this load-balancing problem, and its theoretical foundations were laid by Robert Blumofe and Charles Leiserson in their 1999 paper “Scheduling Multithreaded Computations by Work Stealing.” The key idea is elegant: each processor maintains its own local deque (double-ended queue) of tasks. When a processor finishes its current work, it first checks its own deque. If it is empty, the processor becomes a thief and picks a random victim processor to steal from, taking tasks from the opposite end of the victim’s deque. Blumofe and Leiserson proved that for a computation with total work T1 and critical path length T-infinity, a work-stealing scheduler completes execution in expected time T1/P + O(T-infinity), which is provably close to optimal. Their Cilk system at MIT demonstrated these ideas in practice and showed that stealing from the “old” end of the deque (the larger, higher-level tasks) minimizes both the number of steals and the communication overhead. The “steal half” variant, where the thief takes half the victim’s queue, further reduces the number of steal operations needed to balance load, converging in O(log n) steals.

Work stealing has become ubiquitous in systems software. Java’s ForkJoinPool, introduced in Java 7 (2011), brought work stealing to mainstream concurrent programming, implementing the fork/join paradigm where tasks recursively split into subtasks that are distributed across worker threads with per-thread deques. Intel’s Threading Building Blocks (TBB) and Cilk Plus used the same approach for C++ parallelism. In the Rust ecosystem, the Tokio async runtime uses a work-stealing scheduler where each worker thread has a local queue of async tasks, and idle workers steal from busy ones – a design that provides good latency and throughput for I/O-heavy server workloads. Even the Linux kernel’s Completely Fair Scheduler (CFS) uses a form of work stealing for its multi-core load balancer: each CPU has its own run queue, and the kernel periodically rebalances by pulling tasks from the busiest CPUs to the least busy ones, organized through a hierarchy of scheduling domains that reflect the machine’s NUMA topology. The pattern is the same everywhere – distribute state for fast-path performance, steal for balance.

Work stealing solves the problem of where to run tasks, but a complementary problem remains: how long should a task run before the scheduler reclaims the processor? This is the problem of preemption. Early operating systems like classic Mac OS and Windows 3.1 used cooperative scheduling, where programs voluntarily yielded the processor. This was simple but fragile – a single misbehaving or compute-intensive program could freeze the entire system. Modern operating systems use preemptive scheduling, typically driven by hardware timer interrupts. Linux, for example, offers multiple preemption levels through its CONFIG_PREEMPT settings: from no preemption (only at explicit schedule points, suitable for throughput-oriented servers) to full kernel preemption (suitable for latency-sensitive desktop and real-time workloads). The tension between cooperative and preemptive scheduling is not just historical – it reappears in every user-space runtime that multiplexes its own tasks onto OS threads, including Go, Erlang, and async runtimes in Rust and JavaScript.

This module examines how Go’s runtime addresses both problems. Go’s scheduler implements work stealing across per-P local run queues, with a lock-free steal-half algorithm directly inspired by the Blumofe-Leiserson framework. For preemption, Go takes a layered approach: cooperative preemption via stack-growth checks at function prologues (present since Go 1.2), and asynchronous signal-based preemption via SIGURG (added in Go 1.14 to handle tight loops without function calls). Overseeing everything is the sysmon thread – a watchdog goroutine that runs without a P and enforces 10ms time slices, reclaims processors from goroutines stuck in system calls, and polls the network. Together, these mechanisms form a scheduler that is both highly concurrent and fair, even in the face of uncooperative workloads.

Learning Objectives

By the end of this module, students will understand:

• Why distributed scheduling outperforms centralized run queues
• How Go’s work-stealing scheduler balances load across processors
• The spinning thread concept and its role in avoiding unnecessary thread wake-ups
• How the global run queue provides fairness and overflow handling
• The dual preemption strategy: cooperative (stack checks) and asynchronous (signals)
• The sysmon background thread and its role in enforcing preemption and syscall hand-off

1. Why Distributed Scheduling? (10 min)

The Problem with Centralized Run Queues

Traditional OS schedulers and early Go (before Go 1.1) used a single global run queue protected by a mutex. Every time a goroutine became runnable or a thread needed work, it had to acquire this lock. On a 64-core machine, this single lock becomes a devastating bottleneck.

The Go scheduler explicitly rejected centralization. From the top-of-file comment in proc.go:

// [src/runtime/proc.go, lines 36-56](https://cs.opensource.google/go/go/+/refs/tags/go1.26.1:src/runtime/proc.go;l=36)
// Worker thread parking/unparking.
// We need to balance between keeping enough running worker threads to utilize
// available hardware parallelism and parking excessive running worker threads
// to conserve CPU resources and power. This is not simple for two reasons:
// (1) scheduler state is intentionally distributed (in particular, per-P work
// queues), so it is not possible to compute global predicates on fast paths;
// (2) for optimal thread management we would need to know the future (don't park
// a worker thread when a new goroutine will be readied in near future).
//
// Three rejected approaches that would work badly:
// 1. Centralize all scheduler state (would inhibit scalability).
// 2. Direct goroutine handoff. That is, when we ready a new goroutine and there
//    is a spare P, unpark a thread and handoff it the thread and the goroutine.
//    This would lead to thread state thrashing, as the thread that readied the
//    goroutine can be out of work the very next moment, we will need to park it.
//    Also, it would destroy locality of computation as we want to preserve
//    dependent goroutines on the same thread; and introduce additional latency.
// 3. Unpark an additional thread whenever we ready a goroutine and there is an
//    idle P, but don't do handoff. This would lead to excessive thread parking/
//    unparking as the additional threads will instantly park without discovering
//    any work to do.

Key insight: The scheduler designers rejected three seemingly reasonable approaches. Centralization kills scalability. Direct handoff causes thrashing. Eager unparking wastes energy. The solution is work stealing with spinning threads.

The GMP Model Recap

Recall from Module 4:

• G (goroutine): the unit of work
• M (machine): an OS thread
• P (processor): a scheduling context with a local run queue

Each P has a local run queue (a 256-element ring buffer). Work is distributed across these queues. When a P runs out of work, it steals from others.

2. The Work Stealing Algorithm (15 min)

stealWork: The Top-Level Stealer

When findRunnable() can’t find work locally, it calls stealWork(). This function makes up to 4 passes over all Ps in a random order, trying to steal goroutines:

From src/runtime/proc.go, lines 3828-3895:

// src/runtime/proc.go, lines 3828-3895
func stealWork(now int64) (gp *g, inheritTime bool, rnow, pollUntil int64, newWork bool) {
    pp := getg().m.p.ptr()

    ranTimer := false

    const stealTries = 4
    for i := 0; i < stealTries; i++ {
        stealTimersOrRunNextG := i == stealTries-1

        for enum := stealOrder.start(cheaprand()); !enum.done(); enum.next() {
            if sched.gcwaiting.Load() {
                // GC work may be available.
                return nil, false, now, pollUntil, true
            }
            p2 := allp[enum.position()]
            if pp == p2 {
                continue
            }

            // ...timer stealing on last pass...

            // Don't bother to attempt to steal if p2 is idle.
            if !idlepMask.read(enum.position()) {
                if gp := runqsteal(pp, p2, stealTimersOrRunNextG); gp != nil {
                    return gp, false, now, pollUntil, ranTimer
                }
            }
        }
    }

    // No goroutines found to steal.
    return nil, false, now, pollUntil, ranTimer
}

Design decisions to note:

1. Random start order (stealOrder.start(cheaprand())): Prevents all idle Ps from hammering the same victim. The enumeration visits all Ps in a pseudo-random permutation.

2. 4 attempts (stealTries = 4): Stealing is racy – a victim’s queue might be momentarily empty even if it’s about to receive work. Multiple passes increase the chance of finding work.

3. Timer stealing only on the last pass (stealTimersOrRunNextG): Timers are expensive to steal (requires locking another P’s timer heap). The algorithm tries cheaper steals first.

4. Skip idle Ps (!idlepMask.read(enum.position())): No point stealing from Ps that have no work either.

runqsteal and runqgrab: Steal Half

The actual stealing is a two-step process. runqsteal calls runqgrab to grab half the victim’s goroutines, then returns one to run immediately:

From src/runtime/proc.go, lines 7727-7747:

// src/runtime/proc.go, lines 7727-7747
// Steal half of elements from local runnable queue of p2
// and put onto local runnable queue of p.
// Returns one of the stolen elements (or nil if failed).
func runqsteal(pp, p2 *p, stealRunNextG bool) *g {
    t := pp.runqtail
    n := runqgrab(p2, &pp.runq, t, stealRunNextG)
    if n == 0 {
        return nil
    }
    n--
    gp := pp.runq[(t+n)%uint32(len(pp.runq))].ptr()
    if n == 0 {
        return gp
    }
    h := atomic.LoadAcq(&pp.runqhead) // load-acquire, synchronize with consumers
    if t-h+n >= uint32(len(pp.runq)) {
        throw("runqsteal: runq overflow")
    }
    atomic.StoreRel(&pp.runqtail, t+n) // store-release, makes the item available for consumption
    return gp
}

The runqgrab function is the lock-free core. It steals n - n/2 goroutines (i.e., half, rounding up for the thief):

From src/runtime/proc.go, lines 7662-7725:

// src/runtime/proc.go, lines 7662-7725
func runqgrab(pp *p, batch *[256]guintptr, batchHead uint32, stealRunNextG bool) uint32 {
    for {
        h := atomic.LoadAcq(&pp.runqhead) // load-acquire, synchronize with other consumers
        t := atomic.LoadAcq(&pp.runqtail) // load-acquire, synchronize with the producer
        n := t - h
        n = n - n/2
        if n == 0 {
            if stealRunNextG {
                // Try to steal from pp.runnext.
                if next := pp.runnext; next != 0 {
                    // ...carefully steal runnext with back-off...
                    if !pp.runnext.cas(next, 0) {
                        continue
                    }
                    batch[batchHead%uint32(len(batch))] = next
                    return 1
                }
            }
            return 0
        }
        if n > uint32(len(pp.runq)/2) { // read inconsistent h and t
            continue
        }
        for i := uint32(0); i < n; i++ {
            g := pp.runq[(h+i)%uint32(len(pp.runq))]
            batch[(batchHead+i)%uint32(len(batch))] = g
        }
        if atomic.CasRel(&pp.runqhead, h, h+n) { // cas-release, commits consume
            return n
        }
    }
}

Why steal half? This is the classic work-stealing result from Blumofe and Leiserson (1999). If you steal only one goroutine, you might need to steal again immediately. If you steal all, you over-correct. Stealing half balances the queues in O(log n) steals.

Lock-free protocol: Note there are no mutex locks. The algorithm uses:

• atomic.LoadAcq for reading head/tail (load-acquire semantics)
• atomic.CasRel for committing the steal (compare-and-swap with release semantics)
• Retry loop if the CAS fails (another thread stole concurrently)

The runnext Slot

Each P has a special runnext field – a single goroutine slot for the next G to run. When a goroutine wakes another goroutine (e.g., via channel send), the awakened G goes into runnext rather than the run queue. This preserves locality: the producer and consumer likely share cache lines.

Stealing from runnext is the last resort (only on the final steal attempt). The code even includes a usleep(3) back-off to avoid thrashing when a goroutine has just placed something in runnext and is about to use it (lines 7694-7701).

3. Spinning Threads (10 min)

The Concept

A “spinning” thread is an M that has no work but is actively looking for some. It has not yet gone to sleep. Spinning threads are a controlled form of busy-waiting that trades CPU cycles for reduced latency.

From src/runtime/proc.go, lines 68-83:

// src/runtime/proc.go, lines 68-83
// A worker thread is considered spinning if it is out of local work and did
// not find work in the global run queue or netpoller; the spinning state is
// denoted in m.spinning and in sched.nmspinning. Threads unparked this way are
// also considered spinning; we don't do goroutine handoff so such threads are
// out of work initially. Spinning threads spin on looking for work in per-P
// run queues and timer heaps or from the GC before parking. If a spinning
// thread finds work it takes itself out of the spinning state and proceeds to
// execution. If it does not find work it takes itself out of the spinning
// state and then parks.
//
// If there is at least one spinning thread (sched.nmspinning>1), we don't
// unpark new threads when submitting work. To compensate for that, if the last
// spinning thread finds work and stops spinning, it must unpark a new spinning
// thread. This approach smooths out unjustified spikes of thread unparking,
// but at the same time guarantees eventual maximal CPU parallelism
// utilization.

The invariant: If there is at least one spinning thread, we don’t wake new threads when work arrives. This prevents thundering herd – imagine 1000 goroutines becoming runnable at once; without this, we’d wake 1000 threads.

Note: The Go source comment says nmspinning>1 (more than one), but the text says “at least one” and the actual code in wakep() checks nmspinning == 0 (i.e., wake only if there are zero spinning threads). The >1 in the comment appears to be a long-standing typo; the intended meaning is >0.

The Wake-up Policy: wakep()

The wake-up decision (in wakep()) is:

1. There is an idle P available, AND
2. There are no spinning worker threads (sched.nmspinning == 0)

If both conditions hold, wake one thread. That thread becomes spinning, searches for work, and if it finds some, it calls resetspinning() which calls wakep() again – potentially waking another thread. This creates a cascade that ramps up parallelism smoothly.

From src/runtime/proc.go, lines 4021-4035:

// src/runtime/proc.go, lines 4021-4035
func resetspinning() {
    gp := getg()
    if !gp.m.spinning {
        throw("resetspinning: not a spinning m")
    }
    gp.m.spinning = false
    nmspinning := sched.nmspinning.Add(-1)
    if nmspinning < 0 {
        throw("findRunnable: negative nmspinning")
    }
    // M wakeup policy is deliberately somewhat conservative, so check if we
    // need to wakeup another P here. See "Worker thread parking/unparking"
    // comment at the top of the file for details.
    wakep()
}

The Delicate Dance: Spinning to Non-Spinning Transition

The transition from spinning to non-spinning is the trickiest part of the scheduler. There is a race between a thread submitting new work and a spinning thread deciding to park:

From src/runtime/proc.go, lines 3622-3638:

// src/runtime/proc.go, lines 3622-3638
// Delicate dance: thread transitions from spinning to non-spinning
// state, potentially concurrently with submission of new work. We must
// drop nmspinning first and then check all sources again (with
// #StoreLoad memory barrier in between). If we do it the other way
// around, another thread can submit work after we've checked all
// sources but before we drop nmspinning; as a result nobody will
// unpark a thread to run the work.
//
// This applies to the following sources of work:
//
// * Goroutines added to the global or a per-P run queue.
// * New/modified-earlier timers on a per-P timer heap.
// * Idle-priority GC work (barring golang.org/issue/19112).
//
// If we discover new work below, we need to restore m.spinning as a
// signal for resetspinning to unpark a new worker thread (because
// there can be more than one starving goroutine).

A memory barrier (or fence) is a CPU instruction that constrains the order in which memory operations become visible to other cores. Without barriers, the CPU and compiler may reorder reads and writes for performance. A StoreLoad barrier ensures that all stores before the barrier are visible to other cores before any loads after the barrier execute.

The pattern for work submission:

1. Submit work to queue
2. StoreLoad memory barrier
3. Check sched.nmspinning

The pattern for spinning->non-spinning:

1. Decrement nmspinning
2. StoreLoad memory barrier
3. Re-check all work sources

This is a classic example of the “flag + data” synchronization pattern. The memory barriers ensure that if both sides race, at least one of them will see the other’s update.

Discussion question: What happens if we reverse the order – check for work first, then decrement nmspinning? (Answer: A submitter could add work, see nmspinning > 0, and not wake a thread. Then the spinner decrements nmspinning and parks. The work sits unprocessed.)

4. The Global Run Queue (5 min)

Overflow: runqputslow

When a P’s local run queue is full (256 entries), the next runqput call triggers runqputslow, which moves half the local queue to the global queue:

From src/runtime/proc.go, lines 7524-7560:

// src/runtime/proc.go, lines 7524-7560
func runqputslow(pp *p, gp *g, h, t uint32) bool {
    var batch [len(pp.runq)/2 + 1]*g

    // First, grab a batch from local queue.
    n := t - h
    n = n / 2
    if n != uint32(len(pp.runq)/2) {
        throw("runqputslow: queue is not full")
    }
    for i := uint32(0); i < n; i++ {
        batch[i] = pp.runq[(h+i)%uint32(len(pp.runq))].ptr()
    }
    if !atomic.CasRel(&pp.runqhead, h, h+n) { // cas-release, commits consume
        return false
    }
    batch[n] = gp

    // ...optional randomization...

    // Link the goroutines.
    for i := uint32(0); i < n; i++ {
        batch[i].schedlink.set(batch[i+1])
    }

    q := gQueue{batch[0].guintptr(), batch[n].guintptr(), int32(n + 1)}

    // Now put the batch on global queue.
    lock(&sched.lock)
    globrunqputbatch(&q)
    unlock(&sched.lock)
    return true
}

The global queue requires sched.lock, which is why it’s avoided on the fast path.

Fairness: The schedtick % 61 Check

Without special handling, two goroutines on a local queue could starve goroutines on the global queue by constantly respawning each other. The scheduler prevents this:

From src/runtime/proc.go, lines 3440-3450:

// src/runtime/proc.go, lines 3440-3450
    // Check the global runnable queue once in a while to ensure fairness.
    // Otherwise two goroutines can completely occupy the local runqueue
    // by constantly respawning each other.
    if pp.schedtick%61 == 0 && !sched.runq.empty() {
        lock(&sched.lock)
        gp := globrunqget()
        unlock(&sched.lock)
        if gp != nil {
            return gp, false, false
        }
    }

Why 61? It’s a prime number. Using a prime avoids resonance with any regular patterns in the program. If we used 64, and a program happened to schedule exactly 64 goroutines between global queue checks, we’d check at the same point every time. A prime ensures the check point moves through the schedule cycle.

5. Cooperative Preemption (5 min)

Function Prologue Stack Checks

Every non-leaf Go function begins with a prologue that checks if the stack needs to grow. The compiler inserts code equivalent to:

if SP < g.stackguard0 {
    call runtime.morestack
}

The scheduler exploits this for cooperative preemption. To preempt a goroutine, it sets gp.stackguard0 = stackPreempt (a value so large that any stack check will fail). The goroutine then calls morestack, which detects the preemption request and yields:

From src/runtime/proc.go, lines 6880-6886:

// src/runtime/proc.go, lines 6880-6886
    gp.preempt = true

    // Every call in a goroutine checks for stack overflow by
    // comparing the current stack pointer to gp->stackguard0.
    // Setting gp->stackguard0 to StackPreempt folds
    // preemption into the normal stack overflow check.
    gp.stackguard0 = stackPreempt

From preempt.go, lines 27-33:

// Synchronous safe-points are implemented by overloading the stack
// bound check in function prologues. To preempt a goroutine at the
// next synchronous safe-point, the runtime poisons the goroutine's
// stack bound to a value that will cause the next stack bound check
// to fail and enter the stack growth implementation, which will
// detect that it was actually a preemption and redirect to preemption
// handling.

Limitation: A goroutine in a tight loop with no function calls (e.g., for { i++ }) will never check its stack and never be preempted cooperatively. This was a real problem before Go 1.14.

6. Asynchronous (Signal-Based) Preemption (10 min)

The SIGURG Choice

Go 1.14 introduced true preemptive scheduling via OS signals. The runtime sends a signal to the thread running a goroutine that needs preemption. But which signal?

From src/runtime/signal_unix.go, lines 44-74:

// src/runtime/signal_unix.go, lines 44-74
// sigPreempt is the signal used for non-cooperative preemption.
//
// There's no good way to choose this signal, but there are some
// heuristics:
//
// 1. It should be a signal that's passed-through by debuggers by
// default. On Linux, this is SIGALRM, SIGURG, SIGCHLD, SIGIO,
// SIGVTALRM, SIGPROF, and SIGWINCH, plus some glibc-internal signals.
//
// 2. It shouldn't be used internally by libc in mixed Go/C binaries
// because libc may assume it's the only thing that can handle these
// signals. For example SIGCANCEL or SIGSETXID.
//
// 3. It should be a signal that can happen spuriously without
// consequences. For example, SIGALRM is a bad choice because the
// signal handler can't tell if it was caused by the real process
// alarm or not (arguably this means the signal is broken, but I
// digress). SIGUSR1 and SIGUSR2 are also bad because those are often
// used in meaningful ways by applications.
//
// 4. We need to deal with platforms without real-time signals (like
// macOS), so those are out.
//
// We use SIGURG because it meets all of these criteria, is extremely
// unlikely to be used by an application for its "real" meaning (both
// because out-of-band data is basically unused and because SIGURG
// doesn't report which socket has the condition, making it pretty
// useless), and even if it is, the application has to be ready for
// spurious SIGURG.
const sigPreempt = _SIGURG

This is a great example of systems engineering trade-offs. The signal must be:

• Debugger-transparent (so GDB doesn’t stop on it)
• Not used by libc (for CGo compatibility)
• Safe to receive spuriously (the handler must be idempotent)
• Available on all platforms (no real-time signals)

How Async Preemption Works

From preempt.go, lines 35-43:

// Preemption at asynchronous safe-points is implemented by suspending
// the thread using an OS mechanism (e.g., signals) and inspecting its
// state to determine if the goroutine was at an asynchronous
// safe-point. Since the thread suspension itself is generally
// asynchronous, it also checks if the running goroutine wants to be
// preempted, since this could have changed. If all conditions are
// satisfied, it adjusts the signal context to make it look like the
// signaled thread just called asyncPreempt and resumes the thread.
// asyncPreempt spills all registers and enters the scheduler.

The flow is:

1. sysmon or GC decides goroutine G needs preemption
2. Calls preemptone(pp), which calls preemptM(mp)
3. preemptM sends SIGURG to the OS thread running G
4. Signal handler fires, checks if G is at an async safe-point
5. If safe, rewrites the signal return context to call asyncPreempt
6. asyncPreempt saves all registers and calls schedule()

Async safe-points: Not every instruction is safe for preemption. The runtime needs to be able to find all GC roots (stack pointers). The compiler emits metadata marking which instructions are safe.

preemptone: The Combined Approach

From src/runtime/proc.go, lines 6866-6895:

// src/runtime/proc.go, lines 6866-6895
func preemptone(pp *p) bool {
    mp := pp.m.ptr()
    if mp == nil || mp == getg().m {
        return false
    }
    gp := mp.curg
    if gp == nil || gp == mp.g0 {
        return false
    }

    gp.preempt = true

    // Every call in a goroutine checks for stack overflow by
    // comparing the current stack pointer to gp->stackguard0.
    // Setting gp->stackguard0 to StackPreempt folds
    // preemption into the normal stack overflow check.
    gp.stackguard0 = stackPreempt

    // Request an async preemption of this P.
    if preemptMSupported && debug.asyncpreemptoff == 0 {
        pp.preempt = true
        preemptM(mp)
    }

    return true
}

Notice the belt-and-suspenders approach: it sets BOTH the cooperative preemption flag (stackguard0 = stackPreempt) AND sends the async signal (preemptM(mp)). Whichever fires first wins.

7. The sysmon Thread (10 min)

What is sysmon?

sysmon is a special background thread that runs without a P. It’s the scheduler’s watchdog, responsible for:

• Preempting long-running goroutines
• Retaking Ps from goroutines stuck in syscalls
• Polling the network (if no other thread is doing it)
• Triggering garbage collection

From src/runtime/proc.go, lines 6486-6502:

// src/runtime/proc.go, lines 6486-6502
func sysmon() {
    lock(&sched.lock)
    sched.nmsys++
    checkdead()
    unlock(&sched.lock)

    lastgomaxprocs := int64(0)
    lasttrace := int64(0)
    idle := 0 // how many cycles in succession we had not wokeup somebody
    delay := uint32(0)

    for {
        if idle == 0 { // start with 20us sleep...
            delay = 20
        } else if idle > 50 { // start doubling the sleep after 1ms...
            delay *= 2
        }
        if delay > 10*1000 { // up to 10ms
            delay = 10 * 1000
        }
        usleep(delay)
        // ...
    }
}

Adaptive sleep: sysmon starts by polling every 20 microseconds. If it finds nothing to do for 50 iterations (~1ms), it starts doubling its sleep interval, up to a maximum of 10ms. When it finds work again, it resets. This balances responsiveness against CPU overhead.

retake: Preempting and Reclaiming Ps

The retake function is called by sysmon to enforce time slices and reclaim Ps from syscalls:

From src/runtime/proc.go, lines 6626-6628:

// src/runtime/proc.go, lines 6626-6628
// forcePreemptNS is the time slice given to a G before it is
// preempted.
const forcePreemptNS = 10 * 1000 * 1000 // 10ms

From src/runtime/proc.go, lines 6656-6671:

// src/runtime/proc.go, lines 6656-6671
        // Preempt G if it's running on the same schedtick for
        // too long. This could be from a single long-running
        // goroutine or a sequence of goroutines run via
        // runnext, which share a single schedtick time slice.
        schedt := int64(pp.schedtick)
        if int64(pd.schedtick) != schedt {
            pd.schedtick = uint32(schedt)
            pd.schedwhen = now
        } else if pd.schedwhen+forcePreemptNS <= now {
            preemptone(pp)
            // If pp is in a syscall, preemptone doesn't work.
            sysretake = true
        }

The logic is:

1. If pp.schedtick changed since last check, the P is making progress – update the timestamp
2. If pp.schedtick hasn’t changed for 10ms, the same goroutine (or runnext chain) has been running too long – preempt it

Syscall Hand-off

When a goroutine enters a syscall, its M is blocked in the kernel. But the P is sitting idle. The retake function can steal this P and give it to another M:

From src/runtime/proc.go, lines 6700-6705:

// src/runtime/proc.go, lines 6700-6705
        // On the one hand we don't want to retake Ps if there is no other work to do,
        // but on the other hand we want to retake them eventually
        // because they can prevent the sysmon thread from deep sleep.
        if runqempty(pp) && sched.nmspinning.Load()+sched.npidle.Load() > 0 && pd.syscallwhen+10*1000*1000 > now {
            thread.resume()
            goto done

The P is retaken from a syscalling M unless ALL of these conditions hold:

• The P’s run queue is empty (no work waiting)
• There are spinning threads or idle Ps available (system is not starved)
• The syscall started less than 10ms ago

When the M returns from the syscall, it finds its P gone. It must acquire a new P to continue running Go code, or park itself and put the goroutine on the global queue.

The schedule() Function: Putting It All Together

The main scheduler loop ties everything together:

From src/runtime/proc.go, lines 4135-4231:

// src/runtime/proc.go, lines 4135-4231
func schedule() {
    mp := getg().m

    if mp.locks != 0 {
        throw("schedule: holding locks")
    }

    if mp.lockedg != 0 {
        stoplockedm()
        execute(mp.lockedg.ptr(), false) // Never returns.
    }

    // ...

top:
    pp := mp.p.ptr()
    pp.preempt = false

    // Safety check: if we are spinning, the run queue should be empty.
    if mp.spinning && (pp.runnext != 0 || pp.runqhead != pp.runqtail) {
        throw("schedule: spinning with local work")
    }

    gp, inheritTime, tryWakeP := findRunnable() // blocks until work is available

    // ...

    // This thread is going to run a goroutine and is not spinning anymore,
    // so if it was marked as spinning we need to reset it now and potentially
    // start a new spinning M.
    if mp.spinning {
        resetspinning()
    }

    // ...
}

The search order in findRunnable() is:

1. Check GC work
2. Every 61st tick, check the global run queue (fairness)
3. Check the local run queue
4. Check the global run queue (again, without the tick restriction)
5. Poll network
6. Steal from other Ps via stealWork()
7. If nothing found, park the thread

8. Summary: The Complete Picture

                    ┌─────────────────────────────────────┐
                    │           Global Run Queue           │
                    │  (overflow + fairness, locked)        │
                    └──────────┬────────────┬──────────────┘
                               │            │
                    schedtick%61        runqputslow
                               │            │
         ┌─────────────────────┼────────────┼─────────────────────┐
         │                     │            │                     │
    ┌────▼────┐          ┌─────▼────┐  ┌────▼─────┐         ┌─────────┐
    │  P0     │◄─steal───│  P1     │  │  P2      │──steal──►│  P3     │
    │ runq    │          │ runq    │  │ runq     │          │ runq    │
    │ runnext │          │ runnext │  │ runnext  │          │ runnext │
    └────┬────┘          └────┬────┘  └────┬─────┘         └────┬────┘
         │                    │            │                    │
    ┌────▼────┐          ┌────▼────┐  ┌────▼─────┐         ┌────▼────┐
    │   M0    │          │   M1    │  │   M2     │         │   M3    │
    │ (thread)│          │ (thread)│  │ (thread) │         │ (thread)│
    └─────────┘          └─────────┘  └──────────┘         └─────────┘

                    sysmon thread (no P):
                    - preempts goroutines after 10ms
                    - retakes Ps from syscalls
                    - polls network

Key Takeaways

1. Distributed state is fast but complex: Per-P queues avoid lock contention but require careful synchronization with atomics and memory barriers.

2. Work stealing balances load: Steal half, random victim selection, multiple attempts.

3. Spinning threads prevent thundering herd: At most one spinning thread is sufficient to discover new work and cascade wake-ups.

4. Two-tier preemption: Cooperative (stack checks) handles the common case; async signals (SIGURG) handle tight loops.

5. sysmon is the safety net: It runs independently, enforcing time slices and reclaiming resources from blocked syscalls.

Discussion Questions

1. Why does the scheduler steal half the victim’s goroutines instead of just one? What would change if it stole all of them?

2. The spinning thread protocol uses sched.nmspinning as a shared counter. Why not use per-P spinning flags instead?

3. Consider a program that makes many blocking syscalls (e.g., a web server doing file I/O). How does the P hand-off mechanism keep the program responsive?

4. Why is 61 a better choice than 64 for the global queue fairness check interval?

5. What would happen in a pre-1.14 Go runtime if a goroutine entered for { } (an empty infinite loop)?

Further Reading

• Scalable Go Scheduler Design Doc – Dmitry Vyukov’s original proposal
• Blumofe, R. and Leiserson, C. “Scheduling Multithreaded Computations by Work Stealing” (1999)
• Go 1.14 Release Notes – Asynchronous preemption
• proposal: Non-cooperative goroutine preemption

Module 5: Work Stealing and Preemption

Comprehension Check

Question 1 (Code Reading)

Study the stealWork function from runtime/proc.go:

func stealWork(now int64) (gp *g, inheritTime bool, rnow, pollUntil int64, newWork bool) {
	pp := getg().m.p.ptr()

	ranTimer := false

	const stealTries = 4
	for i := 0; i < stealTries; i++ {
		stealTimersOrRunNextG := i == stealTries-1

		for enum := stealOrder.start(cheaprand()); !enum.done(); enum.next() {
			if sched.gcwaiting.Load() {
				return nil, false, now, pollUntil, true
			}
			p2 := allp[enum.position()]
			if pp == p2 {
				continue
			}
			...

(a) Why does the function try to steal 4 times rather than just once? (b) Why is the iteration order randomized with cheaprand()? (c) Why does it only steal timers and runnext on the last iteration (stealTimersOrRunNextG)?

Question 2 (Short Answer)

Explain the “spinning thread” mechanism. What is a spinning thread, and why does the scheduler limit the number of spinning threads?

Question 3 (True/False with Explanation)

True or False: The Go scheduler’s cooperative preemption (checking stackguard0 at function prologues) guarantees that a goroutine cannot monopolize a CPU for more than 10ms.

Question 4 (What Would Happen If…)

What would happen if work stealing always stole the entire run queue of the victim P instead of stealing half?

Question 5 (Code Reading)

Consider the runqsteal function:

func runqsteal(pp, p2 *p, stealRunNextG bool) *g {
	t := pp.runqtail
	n := runqgrab(p2, &pp.runq, t, stealRunNextG)
	if n == 0 {
		return nil
	}
	n--
	gp := pp.runq[(t+n)%uint32(len(pp.runq))].ptr()
	if n == 0 {
		return gp
	}
	h := atomic.LoadAcq(&pp.runqhead)
	if t-h+n >= uint32(len(pp.runq)) {
		throw("runqsteal: runq overflow")
	}
	atomic.StoreRel(&pp.runqtail, t+n)
	return gp
}

Why does this function use atomic.LoadAcq and atomic.StoreRel? What memory ordering guarantee do these provide that a plain load/store would not?

Question 6 (Short Answer)

Explain how SIGURG-based asynchronous preemption works in Go. Why was SIGURG chosen as the signal?

Question 7 (Short Answer)

What is the retake() function’s role, and what are its two main responsibilities?

Module 6: Synchronization Primitives

Background: The Evolution of Synchronization

The fundamental challenge of concurrent programming is deceptively simple to state: when multiple threads of execution share mutable state, their interleaved operations can produce results that no sequential execution would. A counter incremented by two threads can lose updates; a linked list modified concurrently can develop cycles or dangling pointers; a partially-written data structure can be observed in a state that its author never intended. This problem is as old as multiprogramming itself, and the history of systems software is in many ways the history of increasingly sophisticated answers to it.

The formal study of synchronization begins with Edsger Dijkstra’s 1965 paper introducing the semaphore, a counting primitive with two operations – P (wait) and V (signal) – sufficient to solve mutual exclusion and coordination problems. Dijkstra’s work established the intellectual framework, but practical implementations required hardware support. Early processors offered test-and-set instructions; later architectures introduced compare-and-swap (CAS), first appearing on the IBM System/370 in 1970, which became the workhorse of lock-free programming. Modern hardware provides further refinements: x86 processors implement Total Store Order (TSO), a relatively strong memory model where stores are seen in program order by other cores, while ARM and RISC-V use weaker ordering models that demand explicit barriers or load-acquire/store-release instructions. These architectural differences mean that correct synchronization code is not portable at the instruction level – a lesson that has motivated language-level memory models like C11’s and Go’s.

A pivotal innovation in the Linux ecosystem was the futex (fast userspace mutex), introduced by Hubertus Franke, Matthew Kirkwood, and Rusty Russell in 2002 and merged into Linux 2.5.7. The key insight was that most lock acquisitions are uncontended: if a thread can acquire a lock with a single atomic operation in user space, there is no reason to enter the kernel at all. The futex syscall is invoked only on the slow path – when a thread must sleep because the lock is held, or when it must wake a sleeping waiter. This design eliminated the overhead of a syscall on every lock/unlock cycle, and it remains the foundation of pthread mutexes, condition variables, and many user-space synchronization libraries on Linux to this day. The design also inspired similar approaches on other platforms, though macOS takes a different path using Mach semaphores and pthread primitives.

Real-world synchronization involves tradeoffs that go beyond correctness. Fairness and throughput are often in tension: a lock that always hands ownership to the longest-waiting thread (strict FIFO) prevents starvation but suffers from “lock convoy” effects and poor cache locality, while a lock that lets any thread grab ownership (barging) maximizes throughput at the risk of starving unlucky waiters. The evolution from simple spinlocks to sleeping locks to adaptive mutexes – which spin briefly before sleeping, as seen in Linux’s mutex implementation and Windows’s SRWLOCK – reflects decades of effort to balance these concerns. Priority inversion, famously diagnosed in the 1997 Mars Pathfinder mission when a low-priority thread holding a bus mutex blocked the high-priority watchdog task, led to the widespread adoption of priority inheritance protocols. Reader-writer locks introduce their own design space: Linux’s percpu_rwsem optimizes the common read path by using per-CPU counters, accepting a heavyweight write-side operation in exchange. Lock-free and wait-free data structures, built directly on hardware atomics, eliminate blocking entirely but are notoriously difficult to design correctly and do not always outperform well-tuned locks under real workloads.

Go’s synchronization stack reflects all of these lessons. It is built in layers: hardware atomics at the bottom, OS primitives (futexes on Linux, semaphores on macOS) above them, a runtime mutex for the scheduler’s own internal use, and finally the sema.go semaphore that powers sync.Mutex, sync.RWMutex, and other user-visible primitives. Go’s sync.Mutex even implements an explicit starvation mode inspired by the fairness-vs-throughput tradeoff – it barges under low contention for throughput but switches to direct handoff after a goroutine has waited more than 1ms. This module walks through each layer from the bottom up, showing how these abstractions compose and where the hard-won lessons of synchronization history appear in production code.

Learning Objectives

By the end of this module, students will understand:

• How hardware atomics (CAS, load-acquire, store-release) underpin all synchronization
• The futex abstraction on Linux and its use in the Go runtime
• How macOS uses semaphores instead of futexes for the same purpose
• The runtime mutex implementation (lock_spinbit.go)
• The sema.go semaphore that powers sync.Mutex, including its treap data structure
• How starvation is handled with handoff mode
• The reader-writer mutex and the rwmutexMaxReaders trick

1. The Need for Synchronization (5 min)

Mutual Exclusion

When multiple goroutines (or threads) access shared data, we need to ensure that at most one is modifying the data at a time. Without this, we get data races: torn reads, lost updates, and corrupted state.

Ordering

Beyond mutual exclusion, we often need to ensure that one operation happens before another. A goroutine writing to a channel must complete before the goroutine reading from that channel sees the value. This requires memory ordering guarantees that go beyond simple atomicity.

The Synchronization Stack in Go

Go has a layered synchronization architecture:

User-visible:     sync.Mutex, sync.RWMutex, sync.WaitGroup, channels
                          │
                          ▼
Runtime:          sema.go (semacquire1 / semrelease1)
                          │
                          ▼
Runtime:          runtime mutex (lock_spinbit.go: lock2 / unlock2)
                          │
                          ▼
OS interface:     lock_futex.go (Linux) or lock_sema.go (macOS)
                  notesleep/notewakeup, semasleep/semawakeup
                          │
                          ▼
Hardware:         atomic CAS, load-acquire, store-release

Each layer builds on the one below. We’ll work from the bottom up.

2. Hardware Atomics (5 min)

Compare-and-Swap (CAS)

The fundamental building block. CAS(addr, old, new) atomically:

1. Reads *addr
2. If it equals old, writes new and returns true
3. Otherwise returns false

In Go’s runtime: atomic.Cas(addr, old, new), atomic.Casuintptr(addr, old, new).

Load-Acquire and Store-Release

These provide memory ordering guarantees:

• Load-Acquire (atomic.LoadAcq): No subsequent memory operations can be reordered before this load. Used by consumers to see all writes that happened before the corresponding store-release.

• Store-Release (atomic.StoreRel, atomic.CasRel): No preceding memory operations can be reordered after this store. Used by producers to publish data.

These appear throughout the run queue implementation:

src/runtime/proc.go, lines 7664-7665

// src/runtime/proc.go, lines 7664-7665 (in runqgrab)
h := atomic.LoadAcq(&pp.runqhead) // load-acquire, synchronize with other consumers
t := atomic.LoadAcq(&pp.runqtail) // load-acquire, synchronize with the producer

src/runtime/proc.go, line 7721

// src/runtime/proc.go, line 7721 (in runqgrab)
if atomic.CasRel(&pp.runqhead, h, h+n) { // cas-release, commits consume

Atomic Exchange

atomic.Xchg(addr, new) atomically writes new and returns the old value. Used in the mutex fast path:

src/runtime/lock_spinbit.go, line 165

// src/runtime/lock_spinbit.go, line 165
v8 := atomic.Xchg8(k8, mutexLocked)
if v8&mutexLocked == 0 {
    // We got the lock!
    return
}

3. Futex: The Fundamental Blocking Primitive (Linux) (10 min)

What is a Futex?

A futex (“fast userspace mutex”) is a Linux system call that lets a thread sleep until a memory location changes value. The key insight: the fast path (uncontended lock/unlock) never enters the kernel. Only when contention occurs does the thread make a syscall to sleep.

The Go runtime on Linux (lock_futex.go) uses futexes for two purposes:

1. One-time notifications (notesleep/notewakeup) – used for thread synchronization in the runtime itself
2. Thread parking (semasleep/semawakeup) – used to put Ms to sleep and wake them

notesleep / notewakeup: One-Time Notifications

A note is the simplest synchronization primitive: a one-shot event. One thread sleeps on it; another wakes it. It cannot be reused without calling noteclear.

src/runtime/lock_futex.go, lines 21-33

// src/runtime/lock_futex.go, lines 21-33
// One-time notifications.
func noteclear(n *note) {
    n.key = 0
}

func notewakeup(n *note) {
    old := atomic.Xchg(key32(&n.key), 1)
    if old != 0 {
        print("notewakeup - double wakeup (", old, ")\n")
        throw("notewakeup - double wakeup")
    }
    futexwakeup(key32(&n.key), 1)
}

The protocol:

1. noteclear sets key = 0
2. notewakeup atomically sets key = 1, then calls futexwakeup
3. Double wakeup is a fatal error (this is a one-shot primitive)

The sleeping side:

src/runtime/lock_futex.go, lines 35-53

// src/runtime/lock_futex.go, lines 35-53
func notesleep(n *note) {
    gp := getg()
    if gp != gp.m.g0 {
        throw("notesleep not on g0")
    }
    ns := int64(-1)
    if *cgo_yield != nil {
        // Sleep for an arbitrary-but-moderate interval to poll libc interceptors.
        ns = 10e6
    }
    for atomic.Load(key32(&n.key)) == 0 {
        gp.m.blocked = true
        futexsleep(key32(&n.key), 0, ns)
        if *cgo_yield != nil {
            asmcgocall(*cgo_yield, nil)
        }
        gp.m.blocked = false
    }
}

Key details:

• Must run on g0 (the system stack). This is a low-level primitive for blocking OS threads, not goroutines.
• The futexsleep(key32(&n.key), 0, ns) call says: “sleep if *key == 0”. The kernel checks the value atomically with going to sleep, avoiding the lost-wakeup race.
• The outer for loop handles spurious wakeups.
• With CGo, it polls periodically (every 10ms) for libc interceptors.

semasleep / semawakeup: Thread Parking

These are used by the runtime mutex to park and wake OS threads:

src/runtime/lock_futex.go, lines 138-163

// src/runtime/lock_futex.go, lines 138-163
//go:nosplit
func semasleep(ns int64) int32 {
    mp := getg().m

    for v := atomic.Xadd(&mp.waitsema, -1); ; v = atomic.Load(&mp.waitsema) {
        if int32(v) >= 0 {
            return 0
        }
        futexsleep(&mp.waitsema, v, ns)
        if ns >= 0 {
            if int32(v) >= 0 {
                return 0
            } else {
                return -1
            }
        }
    }
}

//go:nosplit
func semawakeup(mp *m) {
    v := atomic.Xadd(&mp.waitsema, 1)
    if v == 0 {
        futexwakeup(&mp.waitsema, 1)
    }
}

The protocol uses mp.waitsema as a counter:

• semasleep: decrements the counter. If it goes negative, the thread sleeps via futex.
• semawakeup: increments the counter. If it was negative (meaning someone is sleeping), wakes them.

This is a counting semaphore: if semawakeup is called before semasleep, the subsequent semasleep returns immediately.

4. Semaphore-Based Blocking (macOS / Darwin) (5 min)

macOS doesn’t have futexes. Instead, lock_sema.go uses OS-level semaphores (pthread mutexes and condition variables under the hood). The API is the same (notesleep/notewakeup, semasleep/semawakeup), but the implementation differs.

notesleep / notewakeup on macOS

src/runtime/lock_sema.go, lines 23-44

// src/runtime/lock_sema.go, lines 23-44
func notewakeup(n *note) {
    var v uintptr
    for {
        v = atomic.Loaduintptr(&n.key)
        if atomic.Casuintptr(&n.key, v, locked) {
            break
        }
    }

    // Successfully set waitm to locked.
    // What was it before?
    switch {
    case v == 0:
        // Nothing was waiting. Done.
    case v == locked:
        // Two notewakeups! Not allowed.
        throw("notewakeup - double wakeup")
    default:
        // Must be the waiting m. Wake it up.
        semawakeup((*m)(unsafe.Pointer(v)))
    }
}

The note’s key field serves triple duty:

• 0 = not yet signaled, no one waiting
• locked (= 1) = signaled
• Any other value = pointer to the waiting M

src/runtime/lock_sema.go, lines 46-72

// src/runtime/lock_sema.go, lines 46-72
func notesleep(n *note) {
    gp := getg()
    if gp != gp.m.g0 {
        throw("notesleep not on g0")
    }
    semacreate(gp.m)
    if !atomic.Casuintptr(&n.key, 0, uintptr(unsafe.Pointer(gp.m))) {
        // Must be locked (got wakeup).
        if n.key != locked {
            throw("notesleep - waitm out of sync")
        }
        return
    }
    // Queued. Sleep.
    gp.m.blocked = true
    if *cgo_yield == nil {
        semasleep(-1)
    } else {
        const ns = 10e6
        for atomic.Loaduintptr(&n.key) == 0 {
            semasleep(ns)
            asmcgocall(*cgo_yield, nil)
        }
    }
    gp.m.blocked = false
}

The protocol:

1. Sleeper tries to CAS key from 0 to &m (its own M pointer)
2. If CAS fails, the note was already signaled – return immediately
3. If CAS succeeds, call semasleep(-1) to block on the M’s OS semaphore
4. Waker CASes key to locked. If the old value was an M pointer, calls semawakeup(m)

Comparison with futex: The futex version is simpler because the kernel handles the “check-and-sleep” atomicity. On macOS, the runtime must carefully encode the waiting state in the key field and use CAS to avoid races.

5. The Runtime Mutex (10 min)

Architecture: lock_spinbit.go

The runtime mutex (runtime.mutex) is used internally throughout the Go runtime – for the scheduler lock, timer heaps, and more. The current implementation (lock_spinbit.go, introduced in Go 1.24) uses an elegant bit-packing scheme.

src/runtime/lock_spinbit.go, lines 29-45

// src/runtime/lock_spinbit.go, lines 29-45
// The mutex state consists of four flags and a pointer. The flag at bit 0,
// mutexLocked, represents the lock itself. Bit 1, mutexSleeping, is a hint that
// the pointer is non-nil. The fast paths for locking and unlocking the mutex
// are based on atomic 8-bit swap operations on the low byte; bits 2 through 7
// are unused.
//
// Bit 8, mutexSpinning, is a try-lock that grants a waiting M permission to
// spin on the state word. Most other Ms must attempt to spend their time
// sleeping to reduce traffic on the cache line.
//
// Bit 9, mutexStackLocked, is a try-lock that grants an unlocking M permission
// to inspect the list of waiting Ms and to pop an M off of that stack.
//
// The upper bits hold a (partial) pointer to the M that most recently went to
// sleep. The sleeping Ms form a stack linked by their mWaitList.next fields.

The state word layout:

┌──────────────────────────────┬────┬────┬───────┬─────────┬────────┐
│  M pointer (upper bits)      │ 9  │ 8  │ 7..2  │    1    │   0    │
│  (head of waiter stack)      │stkL│spin│unused │sleeping │locked  │
└──────────────────────────────┴────┴────┴───────┴─────────┴────────┘

The Fast Path: 8-bit Swap

src/runtime/lock_spinbit.go, lines 155-171

// src/runtime/lock_spinbit.go, lines 155-171
func lock2(l *mutex) {
    gp := getg()
    if gp.m.locks < 0 {
        throw("runtime·lock: lock count")
    }
    gp.m.locks++

    k8 := key8(&l.key)

    // Speculative grab for lock.
    v8 := atomic.Xchg8(k8, mutexLocked)
    if v8&mutexLocked == 0 {
        if v8&mutexSleeping != 0 {
            atomic.Or8(k8, mutexSleeping)
        }
        return
    }
    semacreate(gp.m)
    // ...slow path...
}

The uncontended case is a single atomic.Xchg8 – an 8-bit atomic swap of just the low byte. This is extremely fast because:

1. It’s a single instruction on modern CPUs
2. It only touches the low byte, leaving the waiter stack pointer intact
3. The 8-bit operation avoids ABA problems with the pointer bits

The Slow Path: Spin then Sleep

src/runtime/lock_spinbit.go, lines 182-257

// src/runtime/lock_spinbit.go, lines 182-257
    var weSpin, atTail, haveTimers bool
    v := atomic.Loaduintptr(&l.key)
tryAcquire:
    for i := 0; ; i++ {
        if v&mutexLocked == 0 {
            // Try to acquire...
        }

        if !weSpin && v&mutexSpinning == 0 && atomic.Casuintptr(&l.key, v, v|mutexSpinning) {
            v |= mutexSpinning
            weSpin = true
        }

        if weSpin || atTail || mutexPreferLowLatency(l) {
            if i < spin {
                procyield(mutexActiveSpinSize)    // ~30 PAUSE instructions
                // ...
            } else if i < spin+mutexPassiveSpinCount {
                osyield()
                // ...
            }
        }

        // Go to sleep
        gp.m.mWaitList.next = mutexWaitListHead(v)
        next := (uintptr(unsafe.Pointer(gp.m)) &^ mutexMMask) | v&mutexMMask | mutexSleeping
        if atomic.Casuintptr(&l.key, v, next) {
            semasleep(-1)
            // ...
        }
    }

The slow path progression:

1. Active spin (procyield): Execute PAUSE instructions. Only for multi-CPU systems, only 4 iterations.
2. Passive spin (osyield): Yield the CPU to other threads. Only 1 iteration.
3. Sleep (semasleep): Push ourselves onto the waiter stack and call the OS to sleep.

The spin bit: Only one thread at a time gets the mutexSpinning bit. This prevents all waiting threads from spinning simultaneously (which would waste CPU). The other waiters go to sleep.

Anti-starvation: Threads at the tail of the waiter stack (atTail = true) get permission to spin, preventing indefinite starvation.

The Waiter Stack

Waiting Ms are linked through their mWaitList.next fields, forming a LIFO stack. The head of the stack is packed into the upper bits of l.key:

src/runtime/lock_spinbit.go, lines 88-91

// src/runtime/lock_spinbit.go, lines 88-91
type mWaitList struct {
    next       muintptr // next m waiting for lock
    startTicks int64    // when this m started waiting
}

When a thread goes to sleep, it CASes its M pointer into the upper bits of l.key, linking to the previous head via mWaitList.next. When unlock2 wakes a thread, it pops from this stack.

6. The sema.go Semaphore (15 min)

Purpose

The sema.go semaphore is the mechanism that sync.Mutex, sync.RWMutex, and sync.WaitGroup use to block and wake goroutines (not OS threads). Unlike the runtime mutex which blocks Ms, this semaphore blocks Gs.

src/runtime/sema.go, lines 5-15

// src/runtime/sema.go, lines 5-15
// Semaphore implementation exposed to Go.
// Intended use is provide a sleep and wakeup
// primitive that can be used in the contended case
// of other synchronization primitives.
// Thus it targets the same goal as Linux's futex,
// but it has much simpler semantics.
//
// That is, don't think of these as semaphores.
// Think of them as a way to implement sleep and wakeup
// such that every sleep is paired with a single wakeup,
// even if, due to races, the wakeup happens before the sleep.

The Hash Table: semTable

To avoid a single global lock, the semaphore system uses a hash table indexed by the address of the semaphore word:

src/runtime/sema.go, lines 48-58

// src/runtime/sema.go, lines 48-58
// Prime to not correlate with any user patterns.
const semTabSize = 251

type semTable [semTabSize]struct {
    root semaRoot
    pad  [cpu.CacheLinePadSize - unsafe.Sizeof(semaRoot{})]byte
}

func (t *semTable) rootFor(addr *uint32) *semaRoot {
    return &t[(uintptr(unsafe.Pointer(addr))>>3)%semTabSize].root
}

• 251 buckets (prime, to avoid correlation with allocation patterns)
• Each bucket padded to a cache line to prevent false sharing
• Address shifted right by 3 before hashing (alignment)

The Treap Data Structure

Each semaRoot contains a treap (tree + heap): a balanced binary search tree where:

• The BST property is on the semaphore address: left subtree has smaller addresses, right subtree has larger
• The heap property is on a random priority (ticket): parents have smaller tickets than children

src/runtime/sema.go, lines 30-44

// src/runtime/sema.go, lines 30-44
// A semaRoot holds a balanced tree of sudog with distinct addresses (s.elem).
// Each of those sudog may in turn point (through s.waitlink) to a list
// of other sudogs waiting on the same address.
// The operations on the inner lists of sudogs with the same address
// are all O(1). The scanning of the top-level semaRoot list is O(log n),
// where n is the number of distinct addresses with goroutines blocked
// on them that hash to the given semaRoot.
type semaRoot struct {
    lock  mutex
    treap *sudog        // root of balanced tree of unique waiters.
    nwait atomic.Uint32 // Number of waiters. Read w/o the lock.
}

The two-level structure:

                treap (BST on address, heap on ticket)
                         ┌───────┐
                         │addr=A │
                         │ticket=3│
                         └───┬───┘
                        /         \
               ┌───────┐           ┌───────┐
               │addr=B │           │addr=C │
               │ticket=7│           │ticket=5│
               └───┬───┘           └───────┘
                   │
              waitlink list (same address B):
              sudog1 -> sudog2 -> sudog3

Within each treap node, goroutines waiting on the same address form a linked list via waitlink/waittail. Operations on this inner list are O(1). Finding the right address in the treap is O(log n).

semacquire1: The Core Acquire

src/runtime/sema.go, lines 146-201

// src/runtime/sema.go, lines 146-201
func semacquire1(addr *uint32, lifo bool, profile semaProfileFlags, skipframes int, reason waitReason) {
    gp := getg()
    if gp != gp.m.curg {
        throw("semacquire not on the G stack")
    }

    // Easy case.
    if cansemacquire(addr) {
        return
    }

    // Harder case:
    //  increment waiter count
    //  try cansemacquire one more time, return if succeeded
    //  enqueue itself as a waiter
    //  sleep
    //  (waiter descriptor is dequeued by signaler)
    s := acquireSudog()
    root := semtable.rootFor(addr)
    // ...
    for {
        lockWithRank(&root.lock, lockRankRoot)
        root.nwait.Add(1)
        // Check cansemacquire to avoid missed wakeup.
        if cansemacquire(addr) {
            root.nwait.Add(-1)
            unlock(&root.lock)
            break
        }
        // Any semrelease after the cansemacquire knows we're waiting
        // (we set nwait above), so go to sleep.
        root.queue(addr, s, lifo)
        goparkunlock(&root.lock, reason, traceBlockSync, 4+skipframes)
        if s.ticket != 0 || cansemacquire(addr) {
            break
        }
    }
    // ...
    releaseSudog(s)
}

The flow:

1. Fast path: Try cansemacquire (atomic decrement if > 0). If it succeeds, return immediately – no locks needed.
2. Slow path: Acquire the semaRoot lock, increment nwait, try once more (to avoid missed wakeup), then enqueue and park via goparkunlock.
3. On wakeup: Check s.ticket (handoff mode) or retry cansemacquire.

cansemacquire: The Atomic Fast Path

src/runtime/sema.go, lines 291-301

// src/runtime/sema.go, lines 291-301
func cansemacquire(addr *uint32) bool {
    for {
        v := atomic.Load(addr)
        if v == 0 {
            return false
        }
        if atomic.Cas(addr, v, v-1) {
            return true
        }
    }
}

A simple CAS loop: atomically decrement the counter if it’s positive. This is the uncontended fast path that avoids all locks.

semrelease1: Release and Wake

src/runtime/sema.go, lines 207-289

// src/runtime/sema.go, lines 207-289
func semrelease1(addr *uint32, handoff bool, skipframes int) {
    root := semtable.rootFor(addr)
    atomic.Xadd(addr, 1)

    // Easy case: no waiters?
    if root.nwait.Load() == 0 {
        return
    }

    // Harder case: search for a waiter and wake it.
    lockWithRank(&root.lock, lockRankRoot)
    if root.nwait.Load() == 0 {
        unlock(&root.lock)
        return
    }
    s, t0, tailtime := root.dequeue(addr)
    if s != nil {
        root.nwait.Add(-1)
    }
    unlock(&root.lock)
    if s != nil {
        // ...profiling...
        if handoff && cansemacquire(addr) {
            s.ticket = 1
        }
        readyWithTime(s, 5+skipframes)
        if s.ticket == 1 && getg().m.locks == 0 && getg() != getg().m.g0 {
            // Direct G handoff
            goyield()
        }
    }
}

Key steps:

1. Atomically increment the semaphore counter
2. Check nwait (without the lock) – if zero, no waiters, return
3. Lock the root, dequeue a waiter from the treap, unlock
4. If handoff mode and we can re-acquire the semaphore, set ticket = 1 (direct grant)
5. Mark the waiter as ready (readyWithTime)
6. In handoff mode, yield immediately via goyield() to let the waiter run

Starvation Handling and Handoff Mode

The handoff parameter is critical for fairness. When sync.Mutex enters starvation mode (a goroutine has been waiting > 1ms), it calls semrelease with handoff=true.

src/runtime/sema.go, lines 260-287

// src/runtime/sema.go, lines 260-287
        if handoff && cansemacquire(addr) {
            s.ticket = 1
        }
        readyWithTime(s, 5+skipframes)
        if s.ticket == 1 && getg().m.locks == 0 && getg() != getg().m.g0 {
            // Direct G handoff
            //
            // readyWithTime has added the waiter G as runnext in the
            // current P; we now call the scheduler so that we start running
            // the waiter G immediately.
            //
            // Note that waiter inherits our time slice: this is desirable
            // to avoid having a highly contended semaphore hog the P
            // indefinitely. goyield is like Gosched, but it emits a
            // "preempted" trace event instead and, more importantly, puts
            // the current G on the local runq instead of the global one.
            // We only do this in the starving regime (handoff=true), as in
            // the non-starving case it is possible for a different waiter
            // to acquire the semaphore while we are yielding/scheduling,
            // and this would be wasteful.
            goyield()
        }

In handoff mode:

1. The releaser atomically grabs the semaphore on behalf of the waiter (s.ticket = 1)
2. The waiter is placed in runnext of the current P
3. The releaser calls goyield() to immediately switch to the waiter
4. This prevents new goroutines from stealing the semaphore between release and acquire

The queue Method: Treap Insertion

src/runtime/sema.go, lines 370-396

// src/runtime/sema.go, lines 370-396
    // Add s as new leaf in tree of unique addrs.
    // The balanced tree is a treap using ticket as the random heap priority.
    // That is, it is a binary tree ordered according to the elem addresses,
    // but then among the space of possible binary trees respecting those
    // addresses, it is kept balanced on average by maintaining a heap ordering
    // on the ticket: s.ticket <= both s.prev.ticket and s.next.ticket.
    // https://en.wikipedia.org/wiki/Treap
    // https://faculty.washington.edu/aragon/pubs/rst89.pdf
    //
    // s.ticket compared with zero in couple of places, therefore set lowest bit.
    // It will not affect treap's quality noticeably.
    s.ticket = cheaprand() | 1
    s.parent = last
    *pt = s

    // Rotate up into tree according to ticket (priority).
    for s.parent != nil && s.parent.ticket > s.ticket {
        if s.parent.prev == s {
            root.rotateRight(s.parent)
        } else {
            if s.parent.next != s {
                panic("semaRoot queue")
            }
            root.rotateLeft(s.parent)
        }
    }

The treap maintains balance probabilistically:

1. Insert as a BST leaf (ordered by address)
2. Assign a random priority (cheaprand() | 1)
3. Rotate up while parent has higher priority
4. The | 1 ensures ticket is never zero (zero has special meaning for handoff)

7. Reader-Writer Mutex (10 min)

Overview

The runtime’s rwmutex (in rwmutex.go) is used internally by the runtime (e.g., for protecting allocator state). It’s a simplified version of sync.RWMutex.

src/runtime/rwmutex.go, lines 18-30

// src/runtime/rwmutex.go, lines 18-30
type rwmutex struct {
    rLock      mutex    // protects readers, readerPass, writer
    readers    muintptr // list of pending readers
    readerPass uint32   // number of pending readers to skip readers list

    wLock  mutex    // serializes writers
    writer muintptr // pending writer waiting for completing readers

    readerCount atomic.Int32 // number of pending readers
    readerWait  atomic.Int32 // number of departing readers

    readRank lockRank // semantic lock rank for read locking
}

The rwmutexMaxReaders Trick

src/runtime/rwmutex.go, line 67

// src/runtime/rwmutex.go, line 67
const rwmutexMaxReaders = 1 << 30

This constant is the key to the reader-writer protocol. readerCount serves dual purpose:

• When positive: number of active readers
• When negative: a writer is pending (readerCount was decremented by rwmutexMaxReaders)

Read Lock (rlock)

src/runtime/rwmutex.go, lines 70-98

// src/runtime/rwmutex.go, lines 70-98
func (rw *rwmutex) rlock() {
    acquireLockRankAndM(rw.readRank)
    lockWithRankMayAcquire(&rw.rLock, getLockRank(&rw.rLock))

    if rw.readerCount.Add(1) < 0 {
        // A writer is pending. Park on the reader queue.
        systemstack(func() {
            lock(&rw.rLock)
            if rw.readerPass > 0 {
                // Writer finished.
                rw.readerPass -= 1
                unlock(&rw.rLock)
            } else {
                // Queue this reader to be woken by the writer.
                m := getg().m
                m.schedlink = rw.readers
                rw.readers.set(m)
                unlock(&rw.rLock)
                notesleep(&m.park)
                noteclear(&m.park)
            }
        })
    }
}

The flow:

1. Atomically increment readerCount
2. If result is positive: no writer pending, proceed (fast path)
3. If result is negative: a writer has subtracted rwmutexMaxReaders
   • Check readerPass – if > 0, the writer already finished and we can proceed
   • Otherwise, add ourselves to the readers list and sleep via notesleep

Write Lock (lock)

src/runtime/rwmutex.go, lines 121-140

// src/runtime/rwmutex.go, lines 121-140
func (rw *rwmutex) lock() {
    // Resolve competition with other writers and stick to our P.
    lock(&rw.wLock)
    m := getg().m
    // Announce that there is a pending writer.
    r := rw.readerCount.Add(-rwmutexMaxReaders) + rwmutexMaxReaders
    // Wait for any active readers to complete.
    lock(&rw.rLock)
    if r != 0 && rw.readerWait.Add(r) != 0 {
        // Wait for reader to wake us up.
        systemstack(func() {
            rw.writer.set(m)
            unlock(&rw.rLock)
            notesleep(&m.park)
            noteclear(&m.park)
        })
    } else {
        unlock(&rw.rLock)
    }
}

The protocol:

1. Acquire wLock to serialize writers
2. Subtract rwmutexMaxReaders from readerCount – this makes it negative, signaling readers
3. The return value r tells us how many readers were active at that moment
4. Add r to readerWait – as readers finish, they decrement readerWait
5. If readerWait becomes 0 (or was already 0), all readers are done
6. Otherwise, sleep until the last departing reader wakes us

Read Unlock (runlock)

src/runtime/rwmutex.go, lines 101-118

// src/runtime/rwmutex.go, lines 101-118
func (rw *rwmutex) runlock() {
    if r := rw.readerCount.Add(-1); r < 0 {
        if r+1 == 0 || r+1 == -rwmutexMaxReaders {
            throw("runlock of unlocked rwmutex")
        }
        // A writer is pending.
        if rw.readerWait.Add(-1) == 0 {
            // The last reader unblocks the writer.
            lock(&rw.rLock)
            w := rw.writer.ptr()
            if w != nil {
                notewakeup(&w.park)
            }
            unlock(&rw.rLock)
        }
    }
    releaseLockRankAndM(rw.readRank)
}

When a reader unlocks:

1. Decrement readerCount
2. If result is negative, a writer is waiting
3. Decrement readerWait – if we’re the last departing reader (readerWait reaches 0), wake the writer

Write Unlock

src/runtime/rwmutex.go, lines 143-164

// src/runtime/rwmutex.go, lines 143-164
func (rw *rwmutex) unlock() {
    // Announce to readers that there is no active writer.
    r := rw.readerCount.Add(rwmutexMaxReaders)
    if r >= rwmutexMaxReaders {
        throw("unlock of unlocked rwmutex")
    }
    // Unblock blocked readers.
    lock(&rw.rLock)
    for rw.readers.ptr() != nil {
        reader := rw.readers.ptr()
        rw.readers = reader.schedlink
        reader.schedlink.set(nil)
        notewakeup(&reader.park)
        r -= 1
    }
    // If r > 0, there are pending readers that aren't on the
    // queue. Tell them to skip waiting.
    rw.readerPass += uint32(r)
    unlock(&rw.rLock)
    // Allow other writers to proceed.
    unlock(&rw.wLock)
}

The writer unlock:

1. Add rwmutexMaxReaders back to readerCount (makes it positive again)
2. Wake all readers on the readers list via notewakeup
3. If r > 0, some readers arrived after the writer and already incremented readerCount but haven’t been queued yet. Set readerPass so they skip the queue.
4. Release wLock to allow other writers

The Elegance of the Design

The rwmutexMaxReaders trick encodes the “writer pending” state directly in the reader counter. This means:

• Readers in the common (no-writer) case do a single atomic add and proceed
• The writer’s presence is communicated through the sign of readerCount
• No separate “writer waiting” flag is needed
• The maximum number of concurrent readers is 2^30 (about 1 billion), which is more than sufficient

8. Summary: The Synchronization Hierarchy

Key insight: The runtime has two parallel stacks of synchronization – one for blocking OS threads (used by the scheduler itself) and one for blocking goroutines (used by Go code). They use the same OS primitives at the bottom but diverge at the level above.

Discussion Questions

1. Why does the runtime need both lock_futex.go and lock_sema.go? Why not use one approach everywhere?

2. In the sema.go treap, why use a treap instead of a simpler structure like a hash table of linked lists?

3. The runtime mutex uses an 8-bit atomic swap for the fast path. Why is this faster than a full word-width CAS?

4. In the rwmutex, why does the writer subtract rwmutexMaxReaders from readerCount instead of using a separate “writer waiting” boolean?

5. Consider semrelease1 with handoff=true. Why does it call goyield() instead of Gosched()? What’s the difference?

Further Reading

• Ulrich Drepper, “Futexes Are Tricky” (2011) – the classic futex reference
• Mullender and Cox, “Semaphores in Plan 9” – cited in sema.go’s header comment
• sync.Mutex source – see how it uses runtime_SemacquireMutex
• Aragon and Seidel, “Randomized Search Trees” (1989) – the treap paper cited in sema.go

Module 6: Synchronization Primitives (futex, semaphores, rwmutex)

Comprehension Check

Question 1 (Code Reading)

Consider this implementation of futexsleep from runtime/os_linux.go:

func futexsleep(addr *uint32, val uint32, ns int64) {
	if ns < 0 {
		futex(unsafe.Pointer(addr), _FUTEX_WAIT_PRIVATE, val, nil, nil, 0)
		return
	}

	var ts timespec
	ts.setNsec(ns)
	futex(unsafe.Pointer(addr), _FUTEX_WAIT_PRIVATE, val, &ts, nil, 0)
}

What does _FUTEX_WAIT_PRIVATE mean, and why does the runtime use the “PRIVATE” variant instead of plain FUTEX_WAIT?

Question 2 (Short Answer)

The Go runtime implements its own mutex type (in lock_futex.go or lock_sema.go) rather than using pthread_mutex_t. What are the key differences between the runtime’s mutex and a pthread mutex? Why not use pthreads?

Question 3 (True/False with Explanation)

True or False: Go’s sync.Mutex is implemented directly using the runtime’s internal runtime.mutex (the futex-based lock).

Question 4 (What Would Happen If…)

Go’s sync.RWMutex allows multiple concurrent readers or a single writer. What would happen if RLock()/RUnlock() used an atomic counter but did not coordinate with writers at all (i.e., readers never blocked even when a writer is waiting)?

Question 5 (Short Answer)

Explain the difference between a semaphore and a futex. How does the Go runtime use each?

Question 6 (Code Reading)

Consider a simplified version of the semaphore acquire path:

func semacquire1(addr *uint32, lifo bool, profile semaProfileFlags, skipframes int, reason waitReason) {
	// Fast path: *addr > 0, try to decrement
	if cansemacquire(addr) {
		return
	}

	// Slow path: increment waiter count, then try to acquire again
	s := acquireSudog()
	root := semtable[addr.hash()].root
	root.queue(addr, s, lifo)
	gopark_m(...)  // park the goroutine
	releaseSudog(s)
}

Why does the slow path increment the waiter count and try again before actually parking? What race condition does this double-check prevent?

Module 7: Channels and Select (60 min)

Background: Message Passing and the CSP Tradition

Concurrent programs must coordinate access to shared state, and the history of systems programming offers two fundamental paradigms for doing so. The first, shared memory with synchronization, gives threads direct access to common data structures and uses locks, condition variables, and atomic operations to prevent races. This is the model of pthreads, Java’s synchronized blocks, and the mutexes we studied earlier in this course. The second paradigm, message passing, structures concurrent programs as independent sequential processes that communicate by sending and receiving messages over explicit channels. Rather than protecting shared data with locks, message-passing programs avoid sharing altogether: data moves between processes by copying (or transferring ownership), and synchronization is implicit in the send/receive handshake. The distinction is not absolute – as we will see, Go’s channel implementation itself uses a mutex internally – but it reflects a deep difference in how programmers reason about concurrency. The shared-memory style asks “how do I protect this data?” while the message-passing style asks “how do I route this data?”

The theoretical foundations of message passing trace to two landmark papers from the 1970s. Carl Hewitt’s Actor model (1973) proposed that computation consists of autonomous actors that communicate by sending asynchronous messages to named recipients; each actor has a mailbox, processes messages one at a time, and can create new actors or send messages in response. Tony Hoare’s Communicating Sequential Processes (CSP, 1978) took a different approach: processes communicate through unnamed, synchronous channels, and a send blocks until a matching receive occurs (and vice versa). Where the Actor model emphasizes identity (messages are sent to a specific actor), CSP emphasizes the channel as a first-class connective tissue between anonymous processes. The Actor model’s asynchronous, identity-based messaging influenced Erlang, Akka, and the “let it crash” school of fault-tolerant systems. CSP’s synchronous, channel-based model influenced occam, Limbo, and – most prominently today – Go. Both models avoid shared mutable state, but they differ in buffering semantics, naming, and the role of the communication medium itself.

These ideas have been realized in strikingly different ways across programming languages and systems. Erlang processes communicate through asynchronous mailboxes with selective receive, giving each process an unbounded message queue and pattern matching to pluck out messages of interest – a direct descendant of the Actor model. Rust’s standard library provides mpsc (multi-producer, single-consumer) channels that transfer ownership of values, leveraging the type system to guarantee that sent data cannot be accessed by the sender afterward; the crossbeam crate extends this with multi-consumer channels and a select! macro. Kotlin offers channels within its coroutine framework, with configurable buffering policies (rendezvous, buffered, conflated, unlimited) that let programmers tune the coupling between producer and consumer. Clojure’s core.async brings CSP-style channels to the JVM with go blocks that compile to state machines, much as Go’s goroutines are multiplexed onto OS threads. Even Unix pipes, dating to Doug McIlroy’s 1964 proposal and Ken Thompson’s 1973 implementation, embody the same principle: composing programs as sequential stages connected by byte streams, with back-pressure arising naturally from the pipe buffer’s bounded capacity. Go’s channels sit squarely in this lineage, offering typed, optionally-buffered channels as first-class values, with goroutines as lightweight processes.

A key operation in any channel-based system is multiplexing: waiting on multiple channels simultaneously and proceeding with whichever is ready first. This problem has deep roots in operating systems. Unix’s select() system call (4.2BSD, 1983) let a process block until any file descriptor in a set became ready for I/O – solving the problem of a server needing to handle multiple clients without dedicating a thread to each one. poll() improved the interface, and Linux’s epoll (2002) scaled it to hundreds of thousands of descriptors. Go’s select statement is the channel-level analogue: it blocks until one of several channel operations can proceed, then executes that case. But Go’s select adds two refinements absent from Unix I/O multiplexing. First, randomized fairness: when multiple cases are ready simultaneously, select chooses one uniformly at random, preventing starvation. Second, deadlock-safe locking: the select implementation acquires locks on all involved channels in address order to prevent lock-ordering deadlocks, a technique borrowed from database concurrency control. The combination of randomization and careful lock management makes select both fair and safe, properties that are surprisingly difficult to achieve together.

In this module, we open up Go’s channel and select implementation to see how these ideas are realized in practice. The runtime’s hchan struct is a mutex-protected circular buffer with embedded wait queues – a bounded producer-consumer buffer, much like the ones studied in OS textbooks, but with several clever optimizations such as direct sender-to-receiver copying that bypasses the buffer entirely. The select implementation is a carefully orchestrated multi-pass algorithm that shuffles cases for fairness, sorts them for deadlock freedom, and uses per-goroutine “pseudo-g” (sudog) structures to allow a single goroutine to wait on many channels at once. Understanding this machinery connects the high-level elegance of CSP to the low-level realities of locks, memory layout, and scheduler integration.

1. Channel Invariants

The channel implementation maintains a key structural invariant, documented at the top of chan.go:

// src/runtime/chan.go, lines 9-18 (https://cs.opensource.google/go/go/+/refs/tags/go1.26.1:src/runtime/chan.go;l=9)

// Invariants:
//  At least one of c.sendq and c.recvq is empty,
//  except for the case of an unbuffered channel with a single goroutine
//  blocked on it for both sending and receiving using a select statement,
//  in which case the length of c.sendq and c.recvq is limited only by the
//  size of the select statement.
//
// For buffered channels, also:
//  c.qcount > 0 implies that c.recvq is empty.
//  c.qcount < c.dataqsiz implies that c.sendq is empty.

These invariants encode a fundamental property: if there is data in the buffer, no goroutine should be blocked waiting to receive; if the buffer is not full, no goroutine should be blocked waiting to send. The only exception is select statements on unbuffered channels, where a single goroutine might appear on both queues simultaneously.

OS connection: These invariants are analogous to the monitor invariants in classical synchronization – they define the legal states of the shared data structure and constrain when threads (goroutines) may block.

2. The hchan Struct: Channel Internals

Every channel in Go is represented at runtime by an hchan struct:

// src/runtime/chan.go, lines 34-55 (https://cs.opensource.google/go/go/+/refs/tags/go1.26.1:src/runtime/chan.go;l=34)

type hchan struct {
    qcount   uint           // total data in the queue
    dataqsiz uint           // size of the circular queue
    buf      unsafe.Pointer // points to an array of dataqsiz elements
    elemsize uint16
    closed   uint32
    timer    *timer // timer feeding this chan
    elemtype *_type // element type
    sendx    uint   // send index
    recvx    uint   // receive index
    recvq    waitq  // list of recv waiters
    sendq    waitq  // list of send waiters
    bubble   *synctestBubble

    // lock protects all fields in hchan, as well as several
    // fields in sudogs blocked on this channel.
    //
    // Do not change another G's status while holding this lock
    // (in particular, do not ready a G), as this can deadlock
    // with stack shrinking.
    lock mutex
}

Field-by-field analysis

The sendx and recvx indices implement a standard circular buffer: they advance through the buffer array and wrap to 0 when they reach dataqsiz. The channel is full when qcount == dataqsiz and empty when qcount == 0.

Important design note: The comment on lock warns against readying a goroutine while holding the channel lock, because goready can trigger stack shrinking, and stack shrinking also needs to acquire channel locks – creating a potential deadlock cycle. This is a concrete example of lock ordering constraints in the runtime.

3. The waitq and sudog Structs

waitq

The wait queue is a simple doubly-linked list:

// src/runtime/chan.go, lines 57-60 (https://cs.opensource.google/go/go/+/refs/tags/go1.26.1:src/runtime/chan.go;l=57)

type waitq struct {
    first *sudog
    last  *sudog
}

sudog (pseudo-g)

A sudog represents a goroutine waiting on a synchronization object. The name stands for “pseudo-g” – it is a proxy for a g (goroutine) in a wait list:

// src/runtime/runtime2.go, lines 396-448 (https://cs.opensource.google/go/go/+/refs/tags/go1.26.1:src/runtime/runtime2.go;l=396)

// sudog (pseudo-g) represents a g in a wait list, such as for sending/receiving
// on a channel.
//
// sudog is necessary because the g <-> synchronization object relation
// is many-to-many. A g can be on many wait lists, so there may be
// many sudogs for one g; and many gs may be waiting on the same
// synchronization object, so there may be many sudogs for one object.
//
// sudogs are allocated from a special pool. Use acquireSudog and
// releaseSudog to allocate and free them.
type sudog struct {
    // The following fields are protected by the hchan.lock of the
    // channel this sudog is blocking on. shrinkstack depends on
    // this for sudogs involved in channel ops.

    g *g

    next *sudog
    prev *sudog

    elem maybeTraceablePtr // data element (may point to stack)

    // The following fields are never accessed concurrently.
    // For channels, waitlink is only accessed by g.
    // For semaphores, all fields (including the ones above)
    // are only accessed when holding a semaRoot lock.

    acquiretime int64
    releasetime int64
    ticket      uint32

    // isSelect indicates g is participating in a select, so
    // g.selectDone must be CAS'd to win the wake-up race.
    isSelect bool

    // success indicates whether communication over channel c
    // succeeded. It is true if the goroutine was awoken because a
    // value was delivered over channel c, and false if awoken
    // because c was closed.
    success bool

    waiters uint16

    parent   *sudog             // semaRoot binary tree
    waitlink *sudog             // g.waiting list or semaRoot
    waittail *sudog             // semaRoot
    c        maybeTraceableChan // channel
}

Why sudogs exist

A goroutine blocked in a select statement may be waiting on multiple channels simultaneously. A single g struct cannot appear in multiple wait queues, so the runtime allocates a separate sudog for each channel case. This is the “many-to-many” relationship the comment describes.

Key fields for channel operations:

• g: Points back to the goroutine this sudog represents.
• next/prev: Links for the channel’s waitq linked list.
• elem: Pointer to the data element being sent or received. May point into the goroutine’s stack.
• isSelect: If true, waking this sudog requires a CAS on g.selectDone to handle the race where multiple channels become ready simultaneously.
• success: Indicates whether the channel operation completed normally (true) or the channel was closed (false).
• waitlink: Used by select to chain all of a goroutine’s sudogs together for cleanup.

OS connection: The sudog pool is analogous to how OS kernels pre-allocate wait queue entries rather than embedding them in thread structures, allowing a thread to wait on multiple events simultaneously (cf. Linux poll/epoll implementation).

4. makechan: Channel Creation

The makechan function is called by the compiler when it encounters make(chan T, size):

// src/runtime/chan.go, lines 75-125 (https://cs.opensource.google/go/go/+/refs/tags/go1.26.1:src/runtime/chan.go;l=75)

func makechan(t *chantype, size int) *hchan {
    elem := t.Elem

    // compiler checks this but be safe.
    if elem.Size_ >= 1<<16 {
        throw("makechan: invalid channel element type")
    }
    if hchanSize%maxAlign != 0 || elem.Align_ > maxAlign {
        throw("makechan: bad alignment")
    }

    mem, overflow := math.MulUintptr(elem.Size_, uintptr(size))
    if overflow || mem > maxAlloc-hchanSize || size < 0 {
        panic(plainError("makechan: size out of range"))
    }

    // Hchan does not contain pointers interesting for GC when elements
    // stored in buf do not contain pointers.
    // buf points into the same allocation, elemtype is persistent.
    // SudoG's are referenced from their owning thread so they can't be collected.
    var c *hchan
    switch {
    case mem == 0:
        // Queue or element size is zero.
        c = (*hchan)(mallocgc(hchanSize, nil, true))
        // Race detector uses this location for synchronization.
        c.buf = c.raceaddr()
    case !elem.Pointers():
        // Elements do not contain pointers.
        // Allocate hchan and buf in one call.
        c = (*hchan)(mallocgc(hchanSize+mem, nil, true))
        c.buf = add(unsafe.Pointer(c), hchanSize)
    default:
        // Elements contain pointers.
        c = new(hchan)
        c.buf = mallocgc(mem, elem, true)
    }

    c.elemsize = uint16(elem.Size_)
    c.elemtype = elem
    c.dataqsiz = uint(size)
    lockInit(&c.lock, lockRankHchan)

    if debugChan {
        print("makechan: chan=", c, "; elemsize=", elem.Size_, "; dataqsiz=", size, "\n")
    }
    return c
}

Three allocation strategies

1. Unbuffered or zero-size elements (mem == 0): Only the hchan struct itself is allocated. No buffer is needed.

2. Pointer-free elements (!elem.Pointers()): The hchan struct and the buffer are allocated in a single mallocgc call. The buffer is placed immediately after the struct in memory (c.buf = add(unsafe.Pointer(c), hchanSize)). This is an optimization: the GC doesn’t need to scan the buffer, so a single non-pointer allocation suffices.

3. Pointer-containing elements (default): Two separate allocations are needed. The buffer must be allocated with type information (mallocgc(mem, elem, true)) so the GC can scan it for pointers.

OS connection: This is a nice example of how a runtime can cooperate with its garbage collector by choosing different allocation strategies based on type information – something impossible with a C-style malloc that has no type awareness.

5. chansend: Sending on a Channel

The chansend function is the heart of the send operation (c <- x):

// src/runtime/chan.go, lines 176-183 (https://cs.opensource.google/go/go/+/refs/tags/go1.26.1:src/runtime/chan.go;l=176)

func chansend(c *hchan, ep unsafe.Pointer, block bool, callerpc uintptr) bool {
    if c == nil {
        if !block {
            return false
        }
        gopark(nil, nil, waitReasonChanSendNilChan, traceBlockForever, 2)
        throw("unreachable")
    }

Sending on a nil channel parks the goroutine forever. This is by design: it allows select statements to disable cases by setting channel variables to nil.

Phase 1: Non-blocking fast path (no lock)

// src/runtime/chan.go, lines 197-215 (https://cs.opensource.google/go/go/+/refs/tags/go1.26.1:src/runtime/chan.go;l=197)

    // Fast path: check for failed non-blocking operation without acquiring the lock.
    //
    // After observing that the channel is not closed, we observe that the channel is
    // not ready for sending. Each of these observations is a single word-sized read
    // (first c.closed and second full()).
    // Because a closed channel cannot transition from 'ready for sending' to
    // 'not ready for sending', even if the channel is closed between the two
    // observations, they imply a moment between the two when the channel was both
    // not yet closed and not ready for sending. We behave as if we observed the
    // channel at that moment, and report that the send cannot proceed.
    if !block && c.closed == 0 && full(c) {
        return false
    }

This is a lock-free fast path for non-blocking sends (used by select with a default case). The comment explains a subtle linearizability argument: even though the two reads (c.closed and full(c)) are not atomic together, the monotonicity of closed (once closed, always closed) guarantees correctness.

Phase 2: Direct send to waiting receiver

// src/runtime/chan.go, lines 222-234 (https://cs.opensource.google/go/go/+/refs/tags/go1.26.1:src/runtime/chan.go;l=222)

    lock(&c.lock)

    if c.closed != 0 {
        unlock(&c.lock)
        panic(plainError("send on closed channel"))
    }

    if sg := c.recvq.dequeue(); sg != nil {
        // Found a waiting receiver. We pass the value we want to send
        // directly to the receiver, bypassing the channel buffer (if any).
        send(c, sg, ep, func() { unlock(&c.lock) }, 3)
        return true
    }

If a receiver is already waiting, the send copies the value directly to the receiver’s stack, bypassing the channel buffer entirely. This is an important optimization: even on a buffered channel, if a receiver is blocked, the data goes straight to the receiver without first going into the buffer and then back out.

Phase 3: Buffered send

// src/runtime/chan.go, lines 236-250 (https://cs.opensource.google/go/go/+/refs/tags/go1.26.1:src/runtime/chan.go;l=236)

    if c.qcount < c.dataqsiz {
        // Space is available in the channel buffer. Enqueue the element to send.
        qp := chanbuf(c, c.sendx)
        typedmemmove(c.elemtype, qp, ep)
        c.sendx++
        if c.sendx == c.dataqsiz {
            c.sendx = 0
        }
        c.qcount++
        unlock(&c.lock)
        return true
    }

Standard circular buffer enqueue: copy the element to the sendx slot, advance the index (wrapping at dataqsiz), increment the count.

Phase 4: Blocking

// src/runtime/chan.go, lines 257-283 (https://cs.opensource.google/go/go/+/refs/tags/go1.26.1:src/runtime/chan.go;l=257)

    // Block on the channel. Some receiver will complete our operation for us.
    gp := getg()
    mysg := acquireSudog()
    mysg.releasetime = 0
    if t0 != 0 {
        mysg.releasetime = -1
    }
    // No stack splits between assigning elem and enqueuing mysg
    // on gp.waiting where copystack can find it.
    mysg.elem.set(ep)
    mysg.waitlink = nil
    mysg.g = gp
    mysg.isSelect = false
    mysg.c.set(c)
    gp.waiting = mysg
    gp.param = nil
    c.sendq.enqueue(mysg)
    // Signal to anyone trying to shrink our stack that we're about
    // to park on a channel. The window between when this G's status
    // changes and when we set gp.activeStackChans is not safe for
    // stack shrinking.
    gp.parkingOnChan.Store(true)
    reason := waitReasonChanSend
    if c.bubble != nil {
        reason = waitReasonSynctestChanSend
    }
    gopark(chanparkcommit, unsafe.Pointer(&c.lock), reason, traceBlockChanSend, 2)

When the buffer is full (or the channel is unbuffered and no receiver is waiting), the goroutine parks itself:

1. Acquires a sudog from the pool.
2. Records the element pointer (mysg.elem) so the eventual receiver can copy the data.
3. Enqueues the sudog on c.sendq.
4. Calls gopark to suspend the goroutine, releasing the channel lock atomically.

When a receiver eventually arrives, it will dequeue this sudog, copy the data from mysg.elem, and call goready to wake the sender.

Critical subtlety: The comment about “No stack splits” is important. Between setting mysg.elem (which points into the sender’s stack) and enqueuing the sudog where copystack can find it, a stack copy would invalidate the pointer. The runtime prevents this by ensuring no function calls that could trigger stack growth occur in this window.

6. chanrecv: Receiving from a Channel

The receive operation (<-c) is implemented by chanrecv, which is symmetric to chansend:

// src/runtime/chan.go, lines 518-524 (https://cs.opensource.google/go/go/+/refs/tags/go1.26.1:src/runtime/chan.go;l=518)

// chanrecv receives on channel c and writes the received data to ep.
// ep may be nil, in which case received data is ignored.
// If block == false and no elements are available, returns (false, false).
// Otherwise, if c is closed, zeros *ep and returns (true, false).
// Otherwise, fills in *ep with an element and returns (true, true).
// A non-nil ep must point to the heap or the caller's stack.
func chanrecv(c *hchan, ep unsafe.Pointer, block bool) (selected, received bool) {

Receive phases

The function follows the same structure as chansend:

1. Nil channel check (line 532-538): Receiving from a nil channel parks forever.

2. Non-blocking fast path (lines 548-579): Check empty(c) without the lock. If the channel is closed and empty, return the zero value.

3. Direct receive from waiting sender (lines 600-609):

   if sg := c.sendq.dequeue(); sg != nil {
       // Found a waiting sender. If buffer is size 0, receive value
       // directly from sender. Otherwise, receive from head of queue
       // and add sender's value to the tail of the queue (both map to
       // the same buffer slot because the queue is full).
       recv(c, sg, ep, func() { unlock(&c.lock) }, 3)
       return true, true
   }
   

   For unbuffered channels, data is copied directly from sender to receiver. For buffered channels (which must be full if a sender is blocked), the receiver takes from the buffer head, and the sender’s value fills the buffer tail.

4. Buffered receive (lines 612-629): Standard circular buffer dequeue.

   if c.qcount > 0 {
       // Receive directly from queue
       qp := chanbuf(c, c.recvx)
       if ep != nil {
           typedmemmove(c.elemtype, ep, qp)
       }
       typedmemclr(c.elemtype, qp)
       c.recvx++
       if c.recvx == c.dataqsiz {
           c.recvx = 0
       }
       c.qcount--
       unlock(&c.lock)
       return true, true
   }
   

5. Blocking (lines 636-685): Acquire a sudog, enqueue on c.recvq, park.

7. closechan: Closing a Channel

Closing a channel wakes up all waiters:

// src/runtime/chan.go, lines 414-486 (https://cs.opensource.google/go/go/+/refs/tags/go1.26.1:src/runtime/chan.go;l=414)

func closechan(c *hchan) {
    if c == nil {
        panic(plainError("close of nil channel"))
    }

    lock(&c.lock)
    if c.closed != 0 {
        unlock(&c.lock)
        panic(plainError("close of closed channel"))
    }

    c.closed = 1

    var glist gList

    // release all readers
    for {
        sg := c.recvq.dequeue()
        if sg == nil {
            break
        }
        if sg.elem.get() != nil {
            typedmemclr(c.elemtype, sg.elem.get())
            sg.elem.set(nil)
        }
        gp := sg.g
        gp.param = unsafe.Pointer(sg)
        sg.success = false
        glist.push(gp)
    }

    // release all writers (they will panic)
    for {
        sg := c.sendq.dequeue()
        if sg == nil {
            break
        }
        sg.elem.set(nil)
        gp := sg.g
        gp.param = unsafe.Pointer(sg)
        sg.success = false
        glist.push(gp)
    }
    unlock(&c.lock)

    // Ready all Gs now that we've dropped the channel lock.
    for !glist.empty() {
        gp := glist.pop()
        gp.schedlink = 0
        goready(gp, 3)
    }
}

Key points

1. Double close panics: The runtime checks c.closed != 0 under the lock.

2. Receivers get zero values: The typedmemclr call zeros out the receive buffer. The success = false flag tells receivers they woke due to close, not data delivery.

3. Senders will panic: Woken senders see success == false, check c.closed, and panic with “send on closed channel.”

4. Lock ordering: All goroutines are collected into glist before releasing the lock, then readied after the lock is released. This follows the constraint from the hchan.lock comment: “do not ready a G while holding this lock.”

OS connection: This wake-all pattern is analogous to pthread_cond_broadcast or the Linux FUTEX_WAKE with a large wake count. Closing a channel is effectively a broadcast signal to all waiters.

8. The select Statement Implementation

The scase struct

Each case in a select statement is described by an scase:

// src/runtime/select.go, lines 20-23 (https://cs.opensource.google/go/go/+/refs/tags/go1.26.1:src/runtime/select.go;l=20)

type scase struct {
    c    *hchan         // chan
    elem unsafe.Pointer // data element
}

This is remarkably simple: just a channel pointer and a pointer to the data element. The direction (send vs. receive) is implicit in the position: send cases come first, followed by receive cases.

Compiler optimizations

Before selectgo is called, the compiler has already optimized away simple cases:

• select {} (no cases): compiled to a direct block() call (parks forever).
• select with one case + default: compiled to a non-blocking chansend/chanrecv.
• select with one case (no default): compiled to a blocking chansend/chanrecv.

Only multi-case selects (or degenerate cases where channels have been niled out) reach selectgo.

selectgo: The Multi-Pass Algorithm

// src/runtime/select.go, lines 107-122 (https://cs.opensource.google/go/go/+/refs/tags/go1.26.1:src/runtime/select.go;l=107)

// selectgo implements the select statement.
//
// cas0 points to an array of type [ncases]scase, and order0 points to
// an array of type [2*ncases]uint16 where ncases must be <= 65536.
// Both reside on the goroutine's stack (regardless of any escaping in
// selectgo).
//
// selectgo returns the index of the chosen scase, which matches the
// ordinal position of its respective select{recv,send,default} call.
// Also, if the chosen scase was a receive operation, it reports whether
// a value was received.
func selectgo(cas0 *scase, order0 *uint16, pc0 *uintptr, nsends, nrecvs int,
              block bool) (int, bool) {

The function takes a flat array of cases and an array that is split into two halves: pollorder (randomized iteration order) and lockorder (sorted by channel address for lock acquisition).

Step 1: Generate randomized poll order

// src/runtime/select.go, lines 167-177

    // generate permuted order
    norder := 0
    for i := range scases {
        cas := &scases[i]

        // Omit cases without channels from the poll and lock orders.
        if cas.c == nil {
            cas.elem = nil // allow GC
            continue
        }

        // ...

        j := cheaprandn(uint32(norder + 1))
        pollorder[norder] = pollorder[j]
        pollorder[j] = uint16(i)
        norder++
    }

Cases with nil channels are skipped. The remaining cases are shuffled using a Fisher-Yates shuffle (cheaprandn). This randomization is critical for fairness: without it, earlier cases in the source code would always have priority, potentially starving later cases.

Step 2: Sort lock order by channel address

// src/runtime/select.go, lines 206-240

    // sort the cases by Hchan address to get the locking order.
    // simple heap sort, to guarantee n log n time and constant stack footprint.
    for i := range lockorder {
        j := i
        // Start with the pollorder to permute cases on the same channel.
        c := scases[pollorder[i]].c
        for j > 0 && scases[lockorder[(j-1)/2]].c.sortkey() < c.sortkey() {
            k := (j - 1) / 2
            lockorder[j] = lockorder[k]
            j = k
        }
        lockorder[j] = pollorder[i]
    }

A heap sort orders cases by channel address. This establishes a global lock ordering that prevents deadlocks when two goroutines running select statements try to lock overlapping sets of channels.

OS connection: This is a textbook application of lock ordering to prevent deadlock – the same technique used in database systems (two-phase locking with ordered resources) and OS kernels (e.g., Linux’s lock ordering for inodes).

Pass 1: Try all cases without blocking (fast path)

// src/runtime/select.go, lines 264-301

    // pass 1 - look for something already waiting
    var casi int
    var cas *scase
    var caseSuccess bool
    for _, casei := range pollorder {
        casi = int(casei)
        cas = &scases[casi]
        c = cas.c

        if casi >= nsends {
            sg = c.sendq.dequeue()
            if sg != nil {
                goto recv
            }
            if c.qcount > 0 {
                goto bufrecv
            }
            if c.closed != 0 {
                goto rclose
            }
        } else {
            if c.closed != 0 {
                goto sclose
            }
            sg = c.recvq.dequeue()
            if sg != nil {
                goto send
            }
            if c.qcount < c.dataqsiz {
                goto bufsend
            }
        }
    }

    if !block {
        selunlock(scases, lockorder)
        casi = -1
        goto retc
    }

After locking all channels (in lock order), the function iterates through cases in the randomized poll order. For each case:

• Receive cases (casi >= nsends): Check for a waiting sender, buffered data, or a closed channel.
• Send cases: Check for close (panic), a waiting receiver, or buffer space.

If any case can proceed immediately, the function jumps to the appropriate handler, which performs the operation, unlocks all channels, and returns.

If no case is ready and there’s a default clause (!block), unlock and return -1.

Pass 2: Enqueue on all channels and park

// src/runtime/select.go, lines 309-351

    // pass 2 - enqueue on all chans
    nextp = &gp.waiting
    for _, casei := range lockorder {
        casi = int(casei)
        cas = &scases[casi]
        c = cas.c
        sg := acquireSudog()
        sg.g = gp
        sg.isSelect = true
        // No stack splits between assigning elem and enqueuing
        // sg on gp.waiting where copystack can find it.
        sg.elem.set(cas.elem)
        sg.c.set(c)
        // Construct waiting list in lock order.
        *nextp = sg
        nextp = &sg.waitlink

        if casi < nsends {
            c.sendq.enqueue(sg)
        } else {
            c.recvq.enqueue(sg)
        }
    }

    // wait for someone to wake us up
    gp.param = nil
    gp.parkingOnChan.Store(true)
    gopark(selparkcommit, nil, waitReason, traceBlockSelect, 1)

If no case is immediately ready, the goroutine enqueues a sudog on every channel and parks itself. Key details:

• Each sudog has isSelect = true, signaling that waking this goroutine requires a CAS on g.selectDone.
• The sudogs are chained together via waitlink in lock order, so pass 3 can efficiently clean up.

Pass 3: Dequeue from losers and return winner

// src/runtime/select.go, lines 354-401

    sellock(scases, lockorder)

    gp.selectDone.Store(0)
    sg = (*sudog)(gp.param)
    gp.param = nil

    // pass 3 - dequeue from unsuccessful chans
    // otherwise they stack up on quiet channels
    // record the successful case, if any.
    casi = -1
    cas = nil
    caseSuccess = false
    sglist = gp.waiting
    // Clear all elem before unlinking from gp.waiting.
    for sg1 := gp.waiting; sg1 != nil; sg1 = sg1.waitlink {
        sg1.isSelect = false
        sg1.elem.set(nil)
        sg1.c.set(nil)
    }
    gp.waiting = nil

    for _, casei := range lockorder {
        k = &scases[casei]
        if sg == sglist {
            // sg has already been dequeued by the G that woke us up.
            casi = int(casei)
            cas = k
            caseSuccess = sglist.success
        } else {
            c = k.c
            if int(casei) < nsends {
                c.sendq.dequeueSudoG(sglist)
            } else {
                c.recvq.dequeueSudoG(sglist)
            }
        }
        sgnext = sglist.waitlink
        sglist.waitlink = nil
        releaseSudog(sglist)
        sglist = sgnext
    }

When the goroutine wakes up:

1. Re-lock all channels (in lock order).
2. Find which sudog caused the wakeup (gp.param).
3. Walk through all sudogs: the winning sudog has already been dequeued by the channel that woke us; dequeue the losers from their respective channels.
4. Release all sudogs back to the pool.

sellock and selunlock: Careful lock management

// src/runtime/select.go, lines 34-43

func sellock(scases []scase, lockorder []uint16) {
    var c *hchan
    for _, o := range lockorder {
        c0 := scases[o].c
        if c0 != c {
            c = c0
            lock(&c.lock)
        }
    }
}

The sellock function locks channels in sorted order, skipping duplicates (when the same channel appears in multiple cases). selunlock unlocks in reverse order, also skipping duplicates.

9. Summary: Channel Operation Decision Tree

Send (c <- x)

c == nil?  --> park forever
           |
           v
[lock channel]
           |
c.closed?  --> panic("send on closed channel")
           |
           v
receiver waiting?  --> copy directly to receiver, wake receiver
           |
           v
buffer space?  --> enqueue in buffer
           |
           v
[block: park until receiver arrives]

Receive (<-c)

c == nil?  --> park forever
           |
           v
[lock channel]
           |
c.closed && empty?  --> return zero value
           |
           v
sender waiting?  --> receive from sender/buffer, wake sender
           |
           v
buffer data?  --> dequeue from buffer
           |
           v
[block: park until sender arrives]

10. OS Concepts Illustrated

Discussion Questions

1. Why does chansend copy data directly to a waiting receiver instead of putting it in the buffer first? What performance benefit does this provide?

2. The non-blocking fast path in chansend reads c.closed and full(c) without holding the lock. Why is this safe? Under what conditions could this produce a “stale” answer, and why is a stale answer acceptable here?

3. In closechan, goroutines are collected into a list before being readied. Why can’t we call goready while holding the channel lock?

4. Why does select use randomization for the poll order but a deterministic sort for the lock order? What would go wrong if we used the same order for both?

5. Consider a select statement with 100 cases, all on the same channel. How does the lock ordering logic handle this? (Hint: look at sellock’s duplicate detection.)

Further Reading

• Source files:
  • /Users/tchajed/sw/go/src/runtime/chan.go – channel implementation
  • /Users/tchajed/sw/go/src/runtime/select.go – select implementation
  • /Users/tchajed/sw/go/src/runtime/runtime2.go – sudog struct (line 406)
• Hoare, C.A.R. “Communicating Sequential Processes.” Communications of the ACM, 1978.
• The Go Memory Model: https://go.dev/ref/mem

Module 7: Channels and Select (hchan, chansend/chanrecv, selectgo)

Comprehension Check

Question 1 (Code Reading)

Study the hchan struct from runtime/chan.go:

type hchan struct {
	qcount   uint           // total data in the queue
	dataqsiz uint           // size of the circular queue
	buf      unsafe.Pointer // points to an array of dataqsiz elements
	elemsize uint16
	closed   uint32
	elemtype *_type // element type
	sendx    uint   // send index
	recvx    uint   // receive index
	recvq    waitq  // list of recv waiters
	sendq    waitq  // list of send waiters

	lock mutex
}

(a) How can you determine from these fields whether a channel is unbuffered? (b) Why does the struct need both sendx/recvx and qcount if the buffer is a circular queue?

Question 2 (Code Reading)

Consider this fast-path check in chansend:

if !block && c.closed == 0 && full(c) {
	return false
}

The comment above this code explains that the reads of c.closed and full(c) are not atomic with respect to each other. Why is this safe for the non-blocking case? Under what condition would this be unsafe?

Question 3 (Short Answer)

Explain what happens during a channel send on an unbuffered channel when there is already a receiver waiting in recvq. Describe the data flow – where is the value copied?

Question 4 (What Would Happen If…)

What would happen if selectgo did not acquire channel locks in a consistent global order (the “lock order” phase)? Describe a specific deadlock scenario.

// Goroutine 1
select {
case ch_A <- 1:
case <-ch_B:
}

// Goroutine 2
select {
case ch_B <- 2:
case <-ch_A:
}

Question 5 (Short Answer)

Describe the three passes that selectgo performs and explain why each is necessary.

Question 6 (Short Answer)

Why does selectgo use a random permutation (pollorder) rather than checking cases in source-code order?

Module 10: File Systems, I/O, and the Network Poller (60 min)

Background: I/O Multiplexing and the Quest for Scalable Networking

I/O is the interface between a program and the outside world – disks, networks, terminals, pipes, and devices. The fundamental challenge of I/O is that it is slow relative to computation. A modern CPU can execute billions of instructions per second, but a network round-trip across the same data center takes tens of microseconds, and a spinning disk seek takes milliseconds – an eternity in CPU time. The simplest programming model, blocking I/O, has a thread call read() and wait until data arrives. This works well for a single connection, but it means one OS thread per concurrent I/O operation. Since OS threads are expensive (each consumes kernel memory for its stack and task struct, and context-switching between thousands of them bogs down the scheduler), blocking I/O hits a wall when you need to handle many connections simultaneously. Unix’s “everything-is-a-file” abstraction gives us a uniform interface (file descriptors, read, write, close), but it does not solve the problem of waiting on many file descriptors at once.

The history of I/O multiplexing on Unix and Linux is a story of successive attempts to solve this scaling problem. The original select() system call, introduced in 4.2BSD in 1983, let a process monitor multiple file descriptors for readiness. But select has painful limitations: it uses a fixed-size bitmask (typically limited to 1024 descriptors), and every call requires copying the entire set of descriptors into the kernel and scanning all of them regardless of how many are actually ready. poll() (SVR3, 1986) removed the fixed-size limit by using an array of pollfd structs, but it still requires a linear scan of all monitored descriptors on every call – O(n) in the total number of descriptors, not the number of ready descriptors. For a server with 10,000 connections where only a handful have data at any moment, this is wasteful. Linux’s epoll (2002) and BSD’s kqueue (2000, FreeBSD 4.1) finally solved this with stateful, kernel-maintained interest sets: you register descriptors once, and the kernel returns only the ready ones, giving O(1) behavior with respect to the total number of monitored descriptors. More recently, Linux’s io_uring (2019) pushes the boundary further with a shared-memory submission/completion ring that eliminates system call overhead entirely for batched I/O, enabling truly asynchronous operations for both network and disk I/O.

These kernel mechanisms were developed in response to a concrete engineering challenge articulated by Dan Kegel in his influential 1999 essay, “The C10K Problem”: how do you build a server that handles 10,000 concurrent connections on commodity hardware? At the time, this was considered very difficult. The thread-per-connection model consumed too much memory and CPU on context switches. Kegel surveyed the landscape of available approaches – non-blocking I/O with select/poll, signal-driven I/O (SIGIO), and the then-nascent asynchronous I/O interfaces – and argued that the OS needed better primitives. The C10K essay directly influenced the development of epoll and kqueue, and shaped the design of high-performance servers like Nginx (which replaced Apache’s thread-per-connection model with a single-threaded event loop). Today, the frontier has moved to the “C10M problem” – ten million concurrent connections – driven by epoll, io_uring, and kernel-bypass techniques like DPDK.

Different language runtimes have taken different approaches to giving programmers a usable API on top of these kernel primitives. Event-driven frameworks like libevent, libev, and libuv abstract over platform differences (epoll on Linux, kqueue on macOS/BSD, IOCP on Windows) and expose a callback-based API: register interest in an event, provide a function to call when it fires. Node.js builds on libuv and embraces this model fully – a single-threaded event loop runs JavaScript callbacks when I/O completes, with a thread pool for operations that lack async OS support (like file I/O on Linux). Java took a different path: the NIO (New I/O) package introduced Selector and non-blocking channels, but the callback-style programming was complex enough that frameworks like Netty arose to manage it. Java 21’s virtual threads (Project Loom) revisit the problem with lightweight user-space threads, much like goroutines. Rust’s async ecosystem (Tokio runtime, mio for portable event notification) uses async/await syntax compiled into state machines, offering zero-cost abstractions but requiring the programmer to reason about pinning, lifetimes, and Send/Sync bounds.

Go’s approach is distinctive: it hides all this complexity behind synchronous, blocking-style I/O calls. When a goroutine calls conn.Read(), it looks like a blocking call, and the goroutine does block – but the OS thread does not. Under the hood, the Go runtime sets file descriptors to non-blocking mode, attempts the I/O, and if the operation would block (EAGAIN), it parks the goroutine on the runtime’s integrated network poller (epoll on Linux, kqueue on macOS/BSD). The OS thread is freed to run other goroutines. When the kernel signals that data is ready, the scheduler wakes the parked goroutine and it retries the operation. The programmer writes simple sequential code; the runtime provides the efficiency of an event-driven architecture. This module traces exactly how this works, from the os.File type through the internal/poll package down to the platform-specific poller implementations and their integration with the scheduler.

1. The File Abstraction in Go (10 min)

The os.File Type

The public-facing file type in Go is os.File. It is defined as a pointer to a private file struct:

// [src/os/file_unix.go, lines 55-66](https://cs.opensource.google/go/go/+/refs/tags/go1.26.1:src/os/file_unix.go;l=55)
// file is the real representation of *File.
// The extra level of indirection ensures that no clients of os
// can overwrite this data, which could cause the finalizer
// to close the wrong file descriptor.
type file struct {
	pfd         poll.FD
	name        string
	dirinfo     atomic.Pointer[dirInfo] // nil unless directory being read
	nonblock    bool                    // whether we set nonblocking mode
	stdoutOrErr bool                    // whether this is stdout or stderr
	appendMode  bool                    // whether file is opened for appending
}

Key observations:

• The actual file descriptor is wrapped in a poll.FD (from internal/poll)
• The nonblock flag tracks whether Go put the descriptor into non-blocking mode
• There is no direct int file descriptor field – it lives inside pfd.Sysfd

The io/fs.FS Interface

Go 1.16 introduced a filesystem abstraction via the io/fs package:

// [src/io/fs/fs.go, lines 40-52](https://cs.opensource.google/go/go/+/refs/tags/go1.26.1:src/io/fs/fs.go;l=40)
type FS interface {
	// Open opens the named file.
	Open(name string) (File, error)
}

And the minimal File interface:

// [src/io/fs/fs.go, lines 95-99](https://cs.opensource.google/go/go/+/refs/tags/go1.26.1:src/io/fs/fs.go;l=95)
type File interface {
	Stat() (FileInfo, error)
	Read([]byte) (int, error)
	Close() error
}

This is a higher-level abstraction than os.File. The os.File type satisfies this interface and much more (writing, seeking, etc.). The fs.FS interface enables in-memory filesystems, embedded filesystems (embed.FS), and testing without touching the real filesystem.

The Layering

The I/O path through Go’s runtime has multiple layers:

User Code
    │
    ▼
os.File.Read()           ← high-level, platform-independent
    │
    ▼
internal/poll.FD.Read()  ← non-blocking I/O with EAGAIN handling
    │
    ▼
syscall.Read()           ← raw system call wrapper
    │
    ▼
kernel                   ← actual I/O

Each layer adds important functionality:

• os.File: name tracking, finalizers, safe concurrent access
• internal/poll.FD: non-blocking mode, poller integration, EAGAIN retry
• syscall: thin wrapper around the kernel interface

2. The internal/poll Package (10 min)

The FD Struct

The internal/poll.FD struct is the workhorse of Go’s I/O system. It wraps a raw file descriptor with synchronization and poller integration:

// [src/internal/poll/fd_unix.go, lines 17-48](https://cs.opensource.google/go/go/+/refs/tags/go1.26.1:src/internal/poll/fd_unix.go;l=17)
// FD is a file descriptor. The net and os packages use this type as a
// field of a larger type representing a network connection or OS file.
type FD struct {
	// Lock sysfd and serialize access to Read and Write methods.
	fdmu fdMutex

	// System file descriptor. Immutable until Close.
	Sysfd int

	// Platform dependent state of the file descriptor.
	SysFile

	// I/O poller.
	pd pollDesc

	// Semaphore signaled when file is closed.
	csema uint32

	// Non-zero if this file has been set to blocking mode.
	isBlocking uint32

	// Whether this is a streaming descriptor, as opposed to a
	// packet-based descriptor like a UDP socket. Immutable.
	IsStream bool

	// Whether a zero byte read indicates EOF. This is false for a
	// message based socket connection.
	ZeroReadIsEOF bool

	// Whether this is a file rather than a network socket.
	isFile bool
}

Key fields:

• fdmu: A specialized fdMutex that serializes concurrent reads (and separately, concurrent writes) while also managing reference counting for safe close.
• Sysfd: The actual OS file descriptor number.
• pd: A pollDesc that hooks into the runtime’s network poller.
• isBlocking: When set, the FD bypasses the poller and uses blocking I/O (tying up the OS thread).

FD Initialization

When a file is opened, the FD.Init method registers it with the poller:

// [src/internal/poll/fd_unix.go, lines 55-73](https://cs.opensource.google/go/go/+/refs/tags/go1.26.1:src/internal/poll/fd_unix.go;l=55)
func (fd *FD) Init(net string, pollable bool) error {
	fd.SysFile.init()

	if net == "file" {
		fd.isFile = true
	}
	if !pollable {
		fd.isBlocking = 1
		return nil
	}
	err := fd.pd.init(fd)
	if err != nil {
		// If we could not initialize the runtime poller,
		// assume we are using blocking mode.
		fd.isBlocking = 1
	}
	return err
}

The pd.init(fd) call ultimately invokes runtime.poll_runtime_pollOpen, which registers the file descriptor with the platform’s event notification mechanism (epoll/kqueue).

The Read Loop with EAGAIN Handling

The most important method is FD.Read. This is where non-blocking I/O meets goroutine scheduling:

// [src/internal/poll/fd_unix.go, lines 140-173](https://cs.opensource.google/go/go/+/refs/tags/go1.26.1:src/internal/poll/fd_unix.go;l=140)
// Read implements io.Reader.
func (fd *FD) Read(p []byte) (int, error) {
	if err := fd.readLock(); err != nil {
		return 0, err
	}
	defer fd.readUnlock()
	if len(p) == 0 {
		return 0, nil
	}
	if err := fd.pd.prepareRead(fd.isFile); err != nil {
		return 0, err
	}
	if fd.IsStream && len(p) > maxRW {
		p = p[:maxRW]
	}
	for {
		n, err := ignoringEINTRIO(syscall.Read, fd.Sysfd, p)
		if err != nil {
			n = 0
			if err == syscall.EAGAIN && fd.pd.pollable() {
				if err = fd.pd.waitRead(fd.isFile); err == nil {
					continue
				}
			}
		}
		err = fd.eofError(n, err)
		return n, err
	}
}

The critical loop:

1. Try the read via syscall.Read (non-blocking, since the fd is in non-blocking mode)
2. If the read returns EAGAIN (no data available):
   • Call fd.pd.waitRead() – this parks the goroutine on the poller
   • When data arrives, the poller wakes the goroutine
   • The goroutine retries the read (continue)
3. If the read succeeds or fails with a real error, return

This is the fundamental trick: blocking I/O semantics from the programmer’s perspective, non-blocking I/O under the hood. The goroutine blocks, but the OS thread does not.

3. Non-Blocking I/O Setup (5 min)

How Files Enter Non-Blocking Mode

When Go opens a file or socket, it puts the descriptor into non-blocking mode. This happens in the newFile function:

// [src/os/file_unix.go, lines 194-210](https://cs.opensource.google/go/go/+/refs/tags/go1.26.1:src/os/file_unix.go;l=194)
	clearNonBlock := false
	if pollable {
		if nonBlocking {
			if kind == kindSock {
				f.nonblock = true
			}
		} else if err := syscall.SetNonblock(fd, true); err == nil {
			f.nonblock = true
			clearNonBlock = true
		} else {
			pollable = false
		}
	}

Then the FD is registered with the poller:

// [src/os/file_unix.go, lines 219-223](https://cs.opensource.google/go/go/+/refs/tags/go1.26.1:src/os/file_unix.go;l=219)
	if pollErr := f.pfd.Init("file", pollable); pollErr != nil && clearNonBlock {
		if err := syscall.SetNonblock(fd, false); err == nil {
			f.nonblock = false
		}
	}

Platform Limitations

Not all file descriptors work with the poller. On macOS/BSD, regular files and directories cannot be polled with kqueue (they always report as ready). The newFile function checks for this:

// [src/os/file_unix.go, lines 165-192](https://cs.opensource.google/go/go/+/refs/tags/go1.26.1:src/os/file_unix.go;l=165)
	if kind == kindOpenFile {
		switch runtime.GOOS {
		case "darwin", "ios", "dragonfly", "freebsd", "netbsd", "openbsd":
			var st syscall.Stat_t
			err := ignoringEINTR(func() error {
				return syscall.Fstat(fd, &st)
			})
			typ := st.Mode & syscall.S_IFMT
			// Don't try to use kqueue with regular files on *BSDs.
			if err == nil && (typ == syscall.S_IFREG || typ == syscall.S_IFDIR) {
				pollable = false
			}
			// On Darwin, kqueue does not work properly with fifos
			if (runtime.GOOS == "darwin" || runtime.GOOS == "ios") && typ == syscall.S_IFIFO {
				pollable = false
			}
		}
	}

When a descriptor cannot use the poller, it falls back to blocking I/O, which will tie up an OS thread. This is acceptable for regular files (which complete quickly) but would be problematic for network sockets (which is why network sockets always use the poller).

4. The Network Poller: Core Design (10 min)

Architecture

The network poller lives in runtime/netpoll.go and provides a platform-independent interface. Each platform implements the actual polling mechanism:

// [src/runtime/netpoll.go, lines 16-37](https://cs.opensource.google/go/go/+/refs/tags/go1.26.1:src/runtime/netpoll.go;l=16)
// A particular implementation (epoll/kqueue/port/AIX/Windows)
// must define the following functions:
//
// func netpollinit()
//     Initialize the poller. Only called once.
//
// func netpollopen(fd uintptr, pd *pollDesc) int32
//     Arm edge-triggered notifications for fd.
//
// func netpollclose(fd uintptr) int32
//     Disable notifications for fd.
//
// func netpoll(delta int64) (gList, int32)
//     Poll the network. If delta < 0, block indefinitely.
//     If delta == 0, poll without blocking.
//     If delta > 0, block for up to delta nanoseconds.
//     Return a list of goroutines built by calling netpollready.
//
// func netpollBreak()
//     Wake up the network poller, assumed to be blocked in netpoll.

The pollDesc Struct

Each file descriptor registered with the poller gets a pollDesc:

// [src/runtime/netpoll.go, lines 75-115](https://cs.opensource.google/go/go/+/refs/tags/go1.26.1:src/runtime/netpoll.go;l=75)
type pollDesc struct {
	_     sys.NotInHeap
	link  *pollDesc      // in pollcache, protected by pollcache.lock
	fd    uintptr        // constant for pollDesc usage lifetime
	fdseq atomic.Uintptr // protects against stale pollDesc

	atomicInfo atomic.Uint32 // atomic pollInfo

	// rg, wg are accessed atomically and hold g pointers.
	rg atomic.Uintptr // pdReady, pdWait, G waiting for read or pdNil
	wg atomic.Uintptr // pdReady, pdWait, G waiting for write or pdNil

	lock    mutex // protects the following fields
	closing bool
	rrun    bool      // whether rt is running
	wrun    bool      // whether wt is running
	user    uint32    // user settable cookie
	rseq    uintptr   // protects from stale read timers
	rt      timer     // read deadline timer
	rd      int64     // read deadline
	wseq    uintptr   // protects from stale write timers
	wt      timer     // write deadline timer
	wd      int64     // write deadline
	self    *pollDesc // storage for indirect interface
}

The pollDesc contains two binary semaphores (rg and wg) for parking reader and writer goroutines respectively.

Poll States

The semaphores (rg, wg) can be in one of four states:

// [src/runtime/netpoll.go, lines 51-68](https://cs.opensource.google/go/go/+/refs/tags/go1.26.1:src/runtime/netpoll.go;l=51)
// pollDesc contains 2 binary semaphores, rg and wg, to park reader and writer
// goroutines respectively. The semaphore can be in the following states:
//
//	pdReady - io readiness notification is pending;
//	          a goroutine consumes the notification by changing the state to pdNil.
//	pdWait - a goroutine prepares to park on the semaphore, but not yet parked;
//	         the goroutine commits to park by changing the state to G pointer,
//	         or, alternatively, concurrent io notification changes the state to pdReady,
//	         or, alternatively, concurrent timeout/close changes the state to pdNil.
//	G pointer - the goroutine is blocked on the semaphore;
//	            io notification or timeout/close changes the state to pdReady or pdNil
//	            and unparks the goroutine.
//	pdNil - none of the above.
const (
	pdNil   uintptr = 0
	pdReady uintptr = 1
	pdWait  uintptr = 2
)

State transition diagram:

                    ┌─────────────┐
                    │   pdNil     │ (initial / idle)
                    └──────┬──────┘
                           │ goroutine wants to read
                           ▼
                    ┌─────────────┐
                    │   pdWait    │ (preparing to park)
                    └──────┬──────┘
                           │ goroutine commits to park
                           ▼
                    ┌─────────────┐
         I/O ready │  G pointer  │ goroutine is blocked
              ─────┤             ├───── timeout/close
              │    └─────────────┘      │
              ▼                         ▼
       ┌─────────────┐          ┌─────────────┐
       │   pdReady   │          │    pdNil     │
       └─────────────┘          └─────────────┘
              │ goroutine wakes and
              │ consumes notification
              ▼
       ┌─────────────┐
       │    pdNil     │
       └─────────────┘

netpollready: Waking Goroutines

When the platform poller detects that a file descriptor is ready, it calls netpollready:

// [src/runtime/netpoll.go, lines 483-510](https://cs.opensource.google/go/go/+/refs/tags/go1.26.1:src/runtime/netpoll.go;l=483)
// netpollready is called by the platform-specific netpoll function.
// It declares that the fd associated with pd is ready for I/O.
// The toRun argument is used to build a list of goroutines to return
// from netpoll. The mode argument is 'r', 'w', or 'r'+'w' to indicate
// whether the fd is ready for reading or writing or both.
func netpollready(toRun *gList, pd *pollDesc, mode int32) int32 {
	delta := int32(0)
	var rg, wg *g
	if mode == 'r' || mode == 'r'+'w' {
		rg = netpollunblock(pd, 'r', true, &delta)
	}
	if mode == 'w' || mode == 'r'+'w' {
		wg = netpollunblock(pd, 'w', true, &delta)
	}
	if rg != nil {
		toRun.push(rg)
	}
	if wg != nil {
		toRun.push(wg)
	}
	return delta
}

This function unblocks the goroutine(s) waiting on the given pollDesc and adds them to the toRun list. The caller (netpoll) returns this list to the scheduler, which puts these goroutines back on run queues.

5. Platform Implementations (10 min)

epoll on Linux

The Linux implementation uses epoll in edge-triggered mode:

// [src/runtime/netpoll_epoll.go, lines 15-19](https://cs.opensource.google/go/go/+/refs/tags/go1.26.1:src/runtime/netpoll_epoll.go;l=15)
var (
	epfd           int32         = -1 // epoll descriptor
	netpollEventFd uintptr            // eventfd for netpollBreak
	netpollWakeSig atomic.Uint32      // used to avoid duplicate calls of netpollBreak
)

Initialization creates an epoll instance and an eventfd for waking the poller:

// [src/runtime/netpoll_epoll.go, lines 21-43](https://cs.opensource.google/go/go/+/refs/tags/go1.26.1:src/runtime/netpoll_epoll.go;l=21)
func netpollinit() {
	var errno uintptr
	epfd, errno = linux.EpollCreate1(linux.EPOLL_CLOEXEC)
	if errno != 0 {
		println("runtime: epollcreate failed with", errno)
		throw("runtime: netpollinit failed")
	}
	efd, errno := linux.Eventfd(0, linux.EFD_CLOEXEC|linux.EFD_NONBLOCK)
	// ... register eventfd with epoll for wakeups ...
	netpollEventFd = uintptr(efd)
}

Registering a file descriptor uses edge-triggered mode (EPOLLET):

// [src/runtime/netpoll_epoll.go, lines 49-55](https://cs.opensource.google/go/go/+/refs/tags/go1.26.1:src/runtime/netpoll_epoll.go;l=49)
func netpollopen(fd uintptr, pd *pollDesc) uintptr {
	var ev linux.EpollEvent
	ev.Events = linux.EPOLLIN | linux.EPOLLOUT | linux.EPOLLRDHUP | linux.EPOLLET
	tp := taggedPointerPack(unsafe.Pointer(pd), pd.fdseq.Load())
	*(*taggedPointer)(unsafe.Pointer(&ev.Data)) = tp
	return linux.EpollCtl(epfd, linux.EPOLL_CTL_ADD, int32(fd), &ev)
}

The flags EPOLLIN | EPOLLOUT | EPOLLRDHUP | EPOLLET mean:

• Monitor for both read and write readiness
• Detect remote hangup (EPOLLRDHUP)
• Edge-triggered (EPOLLET): only notify when readiness changes (e.g., new data arrives), not continuously while data remains available. This contrasts with level-triggered mode (the default) where the kernel repeatedly reports readiness as long as the condition holds. Edge-triggered is more efficient for Go’s model because each fd has exactly one goroutine waiting, so there is no risk of missing events.

The main polling function calls epoll_wait:

// [src/runtime/netpoll_epoll.go, lines 99-176](https://cs.opensource.google/go/go/+/refs/tags/go1.26.1:src/runtime/netpoll_epoll.go;l=99)
func netpoll(delay int64) (gList, int32) {
	// ... convert delay to milliseconds ...
	var events [128]linux.EpollEvent
retry:
	n, errno := linux.EpollWait(epfd, events[:], int32(len(events)), waitms)
	// ... error handling ...
	var toRun gList
	delta := int32(0)
	for i := int32(0); i < n; i++ {
		ev := events[i]
		// ... skip eventfd wakeup events ...
		var mode int32
		if ev.Events&(linux.EPOLLIN|linux.EPOLLRDHUP|linux.EPOLLHUP|linux.EPOLLERR) != 0 {
			mode += 'r'
		}
		if ev.Events&(linux.EPOLLOUT|linux.EPOLLHUP|linux.EPOLLERR) != 0 {
			mode += 'w'
		}
		if mode != 0 {
			tp := *(*taggedPointer)(unsafe.Pointer(&ev.Data))
			pd := (*pollDesc)(tp.pointer())
			tag := tp.tag()
			if pd.fdseq.Load() == tag {
				pd.setEventErr(ev.Events == linux.EPOLLERR, tag)
				delta += netpollready(&toRun, pd, mode)
			}
		}
	}
	return toRun, delta
}

The tagged pointer stored in the epoll event data allows the runtime to map events back to their pollDesc. The fdseq check guards against stale events for file descriptors that have been closed and reused.

kqueue on macOS/BSD

The macOS/BSD implementation uses kqueue with EV_CLEAR (the kqueue equivalent of edge-triggered mode):

// [src/runtime/netpoll_kqueue.go, lines 32-60](https://cs.opensource.google/go/go/+/refs/tags/go1.26.1:src/runtime/netpoll_kqueue.go;l=32)
func netpollopen(fd uintptr, pd *pollDesc) int32 {
	// Arm both EVFILT_READ and EVFILT_WRITE in edge-triggered mode (EV_CLEAR)
	var ev [2]keventt
	*(*uintptr)(unsafe.Pointer(&ev[0].ident)) = fd
	ev[0].filter = _EVFILT_READ
	ev[0].flags = _EV_ADD | _EV_CLEAR
	ev[0].fflags = 0
	ev[0].data = 0
	// ... store tagged pointer in ev[0].udata ...
	ev[1] = ev[0]
	ev[1].filter = _EVFILT_WRITE
	n := kevent(kq, &ev[0], 2, nil, 0, nil)
	if n < 0 {
		return -n
	}
	return 0
}

Key differences from epoll:

• kqueue uses two separate filters (EVFILT_READ, EVFILT_WRITE) instead of a bitmask
• EV_CLEAR is the kqueue equivalent of EPOLLET (edge-triggered)
• No need to explicitly unregister – closing the fd removes all kevents
• kqueue uses kevent() for both registration and waiting

The polling function is structurally similar:

// [src/runtime/netpoll_kqueue.go, lines 90-183](https://cs.opensource.google/go/go/+/refs/tags/go1.26.1:src/runtime/netpoll_kqueue.go;l=90)
func netpoll(delay int64) (gList, int32) {
	// ...
	var events [64]keventt
retry:
	n := kevent(kq, nil, 0, &events[0], int32(len(events)), tp)
	// ...
	for i := 0; i < int(n); i++ {
		ev := &events[i]
		// ... skip wakeup events ...
		var mode int32
		switch ev.filter {
		case _EVFILT_READ:
			mode += 'r'
			if ev.flags&_EV_EOF != 0 {
				mode += 'w'  // closed pipe: wake writers too
			}
		case _EVFILT_WRITE:
			mode += 'w'
		}
		if mode != 0 {
			// ... extract pollDesc from tagged pointer ...
			delta += netpollready(&toRun, pd, mode)
		}
	}
	return toRun, delta
}

Waking the Poller: netpollBreak

Both implementations provide a way to wake the poller from another thread:

Linux uses an eventfd:

// [src/runtime/netpoll_epoll.go, lines 67-89](https://cs.opensource.google/go/go/+/refs/tags/go1.26.1:src/runtime/netpoll_epoll.go;l=67)
func netpollBreak() {
	if !netpollWakeSig.CompareAndSwap(0, 1) {
		return  // already being woken
	}
	var one uint64 = 1
	oneSize := int32(unsafe.Sizeof(one))
	for {
		n := write(netpollEventFd, noescape(unsafe.Pointer(&one)), oneSize)
		if n == oneSize {
			break
		}
		// ... handle EINTR, EAGAIN ...
	}
}

macOS/BSD uses a platform-specific wakeup mechanism (wakeNetpoll):

// [src/runtime/netpoll_kqueue.go, lines 73-80](https://cs.opensource.google/go/go/+/refs/tags/go1.26.1:src/runtime/netpoll_kqueue.go;l=73)
func netpollBreak() {
	if !netpollWakeSig.CompareAndSwap(0, 1) {
		return
	}
	wakeNetpoll(kq)
}

The CAS on netpollWakeSig prevents redundant wakeup writes when multiple goroutines try to break the poller simultaneously.

6. Integration with the Scheduler (10 min)

findRunnable() and the Poller

The scheduler’s findRunnable() function (in proc.go) checks the network poller as part of its work-stealing loop. When there is no other work to do, the scheduler thread blocks in the poller:

// [src/runtime/proc.go, lines 3731-3754](https://cs.opensource.google/go/go/+/refs/tags/go1.26.1:src/runtime/proc.go;l=3731)
	// Poll network until next timer.
	if netpollinited() && (netpollAnyWaiters() || pollUntil != 0) && sched.lastpoll.Swap(0) != 0 {
		sched.pollUntil.Store(pollUntil)
		// ...
		delay := int64(-1)
		if pollUntil != 0 {
			if now == 0 {
				now = nanotime()
			}
			delay = pollUntil - now
			if delay < 0 {
				delay = 0
			}
		}
		// ...
		list, delta := netpoll(delay) // block until new work is available

The netpoll(delay) call blocks the OS thread in epoll_wait / kevent until either:

• A file descriptor becomes ready (I/O event)
• The delay expires (timer event)
• netpollBreak() is called (new work available elsewhere)

The returned gList contains goroutines whose I/O is ready. The scheduler injects them back into run queues.

The Complete I/O Flow

Here is the full lifecycle of a goroutine performing a network read:

1. Goroutine calls conn.Read(buf)
       │
       ▼
2. os.File.Read → internal/poll.FD.Read
       │
       ▼
3. syscall.Read(fd, buf)
       │
       ├── Data available → return n, nil  (fast path)
       │
       └── EAGAIN (no data)
               │
               ▼
4. fd.pd.waitRead() → runtime.poll_runtime_pollWait
       │
       ▼
5. netpollblock():
   - Set rg = pdWait
   - CAS rg from pdWait to G pointer (commit to park)
   - gopark() — goroutine is descheduled
       │
       ▼
6. OS thread is FREE to run other goroutines
       │
       ▼
7. Eventually, data arrives on the socket
       │
       ▼
8. Another thread calls netpoll() (in findRunnable or sysmon)
   - epoll_wait/kevent returns the ready fd
   - netpollready() extracts the parked G from pd.rg
   - G is added to the run queue
       │
       ▼
9. Scheduler picks up the goroutine
       │
       ▼
10. Goroutine resumes in FD.Read loop, retries syscall.Read
    - This time data is available → returns to caller

Why This Matters

This design achieves the best of both worlds:

The programmer writes straightforward sequential code:

for {
    n, err := conn.Read(buf)
    if err != nil {
        break
    }
    process(buf[:n])
}

But under the hood, the goroutine parks when no data is available, freeing the OS thread to run other goroutines. When data arrives, the poller wakes the goroutine and it seamlessly resumes.

7. The Big Picture: How All the Pieces Fit Together (5 min)

┌─────────────────────────────────────────────────────────────┐
│                     User Code                                │
│   conn.Read(buf)    file.Write(data)    http.Get(url)       │
└────────────┬────────────────┬───────────────────┬───────────┘
             │                │                   │
             ▼                ▼                   ▼
┌─────────────────────────────────────────────────────────────┐
│                    os / net packages                         │
│   os.File              net.Conn           http.Client       │
└────────────┬────────────────┬───────────────────┬───────────┘
             │                │                   │
             ▼                ▼                   ▼
┌─────────────────────────────────────────────────────────────┐
│                  internal/poll.FD                            │
│   Non-blocking I/O    EAGAIN retry loop    fdMutex          │
└────────────┬──────────────────────────────────┬─────────────┘
             │                                  │
    syscall.Read/Write                 fd.pd.waitRead/waitWrite
             │                                  │
             ▼                                  ▼
┌────────────────────┐          ┌──────────────────────────────┐
│   Linux kernel     │          │   runtime netpoll             │
│   (actual I/O)     │          │                               │
└────────────────────┘          │   pollDesc (per-fd state)     │
                                │   rg/wg semaphores            │
                                │                               │
                                │   ┌─────────┐  ┌──────────┐  │
                                │   │ epoll   │  │ kqueue   │  │
                                │   │ (Linux) │  │ (macOS)  │  │
                                │   └─────────┘  └──────────┘  │
                                └──────────────┬───────────────┘
                                               │
                                               ▼
                                ┌──────────────────────────────┐
                                │   Scheduler (findRunnable)    │
                                │                               │
                                │   netpoll() returns gList     │
                                │   → goroutines go to run queue│
                                │   → goroutines resume I/O     │
                                └──────────────────────────────┘

Connections to Previous Modules

• Module 4 (Scheduler): findRunnable() integrates poller checks into the scheduling loop. The poller is checked both in the fast path (non-blocking poll) and as a last resort (blocking poll when no other work exists).

• Module 6 (Synchronization): The fdMutex in poll.FD is a specialized lock that supports concurrent readers/writers while managing close semantics. The pollDesc.rg/wg semaphores use atomic CAS operations.

• Module 9 (Stacks): When a goroutine parks on I/O, its stack remains allocated but can be shrunk by the GC if it sits idle long enough.

Summary

Key Takeaways

1. Go files are opened in non-blocking mode so that EAGAIN can be handled by parking the goroutine instead of blocking the OS thread.

2. The EAGAIN retry loop in poll.FD.Read is the core mechanism: try the syscall, if EAGAIN then park on the poller, wake up and retry.

3. pollDesc semaphores (rg/wg) track whether a goroutine is waiting for read/write readiness on each file descriptor.

4. Edge-triggered polling (EPOLLET / EV_CLEAR) avoids redundant wakeups – the poller only notifies on state transitions.

5. The scheduler calls netpoll() to discover goroutines whose I/O is ready, seamlessly integrating I/O events with goroutine scheduling.

6. The result: programmers write simple blocking I/O code, but the runtime multiplexes thousands of connections across a small number of OS threads.

Discussion Questions

1. What happens when you pass an os.File to C code via cgo? Can the poller still manage it, or does the fd revert to blocking mode?

2. Regular files are always reported as ready by epoll on Linux (making epoll effectively useless for them), and regular files on macOS do not support kqueue at all. Given this, how does Go handle os.File.Read on a regular file? (Hint: check the pollable logic in newFile.)

3. Why does the poller use edge-triggered mode instead of level-triggered? What would go wrong with level-triggered notifications in Go’s model?

4. The netpollBreak function uses a CAS to avoid duplicate wakeups. Why is this optimization important? What would happen without it?

Module 10: File Systems, I/O, and the Network Poller

Comprehension Check

Question 1 (Code Reading)

Study this excerpt from runtime/netpoll.go:

// Integrated network poller (platform-independent part).
// A particular implementation (epoll/kqueue/port/AIX/Windows)
// must define the following functions:
//
// func netpollinit()
//     Initialize the poller. Only called once.
//
// func netpollopen(fd uintptr, pd *pollDesc) int32
//     Arm edge-triggered notifications for fd.
//
// func netpollclose(fd uintptr) int32
//     Disable notifications for fd.
//
// func netpoll(delta int64) (gList, int32)
//     Poll the network. If delta < 0, block indefinitely.
//     If delta == 0, poll without blocking.
//     If delta > 0, block for up to delta nanoseconds.
//     Return a list of goroutines built by calling netpollready.

Why does the network poller use edge-triggered rather than level-triggered notifications? What would happen if it used level-triggered mode?

Question 2 (Short Answer)

Describe the path a conn.Read() call takes from user code down to the kernel. Include the role of os.File, poll.FD, pollDesc, and the network poller.

Question 3 (True/False with Explanation)

True or False: File I/O (reading from disk) uses the same network poller mechanism as network I/O in Go.

Question 4 (What Would Happen If…)

The pollDesc struct uses a state machine for its rg (read goroutine) field with states: pdReady, pdWait, nil, and a goroutine pointer. What would happen if the implementation skipped the pdWait intermediate state and went directly from nil to a goroutine pointer?

Normal flow:
1. goroutine sets rg = pdWait    (preparing to park)
2. goroutine sets rg = G pointer (committed to parking)
3. netpoll sets rg = pdReady     (I/O ready, wakes goroutine)

Question 5 (Short Answer)

Explain the role of sysmon in network polling. Why can’t the scheduler rely solely on findRunnable calling netpoll?

Question 6 (Code Reading)

Consider this comment from runtime/netpoll.go describing pollDesc states:

// pollDesc contains 2 binary semaphores, rg and wg, to park reader and writer
// goroutines respectively. The semaphore can be in the following states:
//
//   pdReady - io readiness notification is pending;
//             a goroutine consumes the notification by changing the state to pdNil.
//   pdWait  - a goroutine prepares to park on the semaphore, but not yet parked;
//             the goroutine commits to park by changing the state to G pointer,
//             or, alternatively, concurrent io notification changes the state to pdReady,
//             or, alternatively, concurrent timeout/close changes the state to pdNil.
//   G pointer - the goroutine is blocked on the semaphore;
//               io notification or timeout/close changes the state to pdReady or pdNil
//               respectively and unparks the goroutine.

Draw the state transition diagram for rg. What transitions are valid, and who performs each transition?

                    goroutine                    goroutine
    pdNil -----(prepare to wait)-----> pdWait ------(commit)-----> G pointer
      ^                                  |                            |
      |                                  |                            |
      |              netpollready        |  timeout/close             | netpollready
      |           (sets pdReady)         |  (sets pdNil)              | (sets pdReady,
      |                                  |                            |  unparks G)
      |                                  v                            v
      +---------(goroutine consumes)--- pdReady <-----(io ready)------+

Module 2: System Calls

Duration: 60 minutes

Background: The User-Kernel Boundary

System calls are the fundamental interface between user programs and the operating system kernel. Every time a program opens a file, allocates memory, sends a network packet, or creates a new process, it must cross the boundary from user mode into kernel mode through a system call. This boundary is not merely a software convention – it is enforced by the CPU’s hardware privilege levels. The kernel runs in a privileged mode (ring 0 on x86, EL1 on ARM64) with full access to hardware and memory, while user programs run in a restricted mode (ring 3 / EL0) where attempts to execute privileged instructions or access protected memory trigger a processor fault. System calls are the controlled gateway through this hardware barrier.

The mechanism for crossing this boundary has evolved considerably over the decades. Early systems like the PDP-11 used a TRAP instruction to transfer control to the kernel, and this basic model – a special instruction that atomically switches privilege levels and jumps to a kernel-defined entry point – has persisted in every architecture since. On x86, the history is particularly convoluted: the original 8086 used INT 0x80 (a software interrupt), which required the CPU to look up an interrupt descriptor table entry, save state to the stack, and perform a privilege level change – a process that could take hundreds of CPU cycles on modern processors due to pipeline flushes and serialization. Intel introduced SYSENTER/SYSEXIT with the Pentium II, and AMD introduced SYSCALL/SYSRET with the K6-2, both designed to reduce this overhead by using pre-configured Model-Specific Registers (MSRs) instead of memory-based descriptor tables. On modern x86-64 Linux, SYSCALL is the standard mechanism, bringing the transition cost down to roughly 50-100 nanoseconds – fast, but far from free when called millions of times per second.

Linux today defines over 450 system calls on x86-64 (the exact count grows with each kernel release as new calls are added and old ones are rarely removed, since the syscall ABI is a stability guarantee). These calls fall into several broad categories: process management (fork, clone, execve, wait4, exit), file I/O (open, read, write, close, lseek), memory management (mmap, munmap, brk, madvise), networking (socket, bind, listen, accept, sendto, recvfrom), and inter-process communication (pipe, shmget, semop). Over the years, many syscalls have been superseded by improved versions – open by openat, epoll_wait by epoll_pwait2, clone by clone3 – reflecting the kernel community’s preference for extending the interface with new calls rather than modifying the semantics of existing ones. Newer additions like io_uring_setup (added in Linux 5.1) and landlock_add_rule (Linux 5.13) show that the syscall table continues to grow as the kernel gains capabilities.

The cost of crossing the user-kernel boundary has driven significant innovation in reducing or avoiding syscalls altogether. The vDSO (virtual Dynamic Shared Object) is a kernel-provided shared library mapped into every process’s address space, allowing frequently-called operations like clock_gettime and gettimeofday to execute entirely in user space by reading kernel-maintained data pages – turning a ~100ns syscall into a ~20ns function call. At the other extreme, frameworks like DPDK (Data Plane Development Kit) bypass the kernel entirely for network I/O, mapping NIC hardware directly into user space to achieve line-rate packet processing. Linux’s io_uring, introduced in kernel 5.1, takes a middle path: it uses shared memory ring buffers between user space and the kernel to batch and submit I/O operations asynchronously, dramatically reducing the per-operation syscall overhead for I/O-heavy workloads. These approaches highlight a recurring tension in OS design: syscalls provide safety and abstraction, but that safety comes at a performance cost that matters in high-throughput systems.

This module examines how Go navigates this landscape. Go makes an unusual choice among programming languages: on Linux, the runtime bypasses libc entirely and issues system calls directly via assembly stubs, relying on the kernel’s stable syscall ABI. Most other languages – and even Go itself on macOS and Windows – call through the C library, which serves as a portability layer and insulates programs from kernel interface changes. Go’s approach on Linux yields tighter integration between the runtime scheduler and syscall handling (as we will see with entersyscall/exitsyscall), but it also means the runtime must handle the raw kernel interface, including error conventions, signal safety, and architectural differences. We will trace this path from the raw SYSCALL instruction in assembly, through the syscall package’s Go wrappers, to the runtime’s scheduler integration that keeps goroutines running efficiently even when some are blocked in the kernel.

1. What Are System Calls? (10 min)

The User/Kernel Boundary

A modern operating system divides execution into two privilege levels:

• User mode: where application code runs. It can execute normal instructions but cannot directly access hardware, other processes’ memory, or privileged CPU features.
• Kernel mode: where the OS kernel runs. It has unrestricted access to hardware, memory, and privileged instructions.

A system call (syscall) is the mechanism by which user-mode code requests services from the kernel. On x86-64 Linux, the transition happens via the SYSCALL instruction, which:

1. Saves the user-mode instruction pointer (RIP) and flags (RFLAGS)
2. Loads the kernel-mode instruction pointer and stack pointer from MSRs (Model-Specific Registers — special CPU registers that the kernel configures during boot to specify where to jump when a syscall occurs)
3. Switches to ring 0 (kernel privilege level — x86 CPUs define privilege “rings,” where ring 0 is the most privileged level used by the kernel, and ring 3 is the least privileged level where user code runs)
4. Jumps to the kernel’s syscall entry point

The kernel looks at the value in RAX (the syscall number), dispatches to the appropriate handler, and returns to user mode via SYSRET.

The x86-64 Linux Syscall Convention

On error, RAX contains a negative errno value (e.g., -ENOENT = -2).

2. How Go Makes System Calls on Linux (15 min)

The Go runtime implements system calls directly in assembly, bypassing libc entirely on Linux. The wrappers live in src/runtime/sys_linux_amd64.s.

Syscall Number Definitions

The file begins by defining the syscall numbers as constants, matching the Linux kernel’s numbering:

src/runtime/sys_linux_amd64.s, lines 16-49:

// src/runtime/sys_linux_amd64.s, lines 16-49

#define SYS_read                0
#define SYS_write               1
#define SYS_close               3
#define SYS_mmap                9
#define SYS_munmap              11
#define SYS_brk                 12
#define SYS_rt_sigaction        13
#define SYS_rt_sigprocmask      14
#define SYS_rt_sigreturn        15
#define SYS_sched_yield         24
#define SYS_madvise             28
#define SYS_nanosleep           35
#define SYS_getpid              39
#define SYS_clone               56
#define SYS_exit                60
#define SYS_kill                62
#define SYS_sigaltstack         131
#define SYS_futex               202
#define SYS_clock_gettime       228
#define SYS_exit_group          231
#define SYS_openat              257
#define SYS_pipe2               293

These numbers are stable ABI – the Linux kernel guarantees they won’t change. This is why Go can bypass libc on Linux: the syscall interface is the stable boundary, not the C library.

exit: The Simplest Syscall

src/runtime/sys_linux_amd64.s, lines 51-55:

// src/runtime/sys_linux_amd64.s, lines 51-55

TEXT runtime·exit(SB),NOSPLIT,$0-4
    MOVL    code+0(FP), DI
    MOVL    $SYS_exit_group, AX
    SYSCALL
    RET

Line by line:

1. TEXT runtime·exit(SB),NOSPLIT,$0-4 – declares the function. NOSPLIT means it won’t grow the stack (important because we’re about to leave Go’s world). $0-4 means 0 bytes of local frame, 4 bytes of arguments.
2. MOVL code+0(FP), DI – loads the Go function argument code into DI (first argument register per Linux ABI).
3. MOVL $SYS_exit_group, AX – loads syscall number 231 (exit_group, which kills all threads in the process) into AX.
4. SYSCALL – traps into the kernel.
5. RET – never reached, since exit_group doesn’t return.

Note: Go uses exit_group (not exit) because a Go process has multiple OS threads. exit would only terminate one thread; exit_group terminates the entire process.

read, write: I/O Syscalls

src/runtime/sys_linux_amd64.s, lines 93-109:

// src/runtime/sys_linux_amd64.s, lines 93-109

TEXT runtime·write1(SB),NOSPLIT,$0-28
    MOVQ    fd+0(FP), DI
    MOVQ    p+8(FP), SI
    MOVL    n+16(FP), DX
    MOVL    $SYS_write, AX
    SYSCALL
    MOVL    AX, ret+24(FP)
    RET

TEXT runtime·read(SB),NOSPLIT,$0-28
    MOVL    fd+0(FP), DI
    MOVQ    p+8(FP), SI
    MOVL    n+16(FP), DX
    MOVL    $SYS_read, AX
    SYSCALL
    MOVL    AX, ret+24(FP)
    RET

The pattern is always the same:

1. Load Go function arguments into the Linux ABI registers (DI, SI, DX, R10, R8, R9)
2. Load the syscall number into AX
3. Execute SYSCALL
4. Store the return value from AX back into the Go return slot on the stack

open: Error Handling Pattern

src/runtime/sys_linux_amd64.s, lines 69-81:

// src/runtime/sys_linux_amd64.s, lines 69-81

TEXT runtime·open(SB),NOSPLIT,$0-20
    // This uses openat instead of open, because Android O blocks open.
    MOVL    $AT_FDCWD, DI // AT_FDCWD, so this acts like open
    MOVQ    name+0(FP), SI
    MOVL    mode+8(FP), DX
    MOVL    perm+12(FP), R10
    MOVL    $SYS_openat, AX
    SYSCALL
    CMPQ    AX, $0xfffffffffffff001
    JLS     2(PC)
    MOVL    $-1, AX
    MOVL    AX, ret+16(FP)
    RET

The error-checking pattern at lines 77-79 is interesting:

• CMPQ AX, $0xfffffffffffff001 checks if the return value is in the error range. On Linux, syscall returns in the range [-4095, -1] indicate errors (negative errno).
• 0xfffffffffffff001 is -4095 in unsigned 64-bit representation.
• JLS 2(PC) – if the result is below (unsigned) this threshold, it’s a success; skip the error fixup.
• On error, the function returns -1 rather than the raw negative errno.

Also note that Go uses openat with AT_FDCWD instead of open, because Android O blocked the open syscall in its seccomp filter.

3. macOS: libc Trampolines (10 min)

On macOS (Darwin), Go cannot make syscalls directly. Apple does not guarantee a stable syscall ABI – the syscall numbers and conventions can change between macOS versions. Instead, Go must call through libc.

Since macOS on Apple Silicon uses ARM64, the assembly syntax below differs from the x86-64 we saw above. Key differences: arguments go in registers R0-R7, BL is the branch-and-link (call) instruction, and RSP is the stack pointer.

The trampoline pattern is visible in src/runtime/sys_darwin_arm64.s:

src/runtime/sys_darwin_arm64.s, lines 21-29:

// src/runtime/sys_darwin_arm64.s, lines 21-29

TEXT runtime·open_trampoline(SB),NOSPLIT,$0
    SUB     $16, RSP
    MOVW    8(R0), R1       // arg 2 flags
    MOVW    12(R0), R2      // arg 3 mode
    MOVW    R2, (RSP)       // arg 3 is variadic, pass on stack
    MOVD    0(R0), R0       // arg 1 pathname
    BL      libc_open(SB)
    ADD     $16, RSP
    RET

Key differences from Linux:

1. Indirection through libc: BL libc_open(SB) calls the C library’s open() function, which internally makes the actual syscall. The runtime never issues the SVC instruction directly.
2. Argument marshalling: Arguments arrive packed in a struct pointed to by R0. The trampoline unpacks them into the ARM64 calling convention registers (R0, R1, R2, etc.).
3. Variadic argument handling: open() in C is variadic (the mode argument is only present with O_CREAT). On ARM64, variadic arguments go on the stack, not in registers – hence the MOVW R2, (RSP).

write and read trampolines with error handling

src/runtime/sys_darwin_arm64.s, lines 36-62:

// src/runtime/sys_darwin_arm64.s, lines 36-62

TEXT runtime·write_trampoline(SB),NOSPLIT,$0
    MOVD    8(R0), R1       // arg 2 buf
    MOVW    16(R0), R2      // arg 3 count
    MOVW    0(R0), R0       // arg 1 fd
    BL      libc_write(SB)
    MOVD    $-1, R1
    CMP     R0, R1
    BNE     noerr
    BL      libc_error(SB)
    MOVW    (R0), R0
    NEG     R0, R0          // caller expects negative errno value
noerr:
    RET

TEXT runtime·read_trampoline(SB),NOSPLIT,$0
    MOVD    8(R0), R1       // arg 2 buf
    MOVW    16(R0), R2      // arg 3 count
    MOVW    0(R0), R0       // arg 1 fd
    BL      libc_read(SB)
    MOVD    $-1, R1
    CMP     R0, R1
    BNE     noerr
    BL      libc_error(SB)
    MOVW    (R0), R0
    NEG     R0, R0          // caller expects negative errno value
noerr:
    RET

On macOS, libc functions return -1 on error and set errno (a thread-local variable). The trampoline must:

1. Check if the return is -1
2. If so, call libc_error(SB) to get a pointer to the thread-local errno
3. Load and negate the errno value (the Go runtime expects negative errno, same as Linux kernel convention)

exit trampoline

src/runtime/sys_darwin_arm64.s, lines 72-77:

// src/runtime/sys_darwin_arm64.s, lines 72-77

TEXT runtime·exit_trampoline(SB),NOSPLIT|NOFRAME,$0
    MOVW    0(R0), R0
    BL      libc_exit(SB)
    MOVD    $1234, R0
    MOVD    $1002, R1
    MOVD    R0, (R1)        // fail hard

The crash at the end (writing to address 1002) is a safety net – if libc_exit somehow returns, the process crashes immediately rather than continuing in an undefined state.

Linux vs. macOS: Summary

4. The syscall Package: Go Wrappers (10 min)

User code doesn’t call the runtime’s raw assembly directly. Instead, the syscall package provides Go-level wrappers.

The Syscall/RawSyscall split

From src/syscall/syscall_linux.go (lines 55-90):

src/syscall/syscall_linux.go, lines 55-57:

// src/syscall/syscall_linux.go, lines 55-57
func RawSyscall(trap, a1, a2, a3 uintptr) (r1, r2 uintptr, err Errno) {
    return RawSyscall6(trap, a1, a2, a3, 0, 0, 0)
}

src/syscall/syscall_linux.go, lines 63-68:

// src/syscall/syscall_linux.go, lines 63-68
func RawSyscall6(trap, a1, a2, a3, a4, a5, a6 uintptr) (r1, r2 uintptr, err Errno) {
    var errno uintptr
    r1, r2, errno = linux.Syscall6(trap, a1, a2, a3, a4, a5, a6)
    err = Errno(errno)
    return
}

RawSyscall goes directly to the kernel without notifying the Go scheduler. This is dangerous – if the syscall blocks, the scheduler doesn’t know, and it can’t reuse the goroutine’s P.

The safe version notifies the scheduler:

src/syscall/syscall_linux.go, lines 73-90:

// src/syscall/syscall_linux.go, lines 73-90
func Syscall(trap, a1, a2, a3 uintptr) (r1, r2 uintptr, err Errno) {
    runtime_entersyscall()
    r1, r2, err = RawSyscall6(trap, a1, a2, a3, 0, 0, 0)
    runtime_exitsyscall()
    return
}

The critical difference: Syscall wraps the raw call with runtime_entersyscall() / runtime_exitsyscall(). These are linked to the runtime’s entersyscall and exitsyscall functions:

src/syscall/syscall_linux.go, lines 29-33:

// src/syscall/syscall_linux.go, lines 29-33
//go:linkname runtime_entersyscall runtime.entersyscall
func runtime_entersyscall()

//go:linkname runtime_exitsyscall runtime.exitsyscall
func runtime_exitsyscall()

When to use which?

• Syscall: For potentially blocking calls (read, write, open, connect). Notifies the scheduler so the P can be reassigned.
• RawSyscall: For calls guaranteed to be fast and non-blocking (getpid, getuid). Avoids the overhead of scheduler notification.

Using RawSyscall for a blocking call is a bug: it can cause the entire program to stall because the scheduler thinks the M’s P is still in use.

Errno: Error Handling

From src/syscall/syscall_unix.go (lines 94-108):

src/syscall/syscall_unix.go, lines 94-108:

// src/syscall/syscall_unix.go, lines 94-108

// An Errno is an unsigned number describing an error condition.
// It implements the error interface. The zero Errno is by convention
// a non-error, so code to convert from Errno to error should use:
//
//  err = nil
//  if errno != 0 {
//      err = errno
//  }
//
// Errno values can be tested against error values using [errors.Is].
// For example:
//
//  _, _, err := syscall.Syscall(...)
//  if errors.Is(err, fs.ErrNotExist) ...
type Errno uintptr

Errno is just a uintptr that implements the error interface. It maps to kernel errno values (ENOENT = 2, EACCES = 13, etc.).

The Errno.Is() method (lines 120-131) maps errno values to Go’s standard error sentinels:

src/syscall/syscall_unix.go, lines 120-131:

// src/syscall/syscall_unix.go, lines 120-131
func (e Errno) Is(target error) bool {
    switch target {
    case oserror.ErrPermission:
        return e == EACCES || e == EPERM
    case oserror.ErrExist:
        return e == EEXIST || e == ENOTEMPTY
    case oserror.ErrNotExist:
        return e == ENOENT
    case errorspkg.ErrUnsupported:
        return e == ENOSYS || e == ENOTSUP || e == EOPNOTSUPP
    }
    return false
}

Higher-level wrappers: Read and Write

The syscall package also provides higher-level Go functions. For example:

src/syscall/syscall_unix.go, lines 182-199:

// src/syscall/syscall_unix.go, lines 182-199
func Read(fd int, p []byte) (n int, err error) {
    n, err = read(fd, p)
    if race.Enabled {
        if n > 0 {
            race.WriteRange(unsafe.Pointer(&p[0]), n)
        }
        if err == nil {
            race.Acquire(unsafe.Pointer(&ioSync))
        }
    }
    if msan.Enabled && n > 0 {
        msan.Write(unsafe.Pointer(&p[0]), uintptr(n))
    }
    if asan.Enabled && n > 0 {
        asan.Write(unsafe.Pointer(&p[0]), uintptr(n))
    }
    return
}

Note how the wrapper integrates with Go’s race detector, memory sanitizer (MSan), and address sanitizer (ASan). These instrumentation hooks are invisible to normal users but critical for development tooling.

5. VDSO: Avoiding the Kernel (5 min)

Some syscalls are so frequent that even the overhead of the SYSCALL instruction is too much. The most common example is clock_gettime – called every time the runtime needs a timestamp (scheduling decisions, timer management, time.Now()).

Linux provides the vDSO (virtual Dynamic Shared Object): a small shared library that the kernel maps into every process’s address space. It contains user-space implementations of certain syscalls that can execute without trapping into the kernel.

The Go runtime looks up the vDSO symbol at startup (src/runtime/vdso_linux.go) and uses it as a fast path:

src/runtime/sys_linux_amd64.s, lines 222-298:

// src/runtime/sys_linux_amd64.s, lines 222-298

// func nanotime1() int64
TEXT runtime·nanotime1(SB),NOSPLIT,$16-8
    // We don't know how much stack space the VDSO code will need,
    // so switch to g0.
    ...
    MOVQ    g_m(R14), BX

    // Set vdsoPC and vdsoSP for SIGPROF traceback.
    ...
    MOVL    $1, DI // CLOCK_MONOTONIC
    LEAQ    0(SP), SI
    MOVQ    runtime·vdsoClockgettimeSym(SB), AX
    CMPQ    AX, $0
    JEQ     fallback
    CALL    AX          // call vDSO function (user-space, no kernel trap!)
ret:
    MOVQ    0(SP), AX   // sec
    MOVQ    8(SP), DX   // nsec
    ...
    IMULQ   $1000000000, AX
    ADDQ    DX, AX
    MOVQ    AX, ret+0(FP)
    RET
fallback:
    MOVQ    $SYS_clock_gettime, AX
    SYSCALL             // fall back to real syscall if vDSO unavailable
    JMP     ret

The key optimization:

1. Check if runtime.vdsoClockgettimeSym is non-nil (the vDSO symbol was found at startup)
2. If yes, CALL AX – this is a normal function call, no kernel transition
3. If no, fall back to SYSCALL with SYS_clock_gettime

The vDSO implementation works by having the kernel map a read-only page of time data into user space, updated by the kernel on every timer tick. The user-space code just reads from this page. This turns a ~100ns syscall into a ~20ns function call.

Note the comment: “We don’t know how much stack space the VDSO code will need, so switch to g0.” The runtime switches to the g0 stack (the OS-thread-sized stack) before calling vDSO code, because the vDSO is essentially kernel-provided C code that assumes a normal-sized stack – not the small goroutine stack (which starts at 2 KB).

6. entersyscall/exitsyscall: The Scheduler Handoff (10 min)

This is the most OS-like aspect of Go’s syscall handling. When a goroutine enters a potentially blocking syscall, the runtime must decide what to do with the goroutine’s P (logical processor).

The Problem

Consider: goroutine G7 is running on M3 with P2. G7 calls read() on a network socket. This could block for milliseconds or even seconds. If P2 stays attached to M3 during the block, no other goroutine can use that logical processor – the system loses 1/GOMAXPROCS of its scheduling capacity.

entersyscall: Preparing for the Block

src/runtime/proc.go, lines 4627-4716:

// src/runtime/proc.go, lines 4627-4716
func reentersyscall(pc, sp, bp uintptr) {
    gp := getg()

    // Disable preemption because during this function g is in Gsyscall status,
    // but can have inconsistent g->sched, do not let GC observe it.
    gp.m.locks++

    // Entersyscall must not call any function that might split/grow the stack.
    // Catch calls that might, by replacing the stack guard with something that
    // will trip any stack check and leaving a flag to tell newstack to die.
    gp.stackguard0 = stackPreempt
    gp.throwsplit = true

    // Copy the syscalltick over so we can identify if the P got stolen later.
    gp.m.syscalltick = gp.m.p.ptr().syscalltick

    ...
    // Leave SP around for GC and traceback.
    save(pc, sp, bp)
    gp.syscallsp = sp
    gp.syscallpc = pc
    gp.syscallbp = bp

    ...
    // Transition to _Gsyscall.
    // As soon as we switch to _Gsyscall, we are in danger of losing our P.
    if gp.bubble != nil || !gp.atomicstatus.CompareAndSwap(_Grunning, _Gsyscall) {
        casgstatus(gp, _Grunning, _Gsyscall)
    }
    ...
}

Key steps:

1. Save the goroutine’s registers (PC, SP, BP) so the GC and traceback can still walk the stack
2. Record syscalltick so the M can detect later if its P was stolen
3. Transition G status from _Grunning to _Gsyscall via atomic CAS
4. Do NOT release the P immediately – the optimistic assumption is that the syscall will return quickly

The P remains attached to the M, but the status change to _Gsyscall signals to the rest of the runtime that this P is available for stealing if necessary.

exitsyscall: Coming Back

src/runtime/proc.go, lines 4883-4953:

// src/runtime/proc.go, lines 4883-4953
func exitsyscall() {
    gp := getg()

    gp.m.locks++ // see comment in entersyscall
    if sys.GetCallerSP() > gp.syscallsp {
        throw("exitsyscall: syscall frame is no longer valid")
    }
    gp.waitsince = 0

    ...
    // Check if we still have our P (fast path).
    oldp := gp.m.oldp.ptr()
    gp.m.oldp.set(nil)

    pp := gp.m.p.ptr()
    if pp != nil {
        // Fast path: we still have our P.
        // Transition _Gsyscall -> _Grunning and continue.
        if gp.bubble != nil || !gp.atomicstatus.CompareAndSwap(_Gsyscall, _Grunning) {
            casgstatus(gp, _Gsyscall, _Grunning)
        }
        ...
    } else {
        // Slow path: we lost our P. Try to get another one.
        // If no P available, the goroutine goes _Gsyscall -> _Grunnable
        // and is placed on the global run queue.
        systemstack(func() {
            if pp := exitsyscallTryGetP(oldp); pp != nil {
                // Got a P, install it and continue.
                ...
            } else {
                // No P available. Park this M and put G on global run queue.
                exitsyscallNoP(gp)
            }
        })
    }
    ...
}

Two paths:

1. Fast path (common case): The P is still attached. The syscall was fast enough that sysmon didn’t steal the P. Transition back to _Grunning and continue executing.

2. Slow path: The P was stolen (by sysmon’s retake() function). The M must either acquire a different idle P or park the goroutine on the global run queue and park the M.

sysmon retake: Stealing Ps from Slow Syscalls

The sysmon thread periodically checks for Ps that are stuck in syscalls:

src/runtime/proc.go, lines 6630-6652:

// src/runtime/proc.go, lines 6630-6652
func retake(now int64) uint32 {
    n := 0
    lock(&allpLock)
    for i := 0; i < len(allp); i++ {
        pp := allp[i]
        if pp == nil || atomic.Load(&pp.status) != _Prunning {
            continue
        }
        pd := &pp.sysmontick

        // Preempt G if it's running on the same schedtick for
        // too long. This could be from a single long-running
        // goroutine or a sequence of goroutines run via
        // runnext, which share a single schedtick time slice.
        schedt := int64(pp.schedtick)
        if int64(pd.schedtick) != schedt {
            pd.schedtick = uint32(schedt)
            pd.schedwhen = now
        } else if pd.schedwhen+forcePreemptNS <= now {
            preemptone(pp)
            // If pp is in a syscall, preemptone doesn't work.
            // The goroutine nor the thread can respond to a
            // preemption request because they're not in Go code,
            // so we need to take the P ourselves.
            sysretake = true
        }
        ...
    }
}

sysmon checks the syscalltick to see how long the P has been in a syscall. If it’s been too long (the threshold is 10ms, defined by forcePreemptNS), sysmon steals the P and hands it to another M that has goroutines waiting to run.

The Full Lifecycle

  Goroutine G7 on M3 with P2:

  1. G7 calls syscall.Read()
     → Syscall() calls runtime_entersyscall()
     → G7 status: _Grunning → _Gsyscall
     → P2 stays attached to M3 (optimistic)
     → Raw syscall to kernel

  2a. FAST: Syscall returns quickly
     → runtime_exitsyscall()
     → P2 still attached? Yes!
     → G7 status: _Gsyscall → _Grunning
     → Continue executing

  2b. SLOW: Syscall blocks for a while
     → sysmon's retake() notices P2 stuck in syscall
     → sysmon steals P2, hands it to idle M (or wakes one)
     → Other goroutines can now run on P2
     → Eventually, kernel returns from syscall
     → runtime_exitsyscall()
     → P2 gone! Try to get any idle P.
     → If no P available, park G7 on global run queue, park M3

This design is elegant: the common case (fast syscall) has minimal overhead (just a status CAS), while the slow case ensures the system doesn’t lose scheduling capacity.

Summary

Exercises

1. Trace a syscall: Write a Go program that reads from a file. Use strace -f -e trace=read (Linux) or dtruss (macOS) to observe the actual system calls. Compare the syscall arguments with what you see in the assembly.

2. Syscall vs. RawSyscall: Write a program that makes a slow syscall (e.g., read from a pipe that takes 1 second to produce data) using both syscall.Syscall and syscall.RawSyscall. With GOMAXPROCS=1 and a second goroutine printing to stdout, observe the difference in behavior.

3. VDSO experiment: Write a tight loop calling time.Now() one million times and measure elapsed time. Compare this with a tight loop calling syscall.Syscall(SYS_clock_gettime, ...). How much faster is the vDSO path?

4. Read the entersyscall code: Open src/runtime/proc.go and read reentersyscall (line 4627) through exitsyscall (line 5025). Draw a state diagram showing all the possible transitions for G status, P ownership, and M activity during a syscall.

Further Reading

• Linux syscall table: arch/x86/entry/syscalls/syscall_64.tbl in the Linux kernel source
• The Definitive Guide to Linux System Calls – covers SYSCALL, SYSENTER, vDSO
• Go source: src/runtime/sys_linux_amd64.s – Linux syscall assembly
• Go source: src/runtime/sys_darwin_arm64.s – macOS trampoline assembly
• Go source: src/runtime/proc.go – entersyscall/exitsyscall

Module 2: System Calls (sys_linux_amd64.s, syscall package)

Comprehension Check

Question 1 (Code Reading)

Examine this snippet from runtime/sys_linux_amd64.s:

TEXT runtime·futex(SB),NOSPLIT,$0
	MOVQ	addr+0(FP), DI
	MOVL	op+8(FP), SI
	MOVL	val+12(FP), DX
	MOVQ	ts+16(FP), R10
	MOVQ	addr2+24(FP), R8
	MOVL	val3+32(FP), R9
	MOVL	$SYS_futex, AX
	SYSCALL
	MOVL	AX, ret+40(FP)
	RET

Explain what each MOVQ/MOVL instruction is doing and why the function uses NOSPLIT.

Question 2 (Short Answer)

The Go runtime makes system calls directly via assembly rather than using the C library (libc). What are two advantages and one disadvantage of this approach?

Question 3 (True/False with Explanation)

True or False: When a goroutine makes a blocking system call (like reading from a file), the P (processor) remains attached to the M (OS thread) until the syscall returns.

Question 4 (What Would Happen If…)

What would happen if the Go runtime used blocking read() system calls for network I/O instead of non-blocking I/O with epoll/kqueue? Consider the impact on a web server handling 10,000 concurrent connections.

Question 5 (Short Answer)

Explain the difference between the syscall package and the runtime package’s internal system call mechanisms. When would a Go developer use each?

Question 6 (Code Reading)

Consider this function from runtime/os_linux.go:

func futexsleep(addr *uint32, val uint32, ns int64) {
	if ns < 0 {
		futex(unsafe.Pointer(addr), _FUTEX_WAIT_PRIVATE, val, nil, nil, 0)
		return
	}

	var ts timespec
	ts.setNsec(ns)
	futex(unsafe.Pointer(addr), _FUTEX_WAIT_PRIVATE, val, &ts, nil, 0)
}

Why does the function check the value of val atomically against *addr inside the kernel? What race condition would exist without this check?

Module 8: Memory Management (60 min)

Background: Memory Management from Hardware to Runtime

Memory management is a problem that spans every layer of a computer system, from the hardware up to application code. At the bottom, the CPU’s memory management unit (MMU) translates virtual addresses to physical addresses using page tables – a mechanism first pioneered by the Atlas computer at the University of Manchester in 1962, which introduced the concept of “one-level storage” where programs could address more memory than physically existed. That idea became virtual memory, and by the 1970s it was universal. Modern systems use multi-level page tables (four or five levels on x86-64) to map a 48- or 57-bit virtual address space, with the TLB caching recent translations to avoid the cost of a full page table walk on every memory access. The page – typically 4 KB – is the fundamental unit of this translation, though huge pages (2 MB or 1 GB on x86) reduce TLB pressure for large working sets.

The operating system kernel manages the page tables and physical memory. Linux uses a buddy allocator to manage physical page frames: it maintains free lists of power-of-two-sized page blocks and splits or coalesces them as needed, providing O(log n) allocation with low external fragmentation. On top of the buddy allocator sits the SLUB allocator (the successor to the original slab allocator designed by Jeff Bonwick at Sun in 1994), which carves pages into fixed-size object caches for frequently allocated kernel structures like inodes, dentries, and socket buffers. SLUB maintains per-CPU freelists to minimize lock contention – a design principle we will see repeated at every level of the stack. The kernel exposes memory to user space through system calls like mmap, which creates virtual address mappings that are backed by physical pages lazily, on first access (demand paging). Transparent Huge Pages (THP) allow the kernel to automatically promote 4 KB pages to 2 MB huge pages, improving TLB utilization for large allocations without requiring application changes.

For user-space programs, calling mmap for every allocation would be far too expensive – each call requires a transition to kernel mode and manipulation of page table entries. User-space allocators like dlmalloc (Doug Lea’s malloc, the basis of glibc’s allocator), jemalloc (developed for FreeBSD and later adopted by Facebook), tcmalloc (Google’s Thread-Caching Malloc), and mimalloc (Microsoft Research’s compact allocator) all solve this by requesting large chunks of memory from the OS and then sub-dividing them efficiently. These allocators face a common set of challenges: minimizing fragmentation (both internal fragmentation from rounding up to size classes and external fragmentation from non-contiguous free blocks), reducing lock contention in multi-threaded programs (typically through per-thread or per-CPU caches), and maintaining good cache locality. tcmalloc’s key innovation was its hierarchy of thread-local caches, central freelists, and a global page heap – a design that directly inspired Go’s memory allocator.

Where manual allocators require the programmer to pair every allocation with an explicit free, garbage collection automates memory reclamation by identifying objects that are no longer reachable. The idea dates back to John McCarthy’s Lisp in 1960, which used a simple stop-the-world mark-and-sweep collector. Since then, GC research has produced a rich taxonomy of approaches: reference counting (tracking how many pointers refer to each object, used by CPython and Swift), generational collection (exploiting the observation that most objects die young, pioneered by David Ungar in 1984), copying/compacting collectors (which eliminate fragmentation by relocating live objects), and concurrent/incremental collectors (which minimize pause times by doing GC work alongside the application). Java’s ecosystem showcases this diversity: its G1 collector partitions the heap into regions for incremental collection, ZGC uses colored pointers and load barriers to achieve sub-millisecond pauses on multi-terabyte heaps, and Shenandoah performs concurrent compaction using Brooks forwarding pointers. On modern NUMA systems, allocators must also be topology-aware – allocating memory on the same NUMA node as the thread that will use it, since cross-node accesses can be 2-3x slower.

Go’s runtime sits at the intersection of these ideas. Its memory allocator descends from tcmalloc but has diverged significantly, using a hierarchy of per-P caches (mcache), per-size-class central lists (mcentral), and a global heap (mheap) backed by mmap. Its garbage collector is a concurrent, tri-color mark-and-sweep design – non-generational and non-compacting, but with sub-millisecond STW pauses achieved through concurrent marking with a hybrid write barrier. In this module, we trace an allocation from new(T) all the way down to mmap, then examine how the garbage collector reclaims memory concurrently with the running program.

1. Virtual Memory Review

Pages and page tables

Modern CPUs use virtual memory to provide each process with an isolated address space. The hardware translates virtual addresses to physical addresses using page tables, with a typical page size of 4 KB (though large/huge pages of 2 MB or 1 GB exist).

Key concepts:

• Virtual address space: each process sees a flat, contiguous address space.
• Page table: a per-process data structure (managed by the OS kernel) mapping virtual page numbers to physical frame numbers.
• TLB (Translation Lookaside Buffer): a hardware cache of recent page table entries. TLB misses trigger a page table walk.
• Page fault: when a virtual page has no valid mapping, the CPU traps to the kernel, which may allocate a physical frame, read from disk, or deliver a segfault.

mmap: Requesting memory from the OS

User-space programs request virtual memory from the kernel using mmap:

void *mmap(void *addr, size_t length, int prot, int flags, int fd, off_t offset);

With MAP_ANONYMOUS | MAP_PRIVATE, mmap creates a new anonymous mapping – virtual pages backed by physical memory (lazily, on first access). The Go runtime uses this as its sole interface to the OS for heap memory.

2. Go’s Memory Allocator Architecture

The allocator is documented at the top of malloc.go:

src/runtime/malloc.go lines 5-25:

// src/runtime/malloc.go, lines 5-25

// Memory allocator.
//
// This was originally based on tcmalloc, but has diverged quite a bit.
// http://goog-perftools.sourceforge.net/doc/tcmalloc.html

// The main allocator works in runs of pages.
// Small allocation sizes (up to and including 32 kB) are
// rounded to one of about 70 size classes, each of which
// has its own free set of objects of exactly that size.
// Any free page of memory can be split into a set of objects
// of one size class, which are then managed using a free bitmap.
//
// The allocator's data structures are:
//
//  fixalloc: a free-list allocator for fixed-size off-heap objects,
//      used to manage storage used by the allocator.
//  mheap: the malloc heap, managed at page (8192-byte) granularity.
//  mspan: a run of in-use pages managed by the mheap.
//  mcentral: collects all spans of a given size class.
//  mcache: a per-P cache of mspans with free space.
//  mstats: allocation statistics.

The allocation hierarchy

src/runtime/malloc.go lines 27-47:

// src/runtime/malloc.go, lines 27-47

// Allocating a small object proceeds up a hierarchy of caches:
//
//  1. Round the size up to one of the small size classes
//     and look in the corresponding mspan in this P's mcache.
//     Scan the mspan's free bitmap to find a free slot.
//     If there is a free slot, allocate it.
//     This can all be done without acquiring a lock.
//
//  2. If the mspan has no free slots, obtain a new mspan
//     from the mcentral's list of mspans of the required size
//     class that have free space.
//     Obtaining a whole span amortizes the cost of locking
//     the mcentral.
//
//  3. If the mcentral's mspan list is empty, obtain a run
//     of pages from the mheap to use for the mspan.
//
//  4. If the mheap is empty or has no page runs large enough,
//     allocate a new group of pages (at least 1MB) from the
//     operating system. Allocating a large run of pages
//     amortizes the cost of talking to the operating system.

Deallocation (sweeping) hierarchy

src/runtime/malloc.go lines 49-63:

// src/runtime/malloc.go, lines 49-63

// Sweeping an mspan and freeing objects on it proceeds up a similar
// hierarchy:
//
//  1. If the mspan is being swept in response to allocation, it
//     is returned to the mcache to satisfy the allocation.
//
//  2. Otherwise, if the mspan still has allocated objects in it,
//     it is placed on the mcentral free list for the mspan's size
//     class.
//
//  3. Otherwise, if all objects in the mspan are free, the mspan's
//     pages are returned to the mheap and the mspan is now dead.
//
// Allocating and freeing a large object uses the mheap
// directly, bypassing the mcache and mcentral.

Diagram: Allocation hierarchy

  Goroutine calls new(T)
        |
        v
  mallocgc(size, typ, needzero)
        |
        +------ size == 0? -----> return &zerobase
        |
        +------ size < 16 bytes, no pointers? -----> Tiny allocator
        |
        +------ size <= 32 KB? -----> Small object path:
        |           |
        |           v
        |       mcache (per-P, no lock)
        |           |  no free slot?
        |           v
        |       mcentral (per-size-class, locked)
        |           |  no spans available?
        |           v
        |       mheap (global, locked)
        |           |  no pages?
        |           v
        |       mmap(OS)
        |
        +------ size > 32 KB? -----> Large object path:
                    |
                    v
                mheap directly -> mmap if needed

3. Virtual Memory Layout: Arenas

Go organizes its heap into arenas:

src/runtime/malloc.go lines 79-94:

// src/runtime/malloc.go, lines 79-94

// The heap consists of a set of arenas, which are 64MB on 64-bit and
// 4MB on 32-bit (heapArenaBytes). Each arena's start address is also
// aligned to the arena size.
//
// Each arena has an associated heapArena object that stores the
// metadata for that arena: the heap bitmap for all words in the arena
// and the span map for all pages in the arena. heapArena objects are
// themselves allocated off-heap.
//
// Since arenas are aligned, the address space can be viewed as a
// series of arena frames. The arena map (mheap_.arenas) maps from
// arena frame number to *heapArena, or nil for parts of the address
// space not backed by the Go heap. The arena map is structured as a
// two-level array consisting of a "L1" arena map and many "L2" arena
// maps; however, since arenas are large, on many architectures, the
// arena map consists of a single, large L2 map.

The two-level arena map allows the runtime to cover the entire 48-bit virtual address space without allocating metadata for unused regions:

src/runtime/mheap.go line 150:

// src/runtime/mheap.go, line 150
arenas [1 << arenaL1Bits]*[1 << arenaL2Bits]*heapArena

Each heapArena stores metadata for one 64 MB region:

src/runtime/mheap.go lines 268-338:

// src/runtime/mheap.go, lines 268-338

type heapArena struct {
    _ sys.NotInHeap

    // spans maps from virtual address page ID within this arena to *mspan.
    // For allocated spans, their pages map to the span itself.
    // For free spans, only the lowest and highest pages map to the span itself.
    spans [pagesPerArena]*mspan

    // pageInUse is a bitmap that indicates which spans are in
    // state mSpanInUse.
    pageInUse [pagesPerArena / 8]uint8

    // pageMarks is a bitmap that indicates which spans have any
    // marked objects on them.
    pageMarks [pagesPerArena / 8]uint8

    // zeroedBase marks the first byte of the first page in this
    // arena which hasn't been used yet and is therefore already
    // zero.
    zeroedBase uintptr
}

OS connection: The arena system is Go’s user-space equivalent of a page table. Just as the OS kernel uses a multi-level page table to map virtual addresses to physical frames, Go uses a two-level arena map to map heap addresses to span metadata.

4. The mheap Struct: The Central Heap

The mheap is the single global heap structure:

src/runtime/mheap.go lines 64-264:

// src/runtime/mheap.go, lines 64-264

type mheap struct {
    _ sys.NotInHeap

    // lock must only be acquired on the system stack, otherwise a g
    // could self-deadlock if its stack grows with the lock held.
    lock mutex

    pages pageAlloc // page allocation data structure

    sweepgen uint32 // sweep generation, see comment in mspan

    // allspans is a slice of all mspans ever created.
    allspans []*mspan // all spans out there

    // ...

    // arenas is the heap arena map.
    arenas [1 << arenaL1Bits]*[1 << arenaL2Bits]*heapArena

    // central free lists for small size classes.
    central [numSpanClasses]struct {
        mcentral mcentral
        pad      [...]byte  // cache line padding
    }

    spanalloc  fixalloc // allocator for span
    cachealloc fixalloc // allocator for mcache
    // ...
}

var mheap_ mheap

Key fields

Design note: The central array is padded to cache line boundaries to avoid false sharing between different size classes being accessed by different processors.

5. Size Classes and mspan

Size classes

Go rounds small allocations up to one of approximately 70 size classes. For example: 8, 16, 24, 32, 48, 64, 80, 96, …, 32768 bytes. Each size class has a corresponding span class (which also encodes whether the span contains pointers or not – “noscan” vs. “scan”).

This eliminates external fragmentation for small objects: all objects within a span are the same size, so any free slot can satisfy any request of that class.

The mspan struct

An mspan represents a contiguous run of pages dedicated to one size class:

src/runtime/mheap.go lines 422-516:

// src/runtime/mheap.go, lines 422-516

type mspan struct {
    _    sys.NotInHeap
    next *mspan     // next span in list, or nil if none
    prev *mspan     // previous span in list, or nil if none
    list *mSpanList // For debugging.

    startAddr uintptr // address of first byte of span aka s.base()
    npages    uintptr // number of pages in span

    manualFreeList gclinkptr // list of free objects in mSpanManual spans

    // freeindex is the slot index between 0 and nelems at which to begin
    // scanning for the next free object in this span.
    // Each allocation scans allocBits starting at freeindex until it
    // encounters a 0 indicating a free object.
    freeindex uint16

    nelems uint16 // number of objects in the span

    freeIndexForScan uint16

    // Cache of the allocBits at freeindex. allocCache is shifted
    // such that the lowest bit corresponds to the bit freeindex.
    // allocCache holds the complement of allocBits, thus allowing
    // ctz (count trailing zero) to use it directly.
    allocCache uint64

    // allocBits and gcmarkBits hold pointers to a span's mark and
    // allocation bits.
    allocBits  *gcBits
    gcmarkBits *gcBits
    pinnerBits *gcBits // bitmap for pinned objects

    // sweep generation:
    // if sweepgen == h->sweepgen - 2, the span needs sweeping
    // if sweepgen == h->sweepgen - 1, the span is currently being swept
    // if sweepgen == h->sweepgen, the span is swept and ready to use
    // if sweepgen == h->sweepgen + 1, the span was cached before sweep
    //   began and is still cached, and needs sweeping
    // if sweepgen == h->sweepgen + 3, the span was swept and then cached
    //   and is still cached
    sweepgen              uint32
    divMul                uint32        // for divide by elemsize
    allocCount            uint16        // number of allocated objects
    spanclass             spanClass     // size class and noscan (uint8)
    state                 mSpanStateBox // mSpanInUse etc; accessed atomically
    needzero              uint8         // needs to be zeroed before allocation
    allocCountBeforeCache uint16
    elemsize              uintptr       // computed from sizeclass or from npages
    limit                 uintptr       // end of data in span
    speciallock           mutex         // guards specials list
    specials              *special      // linked list of special records
    largeType             *_type        // malloc header for large objects
}

Allocation bitmaps

Each span has two key bitmaps:

• allocBits: A bit per object slot. 1 = allocated, 0 = free.
• gcmarkBits: A bit per object slot. 1 = marked (reachable), 0 = unmarked.

After GC, the sweep phase replaces allocBits with gcmarkBits (objects that were not marked are now free), and gcmarkBits is reset to all zeros for the next cycle. This is a very efficient way to free dead objects in bulk.

The allocCache field is an optimization: it caches 64 bits of the complement of allocBits starting at freeindex, allowing the allocator to find a free slot using a single ctz (count trailing zeros) CPU instruction.

Span states

src/runtime/mheap.go lines 386-390:

// src/runtime/mheap.go, lines 386-390
const (
    mSpanDead   mSpanState = iota
    mSpanInUse             // allocated for garbage collected heap
    mSpanManual            // allocated for manual management (e.g., stack allocator)
)

Transitions between states are constrained by the GC phase to maintain correctness of concurrent marking.

6. mcache: Per-P Lock-Free Allocation

The mcache is the first level of the allocation hierarchy. There is one per P (logical processor), so allocations from it require no locking:

src/runtime/mcache.go lines 14-66:

// src/runtime/mcache.go, lines 14-66

// Per-thread (in Go, per-P) cache for small objects.
// This includes a small object cache and local allocation stats.
// No locking needed because it is per-thread (per-P).
//
// mcaches are allocated from non-GC'd memory, so any heap pointers
// must be specially handled.
type mcache struct {
    _ sys.NotInHeap

    // The following members are accessed on every malloc,
    // so they are grouped here for better caching.
    nextSample  int64   // trigger heap sample after allocating this many bytes
    memProfRate int     // cached mem profile rate
    scanAlloc   uintptr // bytes of scannable heap allocated

    // Allocator cache for tiny objects w/o pointers.
    // See "Tiny allocator" comment in malloc.go.
    tiny       uintptr
    tinyoffset uintptr
    tinyAllocs uintptr

    // The rest is not accessed on every malloc.

    // alloc contains spans to allocate from, indexed by spanClass.
    alloc [numSpanClasses]*mspan

    stackcache [_NumStackOrders]stackfreelist

    // flushGen indicates the sweepgen during which this mcache
    // was last flushed.
    flushGen atomic.Uint32
}

How mcache allocation works

1. Look up mcache.alloc[spanClass] to get the current mspan for this size class.
2. Use allocCache + ctz to find a free slot in constant time.
3. If no free slots, call mcache.nextFree() which gets a new span from mcentral.

The alloc array has numSpanClasses entries (approximately 140: ~70 size classes x 2 for scan/noscan). Each entry points to an mspan with free slots.

OS connection: This per-CPU caching strategy is the same principle used in Linux’s SLAB/SLUB allocator, which maintains per-CPU freelists to avoid lock contention on the global heap.

7. The Tiny Allocator

For very small allocations (< 16 bytes) that do not contain pointers, Go uses a special tiny allocator that packs multiple objects into a single 16-byte block:

src/runtime/malloc.go lines 1270-1298:

// src/runtime/malloc.go, lines 1270-1298

// Tiny allocator.
//
// Tiny allocator combines several tiny allocation requests
// into a single memory block. The resulting memory block
// is freed when all subobjects are unreachable. The subobjects
// must be noscan (don't have pointers), this ensures that
// the amount of potentially wasted memory is bounded.
//
// Size of the memory block used for combining (maxTinySize) is tunable.
// Current setting is 16 bytes, which relates to 2x worst case memory
// wastage (when all but one subobjects are unreachable).
// 8 bytes would result in no wastage at all, but provides less
// opportunities for combining.
// 32 bytes provides more opportunities for combining,
// but can lead to 4x worst case wastage.
// The best case winning is 8x regardless of block size.
//
// Objects obtained from tiny allocator must not be freed explicitly.
// So when an object will be freed explicitly, we ensure that
// its size >= maxTinySize.
//
// SetFinalizer has a special case for objects potentially coming
// from tiny allocator, it such case it allows to set finalizers
// for an inner byte of a memory block.
//
// The main targets of tiny allocator are small strings and
// standalone escaping variables. On a json benchmark
// the allocator reduces number of allocations by ~12% and
// reduces heap size by ~20%.

Implementation

src/runtime/malloc.go lines 1299-1332:

// src/runtime/malloc.go, lines 1299-1332

c := getMCache(mp)
off := c.tinyoffset
// Align tiny pointer for required (conservative) alignment.
if size&7 == 0 {
    off = alignUp(off, 8)
} else if size&3 == 0 {
    off = alignUp(off, 4)
} else if size&1 == 0 {
    off = alignUp(off, 2)
}
if off+size <= maxTinySize && c.tiny != 0 {
    // The object fits into existing tiny block.
    x := unsafe.Pointer(c.tiny + off)
    c.tinyoffset = off + size
    c.tinyAllocs++
    mp.mallocing = 0
    releasem(mp)
    return x, 0
}
// Allocate a new maxTinySize block.
span := c.alloc[tinySpanClass]
v := nextFreeFast(span)
if v == 0 {
    v, span, checkGCTrigger = c.nextFree(tinySpanClass)
}

The tiny allocator maintains a current 16-byte block (c.tiny) and bumps a pointer (c.tinyoffset) within it. When the block is full, a new 16-byte slot is obtained from the regular span allocator.

Trade-off: The entire 16-byte block cannot be freed until all sub-objects within it become unreachable. This trades a small amount of memory waste for a large reduction in allocation overhead.

8. mallocgc: The Central Allocation Function

Every heap allocation in Go flows through mallocgc:

src/runtime/malloc.go lines 1119-1129:

// src/runtime/malloc.go, lines 1119-1129

func mallocgc(size uintptr, typ *_type, needzero bool) unsafe.Pointer {
    if doubleCheckMalloc {
        if gcphase == _GCmarktermination {
            throw("mallocgc called with gcphase == _GCmarktermination")
        }
    }

    // Short-circuit zero-sized allocation requests.
    if size == 0 {
        return unsafe.Pointer(&zerobase)
    }

Allocation paths

src/runtime/malloc.go lines 1186-1209:

// src/runtime/malloc.go, lines 1186-1209

if size <= maxSmallSize-gc.MallocHeaderSize {
    if typ == nil || !typ.Pointers() {
        // tiny allocations might be kept alive by other co-located values.
        gp := getg()
        if size < maxTinySize && gp.secret == 0 {
            x, elemsize = mallocgcTiny(size, typ)
        } else {
            x, elemsize = mallocgcSmallNoscan(size, typ, needzero)
        }
    } else {
        if !needzero {
            throw("objects with pointers must be zeroed")
        }
        x, elemsize = mallocgcSmallScanHeader(size, typ)
    }
} else {
    x, elemsize = mallocgcLarge(size, typ, needzero)
}

Three paths:

1. Tiny (size < 16, no pointers): bump allocator within 16-byte blocks.
2. Small (size <= 32 KB): round to size class, allocate from mcache span.
3. Large (size > 32 KB): allocate directly from mheap.

GC assist

src/runtime/malloc.go lines 1162-1166:

// src/runtime/malloc.go, lines 1162-1166

// Assist the GC if needed. (On the reuse path, we currently compensate for this;
// changes here might require changes there.)
if gcBlackenEnabled != 0 {
    deductAssistCredit(size)
}

Before allocating, the goroutine may be required to do GC marking work proportional to the amount it is allocating. This is the GC assist mechanism: it ensures that goroutines that allocate heavily also contribute to GC work, preventing the collector from falling behind.

OS connection: GC assist is analogous to proportional-share scheduling – threads that consume more resources (memory) must contribute more work (marking) to maintain system-wide progress.

9. Lazy Zeroing

src/runtime/malloc.go lines 65-75:

// src/runtime/malloc.go, lines 65-75

// If mspan.needzero is false, then free object slots in the mspan are
// already zeroed. Otherwise if needzero is true, objects are zeroed as
// they are allocated. There are various benefits to delaying zeroing
// this way:
//
//  1. Stack frame allocation can avoid zeroing altogether.
//
//  2. It exhibits better temporal locality, since the program is
//     probably about to write to the memory.
//
//  3. We don't zero pages that never get reused.

Rather than zeroing memory when it is freed, Go zeros memory when it is allocated (if needed). Pages obtained fresh from the OS via mmap are guaranteed to be zeroed by the kernel, so the runtime tracks whether a span’s pages have been written to (needzero). This avoids redundant zeroing and improves cache behavior.

OS connection: This is a direct consequence of the OS’s lazy page allocation: mmap returns virtual pages that are mapped to the zero page until first write (copy-on-write). The runtime cooperates with this by tracking which pages are “still clean.”

10. Garbage Collection Overview

The Go garbage collector is described at the top of src/runtime/mgc.go lines 5-10:

// src/runtime/mgc.go, lines 5-10

// Garbage collector (GC).
//
// The GC runs concurrently with mutator threads, is type accurate (aka
// precise), allows multiple GC threads to run in parallel. It is a
// concurrent mark and sweep that uses a write barrier. It is
// non-generational and non-compacting. Allocation is done using size
// segregated per P allocation areas to minimize fragmentation while
// eliminating locks in the common case.

The Four Phases

src/runtime/mgc.go lines 24-82:

// src/runtime/mgc.go, lines 24-82

// 1. GC performs sweep termination.
//    a. Stop the world. This causes all Ps to reach a GC safe-point.
//    b. Sweep any unswept spans.

// 2. GC performs the mark phase.
//    a. Prepare for the mark phase by setting gcphase to _GCmark,
//       enabling the write barrier, enabling mutator assists, and
//       enqueueing root mark jobs. No objects may be scanned until
//       all Ps have enabled the write barrier, which is accomplished
//       using STW.
//    b. Start the world. From this point, GC work is done by mark
//       workers started by the scheduler and by assists performed as
//       part of allocation.
//    c. GC performs root marking jobs. This includes scanning all
//       stacks, shading all globals, and shading any heap pointers in
//       off-heap runtime data structures.
//    d. GC drains the work queue of grey objects, scanning each grey
//       object to black and shading all pointers found in the object.
//    e. Because GC work is spread across local caches, GC uses a
//       distributed termination algorithm to detect when there are no
//       more root marking jobs or grey objects.

// 3. GC performs mark termination.
//    a. Stop the world.
//    b. Set gcphase to _GCmarktermination, and disable workers and
//       assists.
//    c. Perform housekeeping like flushing mcaches.

// 4. GC performs the sweep phase.
//    a. Prepare for the sweep phase by setting gcphase to _GCoff,
//       setting up sweep state and disabling the write barrier.
//    b. Start the world. From this point on, newly allocated objects
//       are white, and allocating sweeps spans before use if necessary.
//    c. GC does concurrent sweeping in the background and in response
//       to allocation.

Phase diagram

   [Normal execution -- gcphase = _GCoff, sweep concurrent]
        |
        v  (GC trigger: heap growth exceeds GOGC target)
   +---------+
   | Phase 1 |  Sweep Termination (STW)
   |  STW    |  - All Ps reach safe-point
   |         |  - Sweep remaining unswept spans
   +---------+
        |
        v
   +---------+
   | Phase 2 |  Mark (CONCURRENT)
   |         |  - Write barrier enabled
   |         |  - Mark workers scan roots and heap
   |         |  - Mutator assists: allocators do marking work
   |         |  - Tri-color invariant maintained
   +---------+
        |
        v
   +---------+
   | Phase 3 |  Mark Termination (STW)
   |  STW    |  - Disable workers
   |         |  - Flush mcaches
   +---------+
        |
        v
   +---------+
   | Phase 4 |  Sweep (CONCURRENT)
   |         |  - gcphase = _GCoff
   |         |  - Write barrier disabled
   |         |  - Spans swept lazily on allocation or by background sweeper
   +---------+
        |
        v
   [Normal execution continues until next GC trigger]

Stop-the-world (STW) pauses

Only phases 1 and 3 require stopping the world. These are typically very short (sub-millisecond on modern Go). The bulk of the work – marking and sweeping – happens concurrently with the application.

11. Tri-Color Marking

Go uses the tri-color abstraction for its concurrent mark phase:

• White: Not yet seen. Presumed garbage at end of marking.
• Grey: Seen but not yet scanned. In the work queue.
• Black: Scanned. All pointers from this object have been traced.

The tri-color invariant

Strong tri-color invariant: No black object may point to a white object.

This invariant must be maintained even as the mutator (application code) runs concurrently with the marker. Without it, the marker might miss reachable objects.

Write barriers

When the mutator writes a pointer during the mark phase, a write barrier executes. Go uses a hybrid write barrier (Dijkstra + Yuasa) that shades both the overwritten pointer and the new pointer value:

From mgc.go, line 42:

The write barrier shades both the overwritten pointer and the new pointer value for any pointer writes.

This means:

• When *p = q is executed during GC, both the old value of *p and the new value q are marked grey.
• This prevents both the “lost update” problem (Dijkstra barrier) and the “premature free” problem (Yuasa barrier).

OS connection: Write barriers are conceptually similar to how OS page table dirty bits track writes for copy-on-write or page-out decisions. Both intercept writes to maintain metadata about memory state. In the GC case, the barrier is implemented in software by the compiler inserting barrier code before pointer writes.

12. Concurrent Sweep

src/runtime/mgc.go lines 84-95:

// src/runtime/mgc.go, lines 84-95

// Concurrent sweep.
//
// The sweep phase proceeds concurrently with normal program execution.
// The heap is swept span-by-span both lazily (when a goroutine needs
// another span) and concurrently in a background goroutine (this helps
// programs that are not CPU bound).
//
// To avoid requesting more OS memory while there are unswept spans,
// when a goroutine needs another span, it first attempts to reclaim
// that much memory by sweeping.

Sweeping is the process of examining each span’s gcmarkBits and freeing unmarked objects. After sweeping, the allocBits for the span are replaced with the gcmarkBits (so marked objects become the new “allocated” set), and gcmarkBits are zeroed for the next cycle.

Two mechanisms drive sweeping:

1. Background sweeper: A dedicated goroutine that sweeps spans one-by-one.
2. Lazy sweeping: When a goroutine needs a new span, it sweeps spans of the required size class first to try to reclaim memory before asking the heap.

The sweepgen field in mspan tracks the sweep state:

src/runtime/mheap.go lines 494-500:

// src/runtime/mheap.go, lines 494-500

// sweep generation:
// if sweepgen == h->sweepgen - 2, the span needs sweeping
// if sweepgen == h->sweepgen - 1, the span is currently being swept
// if sweepgen == h->sweepgen, the span is swept and ready to use
// if sweepgen == h->sweepgen + 1, the span was cached before sweep
//   began and is still cached, and needs sweeping
// if sweepgen == h->sweepgen + 3, the span was swept and then cached
//   and is still cached
// h->sweepgen is incremented by 2 after every GC

13. GC Triggers and Pacing

src/runtime/mgc.go lines 112-118:

// src/runtime/mgc.go, lines 112-118

// GC rate.
// Next GC is after we've allocated an extra amount of memory proportional to
// the amount already in use. The proportion is controlled by GOGC environment
// variable (100 by default). If GOGC=100 and we're using 4M, we'll GC again
// when we get to 8M (this mark is computed by the gcController.heapGoal
// method). This keeps the GC cost in linear proportion to the allocation
// cost. Adjusting GOGC just changes the linear constant
// (and also the amount of extra memory used).

The GC pacer maintains a heap goal: the target heap size at which the next GC cycle should complete. With GOGC=100 (default), the goal is 2x the live heap from the previous cycle. The pacer adjusts the number of GC workers and mutator assists to hit this target.

GOMEMLIMIT (added in Go 1.19) provides a soft memory limit. If set, the GC will trigger more aggressively when approaching the limit, even if the GOGC-based trigger hasn’t fired.

OS connection: GC pacing is analogous to OS memory management policies like the Linux OOM killer threshold or the page replacement daemon (kswapd) that wakes up when free memory drops below a threshold. Both systems try to maintain a target amount of free memory relative to total usage.

14. Summary: The Complete Picture

User code: x := new(MyStruct)
         |
         v
   Compiler: mallocgc(sizeof(MyStruct), type, true)
         |
         v
   +-- Tiny? (<16B, no ptrs) --> bump pointer in 16B block (per-P, no lock)
   |
   +-- Small? (<=32KB) --> mcache.alloc[sizeClass] (per-P, no lock)
   |       |  empty?
   |       v
   |   mcentral (per-size-class, locked)
   |       |  empty?
   |       v
   |   mheap (global, locked)
   |       |  no pages?
   |       v
   |   sysAlloc -> mmap(2) (kernel)
   |
   +-- Large? (>32KB) --> mheap directly
         |  no pages?
         v
     sysAlloc -> mmap(2) (kernel)

   [GC reclaims memory concurrently]
   mark phase:  trace reachable objects (grey -> black)
   sweep phase: free unreachable objects (swap allocBits/gcmarkBits)

Discussion Questions

1. Why does Go use per-P allocation caches (mcache) instead of per-goroutine caches? Consider the number of goroutines vs. the number of Ps.

2. The tiny allocator packs multiple objects into one 16-byte block. What pathological case could this create? How does the “noscan” requirement mitigate the damage?

3. Why does Go’s GC need two STW pauses (sweep termination and mark termination) instead of just one? What would go wrong with a single STW?

4. Compare Go’s size-class allocator with the Linux kernel’s SLAB/SLUB allocator. What structural similarities do you see? What are the key differences?

5. The mspan.sweepgen field uses a generation counter rather than a boolean. Why is a generation counter necessary for concurrent sweeping?

6. How does lazy zeroing (needzero) interact with mmap’s guarantee that new pages are zero-filled? What would happen if the runtime eagerly zeroed all freed memory?

Further Reading

• Source files:
  • /Users/tchajed/sw/go/src/runtime/malloc.go – allocator overview and mallocgc
  • /Users/tchajed/sw/go/src/runtime/mheap.go – mheap, mspan, heapArena structs
  • /Users/tchajed/sw/go/src/runtime/mcache.go – per-P cache
  • /Users/tchajed/sw/go/src/runtime/mgc.go – GC algorithm documentation
  • /Users/tchajed/sw/go/src/runtime/mcentral.go – central span lists
  • /Users/tchajed/sw/go/src/runtime/sizeclasses.go – size class tables
• Bonwick, Jeff. “The Slab Allocator: An Object-Caching Kernel Memory Allocator.” USENIX Summer 1994.
• TCMalloc: http://goog-perftools.sourceforge.net/doc/tcmalloc.html
• Dijkstra, E.W. et al. “On-the-fly garbage collection: an exercise in cooperation.” CACM 1978.

Module 8: Memory Management (mcache, mcentral, mheap, GC phases)

Comprehension Check

Question 1 (Short Answer)

Describe the three-level memory allocation hierarchy in Go: mcache, mcentral, and mheap. For each level, explain what it caches, who accesses it, and what synchronization is needed.

Question 2 (True/False with Explanation)

True or False: The Go garbage collector stops all goroutines for the entire duration of a collection cycle.

Question 3 (Code Reading)

Consider the mheap struct:

type mheap struct {
	_ sys.NotInHeap

	lock mutex

	pages pageAlloc // page allocation data structure

	sweepgen uint32

	allspans []*mspan
	...
}

Why does mheap have the sys.NotInHeap marker? What would go wrong if mheap were heap-allocated?

Question 4 (What Would Happen If…)

What would happen if Go did not have the mcache (per-P cache) and all allocations went directly to mcentral?

Question 5 (Short Answer)

Explain the tri-color abstraction used by Go’s garbage collector. What are white, grey, and black objects, and what invariant must the write barrier maintain?

Question 6 (Short Answer)

What is a “size class” in Go’s memory allocator, and why does Go use approximately 70 size classes rather than allocating exact-sized blocks?

Question 7 (True/False with Explanation)

True or False: Go’s garbage collector can move objects to compact the heap, similar to Java’s G1 or ZGC collectors.

Module 9: Goroutine Stacks (45 min)

Background: Stack Management and the Cost of Concurrency

The call stack is one of the oldest and most fundamental abstractions in computing. Every time a function is called, a new stack frame is pushed onto the stack, holding local variables, the return address, saved registers, and function arguments. When the function returns, its frame is popped. This last-in, first-out (LIFO) discipline is elegant and efficient: allocation is a single pointer decrement (on architectures where the stack grows downward), and deallocation is a pointer increment. There is no fragmentation, no free-list traversal, and no garbage collection – just a stack pointer moving up and down. This simplicity is why nearly every compiled language uses a call stack, and why hardware provides dedicated instructions (CALL, RET, PUSH, POP on x86) to manipulate it.

Operating systems allocate a fixed-size stack for each thread at creation time. On Linux, the default is 8 MB (configurable via ulimit -s or pthread_attr_setstacksize); macOS uses 512 KB for secondary threads; Windows defaults to 1 MB. To detect overflow, the OS places a guard page at the bottom of the stack – a page mapped with no permissions (PROT_NONE). If the stack pointer crosses into the guard page, the hardware triggers a fault, and the OS delivers a signal (SIGSEGV on Unix) or structured exception (on Windows). This mechanism is simple and adds zero runtime overhead to normal execution, since the check is performed by the MMU in hardware. But the fixed-size design forces a difficult trade-off: set the stack too small and deep recursion crashes; set it too large and memory is wasted. Most programs use only a few kilobytes of their multi-megabyte stack, yet the full allocation must be reserved up front. Some systems also deploy stack canaries – sentinel values placed between the return address and local variables that are checked on function return to detect buffer-overflow attacks – but these serve security rather than growth management.

This fixed-size model becomes a serious bottleneck for highly concurrent systems. A server handling 10,000 simultaneous connections with one thread per connection commits 80 GB of virtual address space to stacks alone. At a million connections, the numbers become absurd. Even though modern virtual memory systems use demand paging (physical pages are not allocated until touched), the page table entries, TLB pressure, and kernel bookkeeping for a million threads impose real costs. This is the fundamental tension that drives language runtimes to seek alternatives: how do you support millions of concurrent tasks when each one needs a stack?

Different runtimes have explored different answers. GCC’s split stacks (used by early Go through version 1.3) and similar segmented stack designs allocate small initial stack segments and link new segments on demand. This avoids the upfront memory commitment, but introduces the notorious “hot split” problem: if a function near the segment boundary is called repeatedly in a tight loop, each call allocates a new segment and each return frees it, causing severe thrashing. Erlang’s BEAM VM takes a different approach entirely, giving each process a tiny initial stack (sometimes as small as 233 words) that grows incrementally, enabled by the fact that Erlang’s immutable data model means the stack rarely holds pointers that would complicate relocation. Rust’s async model sidesteps the problem altogether for asynchronous code: async fn compiles to a state machine (a “stackless coroutine”) that captures only the variables live across yield points, requiring no contiguous stack at all. This is memory-efficient but means async and synchronous Rust code use fundamentally different execution models.

Go’s solution, adopted in Go 1.4 and still in use today, is the contiguous growable stack: goroutines start with a 2 KB stack, and when more space is needed, the runtime allocates a new stack of double the size, copies the old stack contents to the new location, adjusts every pointer that referenced the old stack, and frees the old allocation. This approach eliminates the hot split problem (doubling provides amortized O(1) cost per frame), preserves cache locality (the stack remains a single contiguous region), and requires no hardware or OS support beyond ordinary memory allocation. The price is that the compiler must generate precise stack maps – bitmaps that identify which words in every stack frame are pointers – so the runtime knows exactly what to adjust during a copy. This module examines how Go implements this design, from the compiler-inserted prologue checks to the stack copying algorithm and the per-P allocation pools that make it fast.

1. OS Thread Stacks: The Fixed-Size Problem (10 min)

How OS Threads Allocate Stacks

When the OS creates a thread (via pthread_create on Unix or CreateThread on Windows), it allocates a fixed-size stack. Typical defaults:

The stack is allocated as a contiguous block of virtual memory. At the bottom sits a guard page – a page mapped with no permissions (PROT_NONE). If the stack pointer reaches the guard page, the hardware triggers a segfault, which the OS converts into a stack overflow error.

Why Fixed Stacks Are Wasteful

Consider a web server handling 10,000 concurrent connections, one thread per connection:

• At 8 MB per stack: 80 GB of virtual address space committed to stacks
• Most of that stack space is unused – typical functions use only a few KB
• Even with virtual memory (demand paging), the TLB and page table overhead is significant
• Thread creation is expensive: the kernel must set up the stack, allocate a task struct, and perform a system call

This is the fundamental reason Go uses goroutines instead of threads: you cannot have millions of OS threads, because the stack memory alone would be prohibitive.

The Goroutine Advantage

Go goroutines start with a 2 KB stack. At that size:

• 1 million goroutines = ~2 GB of stack memory (vs. 8 TB for threads)
• Stacks grow on demand, so most goroutines never allocate more than they need
• Stack allocation is a userspace operation – no system call required

2. Goroutine Stack Layout (10 min)

The stack Struct

Every goroutine’s stack bounds are tracked in the g struct. The stack itself is described by a simple pair of pointers:

// [src/runtime/runtime2.go, lines 462-465](https://cs.opensource.google/go/go/+/refs/tags/go1.26.1:src/runtime/runtime2.go;l=462)
type stack struct {
	lo uintptr
	hi uintptr
}

The stack occupies exactly the memory range [lo, hi). The stack grows downward (toward lower addresses), so hi is the “top” (where execution starts) and lo is the “bottom” (the limit).

Stack Fields in the G Struct

The first fields of the g struct are dedicated to stack management:

// [src/runtime/runtime2.go, lines 473-483](https://cs.opensource.google/go/go/+/refs/tags/go1.26.1:src/runtime/runtime2.go;l=473)
type g struct {
	// Stack parameters.
	// stack describes the actual stack memory: [stack.lo, stack.hi).
	// stackguard0 is the stack pointer compared in the Go stack growth prologue.
	// It is stack.lo+StackGuard normally, but can be StackPreempt to trigger a preemption.
	// stackguard1 is the stack pointer compared in the //go:systemstack stack growth prologue.
	// It is stack.lo+StackGuard on g0 and gsignal stacks.
	// It is ~0 on other goroutine stacks, to trigger a call to morestackc (and crash).
	stack       stack   // offset known to runtime/cgo
	stackguard0 uintptr // offset known to liblink
	stackguard1 uintptr // offset known to liblink
	// ...
}

Key fields:

• stack: The [lo, hi) bounds of the current stack allocation.
• stackguard0: The threshold for the stack growth check. Normally set to stack.lo + StackGuard. The compiler-inserted prologue compares SP against this value. It can also be set to the special value stackPreempt to force a preemption check.
• stackguard1: Used for //go:systemstack functions. Set to ~0 on regular goroutine stacks so that any system stack growth attempt will crash (system stack code should not need to grow).

Stack Guard and StackSmall

The stack layout includes a guard region at the bottom. From the comments in stack.go:

// [src/runtime/stack.go, lines 20-68](https://cs.opensource.google/go/go/+/refs/tags/go1.26.1:src/runtime/stack.go;l=20)
/*
Stack layout parameters.

The per-goroutine g->stackguard is set to point StackGuard bytes
above the bottom of the stack.  Each function compares its stack
pointer against g->stackguard to check for overflow.  To cut one
instruction from the check sequence for functions with tiny frames,
the stack is allowed to protrude StackSmall bytes below the stack
guard.  Functions with large frames don't bother with the check and
always call morestack.  The sequences are (for amd64, others are
similar):

	guard = g->stackguard
	frame = function's stack frame size
	argsize = size of function arguments (call + return)

	stack frame size <= StackSmall:
		CMPQ guard, SP
		JHI 3(PC)
		MOVQ m->morearg, $(argsize << 32)
		CALL morestack(SB)

	stack frame size > StackSmall but < StackBig
		LEAQ (frame-StackSmall)(SP), R0
		CMPQ guard, R0
		JHI 3(PC)
		MOVQ m->morearg, $(argsize << 32)
		CALL morestack(SB)

	stack frame size >= StackBig:
		MOVQ m->morearg, $((argsize << 32) | frame)
		CALL morestack(SB)
*/

The visual layout of a goroutine stack:

    stack.hi  ────────────────────── (top of stack, where execution begins)
              │                    │
              │  active frames     │  ← SP is somewhere in here
              │  (grows downward)  │
              │                    │
              ├────────────────────┤
              │                    │
              │  unused space      │
              │                    │
    guard0    ├────────────────────┤  ← stackguard0 = stack.lo + StackGuard
              │                    │
              │  guard area        │  ← room for nosplit functions + morestack
              │  (StackGuard bytes)│
              │                    │
    stack.lo  ────────────────────── (bottom of stack)

Constants

// [src/runtime/stack.go, lines 70-103](https://cs.opensource.google/go/go/+/refs/tags/go1.26.1:src/runtime/stack.go;l=70)
const (
	stackSystem = goos.IsWindows*4096 + goos.IsPlan9*512 + goos.IsIos*goarch.IsArm64*1024

	// The minimum size of stack used by Go code
	stackMin = 2048

	// stackNosplit is the maximum number of bytes that a chain of NOSPLIT
	// functions can use.
	stackNosplit = abi.StackNosplitBase * sys.StackGuardMultiplier

	// The stack guard leaves enough room for a stackNosplit chain of NOSPLIT calls
	// plus one stackSmall frame plus stackSystem bytes for the OS.
	stackGuard = stackNosplit + stackSystem + abi.StackSmall
)

The initial stack size is stackMin = 2048 bytes (2 KB). The stackSystem constant adds extra space on platforms that need it (4 KB on Windows for signal handling on the goroutine stack, for example).

Preemption via stackguard0

The stackguard0 field serves double duty. Beyond stack overflow detection, it is used for cooperative preemption:

// [src/runtime/stack.go, lines 70-103](https://cs.opensource.google/go/go/+/refs/tags/go1.26.1:src/runtime/stack.go;l=70)
const (
	// Goroutine preemption request.
	// 0xfffffade in hex.
	stackPreempt = uintptrMask & -1314

	// Thread is forking. Causes a split stack check failure.
	// 0xfffffb2e in hex.
	stackFork = uintptrMask & -1234
)

When the scheduler wants to preempt a goroutine, it sets stackguard0 = stackPreempt. This special sentinel value is larger than any real SP, so the next prologue check will always fail, causing a call to morestack, which then checks for preemption rather than actually growing the stack. This is how Go achieves cooperative preemption at function call boundaries (complementing the asynchronous signal-based preemption added in Go 1.14).

3. Stack Growth Mechanism (10 min)

The Prologue Check

The Go compiler inserts a stack growth prologue at the beginning of every function (except those marked //go:nosplit). For a small function on amd64, the generated code is essentially:

CMPQ    g.stackguard0, SP
JHI     morestack_call

If SP has dropped below stackguard0, the function calls morestack, which is an assembly trampoline that saves the current context and calls newstack().

newstack(): The Growth Entry Point

// [src/runtime/stack.go, lines 1014-1026, 1026](https://cs.opensource.google/go/go/+/refs/tags/go1.26.1:src/runtime/stack.go;l=1014)
// Called from runtime·morestack when more stack is needed.
// Allocate larger stack and relocate to new stack.
// Stack growth is multiplicative, for constant amortized cost.
//
// ...
func newstack() {

The newstack function (line 1026) handles both preemption and actual stack growth. After checking for preemption (lines 1093-1146), it doubles the stack size:

// [src/runtime/stack.go, lines 1148-1151](https://cs.opensource.google/go/go/+/refs/tags/go1.26.1:src/runtime/stack.go;l=1148)
	// Allocate a bigger segment and move the stack.
	oldsize := gp.stack.hi - gp.stack.lo
	newsize := oldsize * 2

It then ensures the new size is sufficient for the pending frame:

// [src/runtime/stack.go, lines 1152-1162](https://cs.opensource.google/go/go/+/refs/tags/go1.26.1:src/runtime/stack.go;l=1152)
	// Make sure we grow at least as much as needed to fit the new frame.
	if f := findfunc(gp.sched.pc); f.valid() {
		max := uintptr(funcMaxSPDelta(f))
		needed := max + stackGuard
		used := gp.stack.hi - gp.sched.sp
		for newsize-used < needed {
			newsize *= 2
		}
	}

The goroutine transitions to _Gcopystack status (preventing the GC from scanning it during the copy), and then calls copystack:

// [src/runtime/stack.go, lines 1183-1192](https://cs.opensource.google/go/go/+/refs/tags/go1.26.1:src/runtime/stack.go;l=1183)
	casgstatus(gp, _Grunning, _Gcopystack)
	copystack(gp, newsize)
	casgstatus(gp, _Gcopystack, _Grunning)
	gogo(&gp.sched)

copystack(): The Heart of Stack Growth

The copystack function (line 900) performs the actual stack relocation:

// [src/runtime/stack.go, lines 898-904](https://cs.opensource.google/go/go/+/refs/tags/go1.26.1:src/runtime/stack.go;l=898)
// Copies gp's stack to a new stack of a different size.
// Caller must have changed gp status to Gcopystack.
func copystack(gp *g, newsize uintptr) {
	if gp.syscallsp != 0 {
		throw("stack growth not allowed in system call")
	}
	old := gp.stack

The algorithm:

1. Allocate a new stack of the requested size:

   // line 916
   new := stackalloc(uint32(newsize))
   

2. Compute the adjustment delta (difference between new and old base addresses):

   // lines 925-927
   var adjinfo adjustinfo
   adjinfo.old = old
   adjinfo.delta = new.hi - old.hi
   

3. Adjust sudog pointers – goroutines blocked on channel operations may have pointers into the stack that need updating.

4. Copy the used portion of the stack to the new location:

   // line 956
   memmove(unsafe.Pointer(new.hi-ncopy), unsafe.Pointer(old.hi-ncopy), ncopy)
   

5. Walk all stack frames and adjust every pointer that points into the old stack:

   // lines 974-978
   var u unwinder
   for u.init(gp, 0); u.valid(); u.next() {
       adjustframe(&u.frame, &adjinfo)
   }
   

6. Update the G’s stack fields and free the old stack:

   // lines 968-971
   gp.stack = new
   gp.stackguard0 = new.lo + stackGuard
   gp.sched.sp = new.hi - used
   

Pointer Adjustment in Detail

The adjustpointer function (line 610) checks whether a pointer falls within the old stack bounds and, if so, adjusts it by the delta:

// [src/runtime/stack.go, lines 610-615](https://cs.opensource.google/go/go/+/refs/tags/go1.26.1:src/runtime/stack.go;l=610)
func adjustpointer(adjinfo *adjustinfo, vpp unsafe.Pointer) {
	pp := (*uintptr)(vpp)
	p := *pp
	if adjinfo.old.lo <= p && p < adjinfo.old.hi {
		*pp = p + adjinfo.delta
	}
}

This works because the compiler generates stack maps (pointer bitmaps) for every function at every safe point. These maps tell the runtime exactly which words on the stack are pointers, enabling precise adjustment.

4. Stack Allocation Pools (7 min)

Small Stacks: Per-P Free Lists

For small stacks (up to a few orders above the fixed stack size), the runtime maintains per-P caches for lock-free allocation:

// [src/runtime/stack.go, lines 147-168](https://cs.opensource.google/go/go/+/refs/tags/go1.26.1:src/runtime/stack.go;l=147)
// Global pool of spans that have free stacks.
// Stacks are assigned an order according to size.
//
//	order = log_2(size/FixedStack)
//
// There is a free list for each order.
var stackpool [_NumStackOrders]struct {
	item stackpoolItem
	_    [(cpu.CacheLinePadSize - unsafe.Sizeof(stackpoolItem{})%cpu.CacheLinePadSize) % cpu.CacheLinePadSize]byte
}

type stackpoolItem struct {
	_    sys.NotInHeap
	mu   mutex
	span mSpanList
}

// Global pool of large stack spans.
var stackLarge struct {
	lock mutex
	free [heapAddrBits - gc.PageShift]mSpanList
}

The allocation fast path in stackalloc (line 344) checks the per-P mcache first:

// [src/runtime/stack.go, lines 388-397](https://cs.opensource.google/go/go/+/refs/tags/go1.26.1:src/runtime/stack.go;l=388)
		} else {
			c := thisg.m.p.ptr().mcache
			x = c.stackcache[order].list
			if x.ptr() == nil {
				stackcacherefill(c, order)
				x = c.stackcache[order].list
			}
			c.stackcache[order].list = x.ptr().next
			c.stackcache[order].size -= uintptr(n)
		}

This is the same pattern used for heap allocation (module 8): keep a per-P cache to avoid lock contention. When the local cache is empty, stackcacherefill grabs a batch from the global pool (holding a lock briefly).

Large Stacks: From the Heap

For large stacks (beyond the cached sizes), stackalloc allocates directly from the memory heap:

// [src/runtime/stack.go, lines 405-430](https://cs.opensource.google/go/go/+/refs/tags/go1.26.1:src/runtime/stack.go;l=405)
	} else {
		var s *mspan
		npage := uintptr(n) >> gc.PageShift
		log2npage := stacklog2(npage)

		// Try to get a stack from the large stack cache.
		lock(&stackLarge.lock)
		if !stackLarge.free[log2npage].isEmpty() {
			s = stackLarge.free[log2npage].first
			stackLarge.free[log2npage].remove(s)
		}
		unlock(&stackLarge.lock)

		if s == nil {
			// Allocate a new stack from the heap.
			s = mheap_.allocManual(npage, spanAllocStack)
			if s == nil {
				throw("out of memory")
			}
			osStackAlloc(s)
			s.elemsize = uintptr(n)
		}
		v = unsafe.Pointer(s.base())
	}

stackfree(): Returning to Pools

The stackfree function (line 463) mirrors stackalloc. Small stacks are returned to the per-P cache:

// [src/runtime/stack.go, lines 501-532](https://cs.opensource.google/go/go/+/refs/tags/go1.26.1:src/runtime/stack.go;l=501)
	if n < fixedStack<<_NumStackOrders && n < _StackCacheSize {
		// ...
		if stackNoCache != 0 || gp.m.p == 0 || gp.m.preemptoff != "" {
			lock(&stackpool[order].item.mu)
			stackpoolfree(x, order)
			unlock(&stackpool[order].item.mu)
		} else {
			c := gp.m.p.ptr().mcache
			if c.stackcache[order].size >= _StackCacheSize {
				stackcacherelease(c, order)
			}
			x.ptr().next = c.stackcache[order].list
			c.stackcache[order].list = x
			c.stackcache[order].size += n
		}

Large stacks are returned differently depending on GC phase:

// [src/runtime/stack.go, lines 533-555](https://cs.opensource.google/go/go/+/refs/tags/go1.26.1:src/runtime/stack.go;l=533)
	} else {
		s := spanOfUnchecked(uintptr(v))
		if gcphase == _GCoff {
			// Free the stack immediately if we're sweeping.
			osStackFree(s)
			mheap_.freeManual(s, spanAllocStack)
		} else {
			// If the GC is running, we can't return a stack span to the heap
			// because it could be reused as a heap span, and this state
			// change would race with GC. Add it to the large stack cache instead.
			log2npage := stacklog2(s.npages)
			lock(&stackLarge.lock)
			stackLarge.free[log2npage].insert(s)
			unlock(&stackLarge.lock)
		}
	}

5. Stack Shrinking (3 min)

Stacks only grow during execution, but can be shrunk by the garbage collector. The shrinkstack function (line 1257) halves the stack if the goroutine is using less than 1/4 of its allocated space:

// [src/runtime/stack.go, lines 1253-1306](https://cs.opensource.google/go/go/+/refs/tags/go1.26.1:src/runtime/stack.go;l=1253)
// Maybe shrink the stack being used by gp.
func shrinkstack(gp *g) {
	// ...
	oldsize := gp.stack.hi - gp.stack.lo
	newsize := oldsize / 2
	// Don't shrink the allocation below the minimum-sized stack allocation.
	if newsize < fixedStack {
		return
	}
	// Compute how much of the stack is currently in use and only
	// shrink the stack if gp is using less than a quarter of its
	// current stack.
	avail := gp.stack.hi - gp.stack.lo
	if used := gp.stack.hi - gp.sched.sp + stackNosplit; used >= avail/4 {
		return
	}
	// ...
	copystack(gp, newsize)
}

Stack shrinking reuses the same copystack mechanism as growth – allocate a smaller stack, copy the contents, adjust pointers. Shrinking can only happen at safe points (not during syscalls, not at asynchronous safe points, not while parking on a channel).

The preemptShrink flag in the G struct allows the GC to request that a goroutine shrink its stack at the next synchronous safe point:

// [src/runtime/stack.go, lines 1130-1135](https://cs.opensource.google/go/go/+/refs/tags/go1.26.1:src/runtime/stack.go;l=1130)
		if gp.preemptShrink {
			// We're at a synchronous safe point now, so
			// do the pending stack shrink.
			gp.preemptShrink = false
			shrinkstack(gp)
		}

6. Segmented Stacks vs. Contiguous Stacks (5 min)

The Segmented Stack Era (Go 1.0 - 1.3)

Early versions of Go used segmented stacks (also called “split stacks”). When a goroutine needed more stack space:

1. A new segment was allocated (not necessarily adjacent to the old one)
2. A “stack link” connected the new segment to the old one
3. When the function returned, the segment was freed

Problem: the “hot split”. Consider a function right at the stack boundary that is called in a tight loop. Each call allocates a new segment; each return frees it. This thrashing destroyed performance for certain workloads.

The Contiguous Stack Solution (Go 1.4+)

Go 1.4 switched to contiguous stacks (also called “stack copying”). Instead of linking segments:

1. Allocate a new, larger, contiguous stack (2x the old size)
2. Copy the entire old stack to the new one
3. Walk all frames and adjust every pointer into the stack
4. Free the old stack

Advantages:

• No hot split problem – growth is amortized (doubling means O(log n) copies)
• Better cache locality – the entire stack is contiguous
• Simpler runtime – no need to manage linked segments or handle cross-segment returns

Requirement:

• The runtime must have precise pointer maps for every stack frame, so it knows which words are pointers and need adjustment. This is why Go compiles with full stack map information – it is not optional.

Why Not Virtual Memory?

One might ask: why not just allocate a huge virtual address range and let the OS page in memory on demand? This is what Go’s g0 (system) stacks and OS threads do. For goroutines, this approach fails because:

• Each goroutine would need a reserved virtual address range (e.g., 1 MB)
• With 1 million goroutines, that is 1 TB of virtual address space
• Page table entries, TLB pressure, and mmap overhead make this impractical
• On 32-bit systems (historically relevant), address space is far too limited

Summary

Key Takeaways

1. 2 KB initial stacks make goroutines cheap to create – 4000x less memory than a typical OS thread stack.
2. Compiler-inserted prologues check SP against stackguard0 at every function entry, making stack growth transparent to user code.
3. Contiguous stack copying with pointer adjustment is the mechanism that makes growable stacks possible. It requires precise pointer maps from the compiler.
4. Per-P stack caches make allocation and deallocation fast, following the same pattern as the memory allocator (module 8).
5. stackguard0 serves double duty: stack overflow detection and cooperative preemption, unifying two mechanisms into one check.

Discussion Questions

1. What happens if a goroutine calls a C function via cgo? Can the goroutine’s stack be moved while C code holds pointers into it?
2. Why does copystack need to handle channel sudogs specially? What race condition could occur?
3. The maximum stack size defaults to 1 GB. What would happen if a goroutine legitimately needed more? Should the runtime support larger stacks?

Module 9: Goroutine Stacks (Stack Growth, Copying, Pools)

Comprehension Check

Question 1 (Code Reading)

Study this excerpt from the stack growth prologue description in runtime/stack.go:

guard = g->stackguard
frame = function's stack frame size
argsize = size of function arguments (call + return)

stack frame size <= StackSmall:
    CMPQ guard, SP
    JHI 3(PC)
    MOVQ m->morearg, $(argsize << 32)
    CALL morestack(SB)

stack frame size > StackSmall but < StackBig:
    LEAQ (frame-StackSmall)(SP), R0
    CMPQ guard, R0
    JHI 3(PC)
    MOVQ m->morearg, $(argsize << 32)
    CALL morestack(SB)

Why are there different code sequences for small vs. large stack frames? What optimization does the small-frame case provide?

Question 2 (Code Reading)

Consider the copystack function:

func copystack(gp *g, newsize uintptr) {
	old := gp.stack
	used := old.hi - gp.sched.sp

	// allocate new stack
	new := stackalloc(uint32(newsize))

	// Compute adjustment.
	var adjinfo adjustinfo
	adjinfo.old = old
	adjinfo.delta = new.hi - old.hi

	// Adjust sudogs, synchronizing with channel ops if necessary.
	ncopy := used
	...
	// Copy the stack to the new location
	memmove(unsafe.Pointer(new.hi-ncopy), unsafe.Pointer(old.hi-ncopy), ncopy)

	// Adjust pointers in the new stack.
	adjustctxt(gp, &adjinfo)
	adjustdefers(gp, &adjinfo)
	adjustpanics(gp, &adjinfo)
	adjustsudogs(gp, &adjinfo)
	...
	gp.stack = new
	gp.stackguard0 = new.lo + stackGuard
}

What does adjinfo.delta represent, and why must pointers within the stack be adjusted after copying?

Question 3 (True/False with Explanation)

True or False: Goroutine stacks can only grow, never shrink.

Question 4 (What Would Happen If…)

Go starts goroutine stacks at 2KB (stackMin = 2048). What would happen if the initial stack size were instead set to 1MB (like a typical OS thread stack)?

Question 5 (Short Answer)

Explain how the stack pool works. What are stack pools, and why does the runtime cache freed stacks instead of always returning memory to the heap?

Question 6 (Short Answer)

What is the “hot split” problem that Go’s original segmented stacks had, and how did the switch to contiguous (copyable) stacks solve it?

func outer() {
    for {
        inner() // inner's frame crosses the segment boundary
    }
}

Question 7 (Code Reading)

The constant stackMin is defined as:

const (
	stackMin = 2048
)

And in newstack(), the decision to grow uses doubling:

oldsize := gp.stack.hi - gp.stack.lo
newsize := oldsize * 2

Why doubling? What would be the time complexity of N nested function calls if the stack grew by a fixed additive amount (e.g., +4KB each time) instead of doubling?

Assignment 1: System Call Tracer and Scheduler Event Analyzer

Reinforces: Module 2 (System Calls), Module 3 (Processes, Threads, Goroutines)

Overview

In this assignment, you will build a tool that traces and analyzes the interaction between Go programs and the operating system. You will use Go’s runtime/trace package to capture scheduler events and system call behavior, then build an analyzer that processes the trace data to produce human-readable reports about goroutine lifecycle, system call patterns, and scheduler decisions.

This assignment gives you hands-on experience with the boundary between the Go runtime and the OS kernel, and helps you understand how goroutines map to threads and how system calls affect scheduling.

Learning Objectives

By completing this assignment, you will be able to:

1. Explain how the Go runtime interacts with the kernel via system calls.
2. Use runtime/trace to capture and interpret scheduler events.
3. Identify the relationship between goroutines (Gs), threads (Ms), and processors (Ps) in a running program.
4. Analyze how blocking system calls cause P handoffs and thread creation.
5. Measure and reason about the overhead of system calls on goroutine scheduling.

Specification

Part A: Trace Capture Wrapper (30 points)

Build a package tracer that wraps a workload function and captures a runtime trace.

package tracer

import (
	"os"
	"runtime/trace"
)

// CaptureTrace runs the given function while recording a runtime trace
// to the specified output file. Returns an error if tracing cannot be started.
func CaptureTrace(outputFile string, workload func()) error {
	// TODO: Implement
	// 1. Create the output file
	// 2. Start tracing with trace.Start()
	// 3. Run the workload
	// 4. Stop tracing with trace.Stop()
	// 5. Close the file
	return nil
}

You must also implement three workload functions that exercise different runtime behaviors:

// CPUBoundWorkload spawns N goroutines that each perform pure computation
// (no I/O, no syscalls) for the given duration.
func CPUBoundWorkload(numGoroutines int, duration time.Duration)

// IOBoundWorkload spawns N goroutines that each perform file I/O operations
// (creating temp files, writing, reading, deleting).
func IOBoundWorkload(numGoroutines int, numOps int)

// MixedWorkload spawns goroutines that communicate over channels,
// perform some computation, and do some file I/O.
func MixedWorkload(numProducers, numConsumers int, itemsPerProducer int)

Part B: Trace Analyzer (50 points)

Build a analyzer package that reads a trace file (produced by runtime/trace) and extracts statistics. You will use the internal/trace or golang.org/x/exp/trace package to parse trace events.

Your analyzer must produce a report with the following information:

B.1: Goroutine Statistics

• Total number of goroutines created
• Maximum number of concurrently running goroutines
• Distribution of goroutine lifetimes (min, max, median, p99)
• Number of goroutines that blocked on syscalls vs. channels vs. mutexes

B.2: Thread (M) Statistics

• Maximum number of OS threads active
• Number of thread creations during the trace
• Time spent with threads parked (idle) vs. active

B.3: Syscall Analysis

• Count of distinct syscall-induced scheduling events (goroutine entering/exiting syscall state)
• Number of P handoffs caused by long syscalls
• Average and maximum syscall durations

B.4: Scheduler Event Timeline

• For each P, produce a timeline showing which goroutines ran on it and when context switches occurred
• Identify the longest uninterrupted goroutine execution (potential scheduling latency issue)

package analyzer

// Report contains the analyzed trace statistics.
type Report struct {
	Goroutines   GoroutineStats
	Threads      ThreadStats
	Syscalls     SyscallStats
	Timeline     []PTimeline
}

// Analyze reads a trace file and produces a Report.
func Analyze(traceFile string) (*Report, error) {
	// TODO: Implement
	return nil, nil
}

// String returns a human-readable summary of the report.
func (r *Report) String() string {
	// TODO: Implement
	return ""
}

Part C: Comparative Analysis Report (20 points)

Write a short report (1-2 pages) that:

1. Runs each of the three workloads from Part A with at least two different values of GOMAXPROCS (e.g., 1, 4, and the number of CPUs).
2. Analyzes the traces using your Part B analyzer.
3. Answers these questions with evidence from your traces:
   • How does the number of OS threads compare to GOMAXPROCS for each workload type? Why?
   • In the I/O-bound workload, how many P handoffs occurred? What triggered them?
   • In the CPU-bound workload, did any goroutine run for more than 10ms without being preempted? If so, explain why.
   • How does channel communication in the mixed workload affect goroutine scheduling (look at runnext usage)?

Starter Code Structure

assignment1/
├── cmd/
│   └── tracer/
│       └── main.go          # CLI tool: capture trace and/or analyze
├── tracer/
│   ├── capture.go           # CaptureTrace function
│   └── workloads.go         # Three workload functions
├── analyzer/
│   ├── analyzer.go          # Trace parsing and analysis
│   ├── report.go            # Report struct and formatting
│   └── analyzer_test.go     # Tests with known trace files
├── go.mod
└── report/
    └── analysis.md          # Your Part C report

Grading Rubric

Hints

1. Start with go tool trace to visually inspect your trace files before writing the analyzer. This gives you ground truth to validate against.

2. The runtime/trace package is straightforward for capture. The harder part is parsing. Look at golang.org/x/exp/trace for a stable parsing API, or study how cmd/trace parses events internally.

3. For the I/O workload, use os.CreateTemp, os.WriteFile, and os.ReadFile. These will trigger real blocking syscalls (unlike network I/O, which uses the poller).

4. To detect P handoffs in your analyzer, look for events where a goroutine’s P assignment changes between entering and exiting a syscall.

5. For the mixed workload, use buffered and unbuffered channels to see different scheduling behaviors.

6. Run with GODEBUG=schedtrace=1000 alongside your tracer to cross-reference scheduler state.

7. The runtime.NumGoroutine() and runtime.GOMAXPROCS(0) functions can help instrument your workloads.

8. Keep your workloads running for at least 1-2 seconds to get meaningful trace data, but not so long that trace files become unmanageably large.

Assignment 2: Simplified Goroutine Scheduler

Reinforces: Module 4 (The Go Scheduler), Module 5 (Work Stealing and Preemption)

Overview

In this assignment, you will implement a simplified version of Go’s GMP scheduler as a user-space library. You will use real OS threads as Ms, a struct with a run queue as P, and closures as Gs. Your scheduler must support local run queues, a global run queue, work stealing between Ps, and a basic preemption mechanism.

This hands-on implementation will solidify your understanding of why the GMP model exists, the tradeoffs in scheduling design, and the subtlety of concurrent data structures in the scheduler.

Learning Objectives

By completing this assignment, you will be able to:

1. Implement the GMP scheduling model with correct concurrency control.
2. Build a lock-free (or low-lock) work queue with work stealing.
3. Implement cooperative and timer-based preemption of tasks.
4. Reason about thread parking, spinning, and wakeup strategies.
5. Measure and analyze the performance characteristics of your scheduler.

Specification

Core Types

package scheduler

import (
	"sync"
	"sync/atomic"
	"time"
)

// Task represents a unit of work (analogous to a G).
type Task struct {
	id       int64
	fn       func()          // The function to execute
	state    int32           // TaskRunnable, TaskRunning, TaskDone
	// Add fields as needed for preemption, scheduling metadata, etc.
}

const (
	TaskRunnable = iota
	TaskRunning
	TaskDone
)

// Processor represents a scheduling context (analogous to a P).
type Processor struct {
	id       int
	runq     [256]*Task      // Local circular run queue
	runqHead uint32
	runqTail uint32
	runnext  atomic.Pointer[Task]  // Fast-path for producer-consumer patterns

	// Add fields as needed
}

// Machine represents a worker thread (analogous to an M).
// Each Machine runs on its own OS goroutine (acting as a thread).
type Machine struct {
	id       int
	proc     *Processor      // Currently attached P (may be nil)
	spinning bool

	// Add fields as needed
}

// Scheduler is the global scheduler state (analogous to the sched struct).
type Scheduler struct {
	processors []*Processor
	globalRunq []*Task        // Global run queue (protected by lock)
	globalLock sync.Mutex

	// Add fields for idle machines, spinning count, etc.
}

Required Functions

Part A: Basic Scheduling (35 points)

Implement the core scheduling loop:

// NewScheduler creates a scheduler with the given number of Ps.
// It starts one M (OS goroutine) per P.
func NewScheduler(numProcessors int) *Scheduler

// Spawn adds a new task to the scheduler. It should be placed on the
// current P's local run queue if called from a worker, or on the
// global run queue otherwise. If a local queue is full, shed half
// to the global queue (like runqputslow).
func (s *Scheduler) Spawn(fn func())

// schedule is the main scheduling loop for each M.
// It finds a runnable task and executes it, then repeats.
// It should check: (1) local run queue, (2) global run queue
// (every 61st tick), (3) try work stealing.
func (m *Machine) schedule()

// findRunnable finds the next task to run, blocking if necessary.
func (m *Machine) findRunnable() *Task

// Shutdown gracefully stops the scheduler after all tasks complete.
func (s *Scheduler) Shutdown()

Your schedule() function must:

1. Check runnext first (fast path for communicating tasks).
2. Check the local run queue.
3. Check the global run queue every 61st scheduling tick.
4. If no local work, attempt work stealing.
5. If no work anywhere, park the machine (block on a condition variable or channel).

Part B: Work Stealing (30 points)

Implement work stealing between processors:

// runqSteal steals half the tasks from another processor's run queue
// and puts them in the caller's run queue. Returns the first stolen
// task (to execute immediately), or nil if nothing was stolen.
func (p *Processor) runqSteal(victim *Processor) *Task

// stealWork tries to steal from a random processor.
// Makes up to 4 attempts with randomized victim selection.
func (m *Machine) stealWork() *Task

Requirements for work stealing:

• Steal half the victim’s run queue (not all of it).
• Use atomic operations for the lock-free run queue (or a mutex if you prefer, but document the tradeoff).
• Randomize the starting victim to avoid contention.
• On the last attempt, also try to steal from runnext.
• Track stealing statistics (number of steal attempts, successes, tasks stolen).

Part C: Preemption (20 points)

Implement a preemption mechanism:

// Each task gets a time quantum. If a task runs longer than the quantum,
// it should be preempted and placed back on the run queue.
const TimeQuantum = 10 * time.Millisecond

Since you cannot preempt arbitrary Go code mid-execution (that would require signal handling), implement cooperative preemption with a check function:

// ShouldYield returns true if the current task has exceeded its time quantum.
// Long-running tasks should call this periodically.
func (s *Scheduler) ShouldYield() bool

// Yield voluntarily yields the current task's execution, placing it back
// on the run queue. Returns when the task is rescheduled.
func (s *Scheduler) Yield()

Also implement a monitor goroutine (analogous to sysmon) that:

• Runs on its own goroutine (not tied to a P).
• Periodically checks for tasks running longer than TimeQuantum.
• Sets a preemption flag on long-running tasks.
• Detects and reports if any processor has been idle for too long while work exists elsewhere.

Part D: Performance Evaluation (15 points)

Write benchmarks that compare your scheduler against Go’s native goroutine scheduler:

// BenchmarkFanOut creates N tasks that each spawn M subtasks.
// Compare: your scheduler vs. native goroutines.
func BenchmarkFanOut(b *testing.B)

// BenchmarkPingPong creates pairs of tasks that alternate execution
// (simulating channel-like communication).
func BenchmarkPingPong(b *testing.B)

// BenchmarkWorkStealingEfficiency creates an imbalanced workload
// (all tasks initially on one P) and measures how quickly
// work is distributed.
func BenchmarkWorkStealingEfficiency(b *testing.B)

Write a short analysis (in comments or a separate file) discussing:

• How does your scheduler’s throughput compare to Go’s? Why?
• What is the overhead of your work stealing vs. Go’s lock-free implementation?
• How effective is your preemption mechanism?

Starter Code Structure

assignment2/
├── scheduler/
│   ├── scheduler.go         # Scheduler, NewScheduler, Spawn, Shutdown
│   ├── processor.go         # Processor, run queue operations
│   ├── machine.go           # Machine, schedule(), findRunnable()
│   ├── stealing.go          # Work stealing implementation
│   ├── preemption.go        # Preemption and sysmon-like monitor
│   ├── scheduler_test.go    # Correctness tests
│   └── scheduler_bench_test.go  # Performance benchmarks
├── cmd/
│   └── demo/
│       └── main.go          # Demo program showing the scheduler in action
└── go.mod

Grading Rubric

Hints

1. Start simple. Get Part A working with mutexes everywhere before optimizing. A correct mutex-based scheduler is worth more than a buggy lock-free one.

2. Use Go goroutines as your “OS threads.” Each Machine’s schedule() loop runs in its own goroutine, which Go maps to an OS thread. This is a legitimate simplification – you are building a scheduler on top of Go’s scheduler, focusing on the algorithm rather than the OS interface.

3. For task preemption, since you cannot interrupt a running closure, have your long-running tasks call s.ShouldYield() in their inner loops. The monitor goroutine sets a flag; ShouldYield() checks it. This mirrors Go’s cooperative preemption at function prologues.

4. Machine parking: Use sync.Cond or a channel to park idle machines. When Spawn() adds work, signal a parked machine to wake up. Be careful about the “lost wakeup” problem (the same issue futexsleep solves with its atomic check).

5. For the lock-free run queue, study how runtime/proc.go implements runqput, runqget, runqgrab using atomic.LoadAcq and atomic.StoreRel. The key insight is that the queue is single-producer (owning P) and multi-consumer (stealing Ps), which simplifies the protocol.

6. The 61-tick global queue check: Use an atomic counter on each Processor. Every 61st call to findRunnable, check the global queue first. This prevents starvation of globally-queued tasks.

7. Testing concurrency: Use go test -race religiously. Your scheduler is a concurrent data structure and race conditions will be subtle. Write tests that spawn thousands of tasks to surface races.

8. Spinning threads: Start without the spinning optimization. Once basic stealing works, add it: a machine that has no local work and is searching for work is “spinning.” Limit the number of spinning machines to avoid wasting CPU. When you Spawn(), only wake a parked machine if there are no spinning machines.

Assignment 3: Concurrent Channel Implementation

Reinforces: Module 6 (Synchronization Primitives), Module 7 (Channels and Select)

Overview

In this assignment, you will implement a channel-like concurrent communication primitive from scratch, using only mutexes and condition variables (no Go channels allowed in the implementation). Your channel must support unbuffered and buffered modes, blocking send/receive, non-blocking try-send/try-receive, and a simplified select-like multiplexing operation.

This assignment will give you deep understanding of how channels work internally – the waiting queues, the direct-send optimization, lock ordering, and the subtleties of blocking and waking goroutines.

Learning Objectives

By completing this assignment, you will be able to:

1. Implement blocking synchronization using mutexes and condition variables.
2. Build a circular buffer with correct concurrent access.
3. Handle the direct-send/receive optimization for unbuffered channels.
4. Implement lock ordering to prevent deadlocks in multi-channel operations.
5. Reason about memory visibility, lost wakeups, and spurious wakeups.

Specification

Core Type

package channel

import (
	"sync"
	"errors"
)

var (
	ErrClosed    = errors.New("send on closed channel")
	ErrEmpty     = errors.New("receive on empty closed channel")
)

// Chan is a typed channel supporting buffered and unbuffered operation.
// The type parameter T is the element type.
type Chan[T any] struct {
	// TODO: Add fields
	// Consider: mutex, circular buffer, send/recv wait queues,
	// closed flag, capacity, count, etc.
}

// New creates a new channel with the given buffer capacity.
// A capacity of 0 creates an unbuffered channel.
func New[T any](capacity int) *Chan[T]

Required Operations

Part A: Buffered Channel (30 points)

// Send sends a value on the channel. Blocks if the buffer is full.
// Panics if the channel is closed.
func (c *Chan[T]) Send(val T)

// Recv receives a value from the channel. Blocks if the buffer is empty.
// Returns the value and true if successful, or the zero value and false
// if the channel is closed and empty.
func (c *Chan[T]) Recv() (T, bool)

// Close closes the channel. After closing, Send will panic.
// Recv will return remaining buffered values, then (zero, false).
// Close may be called only once; subsequent calls panic.
func (c *Chan[T]) Close()

// Len returns the number of elements currently in the buffer.
func (c *Chan[T]) Len() int

// Cap returns the channel's buffer capacity.
func (c *Chan[T]) Cap() int

Requirements:

• The buffer must be a circular queue (not a slice that grows/shrinks).
• Send must block (not spin) when the buffer is full.
• Recv must block (not spin) when the buffer is empty.
• After Close(), Recv must drain remaining buffer contents before returning (zero, false).
• Calling Send on a closed channel must panic (matching Go’s behavior).
• Must be safe for concurrent use by multiple goroutines.

Part B: Unbuffered Channel (25 points)

Your channel with capacity == 0 must behave as an unbuffered (synchronous) channel:

• Send blocks until a corresponding Recv is ready.
• Recv blocks until a corresponding Send is ready.
• When both are ready, the value is transferred directly from sender to receiver (the direct send optimization – no intermediate buffer copy).
• A Send and Recv that rendezvous must both unblock.

// For unbuffered channels, implement a handshake mechanism:
// the sender waits until a receiver is available, then transfers
// the value directly and both proceed.

You must implement waiting queues for blocked senders and receivers (analogous to sendq and recvq in the Go runtime’s hchan):

// waiter represents a blocked goroutine waiting to send or receive.
type waiter[T any] struct {
	val    T            // For senders: the value to send.
	                    // For receivers: where to store the received value.
	done   chan struct{} // Signal that the operation completed.
	                    // (This is the ONE place you may use a Go channel --
	                    //  as a goroutine wake-up mechanism analogous to gopark/goready.
	                    //  Alternatively, use sync.Cond.)
}

Part C: Non-blocking Operations (15 points)

// TrySend attempts to send without blocking.
// Returns true if the value was sent, false if the channel is full or closed.
func (c *Chan[T]) TrySend(val T) bool

// TryRecv attempts to receive without blocking.
// Returns the value, true, true if a value was received.
// Returns zero, true, false if the channel is closed and empty.
// Returns zero, false, false if the channel is empty but open (would block).
func (c *Chan[T]) TryRecv() (val T, ok bool, received bool)

These must be truly non-blocking: they acquire the lock, check the state, and return immediately.

Part D: Select (Multiplexing) (20 points)

Implement a simplified Select that waits on multiple channel operations:

// SelectCase describes a single case in a Select operation.
type SelectCase[T any] struct {
	Chan      *Chan[T]
	Dir       SelectDir      // SelectSend or SelectRecv
	SendVal   T              // Value to send (for SelectSend)
}

type SelectDir int

const (
	SelectSend SelectDir = iota
	SelectRecv
)

// SelectResult contains the outcome of a Select.
type SelectResult[T any] struct {
	Index    int    // Which case was selected
	Value    T      // Received value (for SelectRecv cases)
	Ok       bool   // false if channel was closed (for SelectRecv)
}

// Select waits until one of the given cases can proceed.
// If multiple cases are ready, one is chosen at random.
// Select blocks until at least one case is ready (no default case).
func Select[T any](cases []SelectCase[T]) SelectResult[T]

Requirements:

• Lock ordering: Acquire channel locks in address order to prevent deadlocks (matching selectgo’s approach).
• Random polling: Check cases in a random order to prevent starvation.
• Correct blocking: If no case is ready, enqueue the goroutine on ALL channels’ wait queues. When one operation completes, dequeue from all others.
• Handle the case where the same channel appears in multiple cases.

Part E: Testing and Verification (10 points)

Write comprehensive tests:

// TestBufferedBasic - single producer, single consumer with buffered channel.
// TestUnbufferedRendezvous - verify sender and receiver synchronize.
// TestConcurrentSendRecv - many goroutines sending and receiving simultaneously.
// TestCloseSemantics - verify close behavior matches Go's channels.
// TestSelectBasic - select with multiple channels, verify fairness.
// TestSelectDeadlock - verify select with all-blocked channels does not spin.
// TestDirectSend - verify unbuffered channel copies directly (measure allocation?).
// TestStress - 1000 goroutines, mix of operations, run with -race.

Constraints

• No Go channels in your implementation (except: you may use a chan struct{} inside the waiter struct as a goroutine wake-up mechanism, since you cannot call gopark/goready from user code. Alternatively, use sync.Cond).
• No busy-waiting / spinning. All blocking must use sync.Cond.Wait() or equivalent.
• No unsafe package. Use generics for type safety.
• Must pass go test -race with no data races.

Starter Code Structure

assignment3/
├── channel/
│   ├── chan.go              # Chan struct, New, Send, Recv, Close
│   ├── unbuffered.go        # Unbuffered channel specialization
│   ├── nonblocking.go       # TrySend, TryRecv
│   ├── select.go            # Select implementation
│   ├── chan_test.go          # Correctness tests
│   └── chan_bench_test.go    # Performance benchmarks
├── cmd/
│   └── demo/
│       └── main.go          # Demo: producer-consumer, fan-in, pipeline
└── go.mod

Grading Rubric

Hints

1. Start with the buffered channel. Get Send/Recv/Close working for capacity > 0 before tackling unbuffered. The buffered case is simpler because the buffer decouples senders from receivers.

2. Use sync.Cond for blocking. Create two condition variables on the same mutex: one for “buffer not full” (senders wait on this) and one for “buffer not empty” (receivers wait on this). Remember to use a for loop around Cond.Wait() to handle spurious wakeups.

3. The unbuffered case is a rendezvous. Think of it as: the sender adds itself to a “waiting senders” queue with its value, then waits. When a receiver arrives, it takes the value directly from the sender’s waiter struct, signals the sender, and both proceed. Vice versa if the receiver arrives first.

4. For direct send in unbuffered channels, study how runtime/chan.go’s send() function copies directly to the receiver’s memory via sendDirect. Your version will copy from the waiter.val field to the receiver’s return value.

5. Lock ordering for Select: Sort the cases by &c (the channel pointer converted to uintptr) before acquiring locks. This is exactly what selectgo does with lockorder. After sorting, lock all channels, check each case in random order, and if none are ready, enqueue on all channels before releasing locks.

6. The hardest bug will be lost wakeups. When a receiver arrives and finds the buffer empty:

   • It must atomically (under the lock) check the buffer AND add itself to the wait queue.
   • If it checks, finds empty, releases the lock, then enqueues, a sender could deposit a value and signal between the check and the enqueue, causing a lost wakeup.

7. Testing concurrent close: Close must wake ALL blocked senders (they should panic) and ALL blocked receivers (they should return zero, false). Use sync.Cond.Broadcast() for this.

8. Benchmarking: Compare your channel against Go’s built-in channels for single-producer-single-consumer and multi-producer-multi-consumer patterns. Expect your implementation to be 5-20x slower (Go’s channels use runtime internals like gopark and direct stack manipulation that you cannot access). Analyze why in your benchmark comments.

Final Exam: Operating Systems Through the Go Runtime

Duration: 3 hours
Total Points: 100
Open book: You may refer to the Go source code and course materials.

Part A: Concept Questions (40 points, 4 points each)

Answer each question in 3-5 sentences.

A1 (Modules 1-2)

The Go runtime makes system calls directly via assembly (e.g., sys_linux_amd64.s) rather than calling through libc. Explain one situation where this causes a compatibility problem and one situation where it provides a significant advantage.

A2 (Modules 1-2)

When a goroutine makes a blocking system call, the runtime calls entersyscall() before the call and exitsyscall() after it returns. What would happen if these calls were omitted? Describe the impact on both the specific goroutine and the overall scheduler.

A3 (Modules 3-4)

Explain why the Go scheduler uses an M:N threading model rather than a 1:1 model (one goroutine per OS thread) or an N:1 model (all goroutines on one OS thread). What specific limitations of 1:1 and N:1 does M:N overcome?

A4 (Modules 3-4)

The g0 goroutine exists on every M and has a larger, fixed-size stack. Why can’t the scheduler run on a regular goroutine’s stack? Give two specific operations that require g0.

A5 (Modules 5-6)

The Go scheduler limits the number of “spinning” threads. Explain the tradeoff: what happens if no spinning threads are allowed? What happens if the limit is removed and all idle threads spin?

A6 (Modules 5-6)

Compare how Go’s sync.Mutex and the runtime’s internal runtime.mutex handle contention. What happens when a goroutine blocks on each? Why are two different mutex implementations necessary?

A7 (Modules 7-8)

When sending a value on an unbuffered channel and a receiver is already waiting, Go copies the value directly from the sender’s stack to the receiver’s stack. Explain why this is safe even though writing to another goroutine’s stack is normally forbidden.

A8 (Modules 7-8)

Explain why Go’s garbage collector must use a write barrier during the concurrent mark phase. Give a concrete example of an object that would be incorrectly freed without it.

A9 (Modules 9-10)

A goroutine’s stack starts at 2KB and doubles when it needs to grow. Explain the amortized cost argument for why this is efficient. What is the total copy work done for a goroutine that ultimately needs a 64KB stack?