The Evolution of the Go Runtime and the Constraints That Matter in Practice

Surveying the scheduler, GC, and stacks along the version axis


Posted on Sun, Aug 2, 2026
Tags golang, runtime, gc, scheduler, performance, cowork-with-llm
golang, runtime, gc, scheduler, performance, cowork-with-llm
πŸ“ This article is a translation of the original Japanese post. View original

The Evolution of the Go Runtime and the Constraints That Matter in Practice

Go is a language with a small specification but a thick runtime. That goroutines are lightweight, and that GC pause times are short, is supported by the runtime’s implementation rather than the compiler.

And the runtime has changed a great deal from one Go version to the next. Go 1.0’s GC and Go 1.26’s GC are different things, and writing code while still dragging along the conventional wisdom of that era means applying optimizations that are meaningless on today’s runtime, or conversely stepping into pitfalls you needn’t step into.

This article follows how code becomes a binary and starts running on the OS, organizes the components of the Go runtime, and lays out the changes per version in chronological order. It then goes down to how the runtime uses CPU, memory, storage, and NIC, and distills all of that into 13 constraints answering “so what should I watch out for when writing code?” All the figures run in the browser, so please read along with your hands on them.

Summary

Here are the key points up front.

  • The core of the Go runtime is five things: the scheduler (G-P-M), the GC, the memory allocator, stack management, and the netpoller/timers
  • The runtime is statically bundled into the binary, and by the time you reach main.main the scheduler and GC are already running
  • The runtime’s design philosophy boils down to two things: “avoid system calls as much as possible, and where unavoidable, confine the impact to one thread” and “stop sharing and localize per resource
  • The direction of evolution has consistently moved from “reducing pause time” to “securing predictability,” and then to “fitting container environments
  • The turning points that matter in practice are Go 1.5 (concurrent GC), Go 1.8 (hybrid write barrier), Go 1.14 (asynchronous preemption), Go 1.19 (GOMEMLIMIT), and Go 1.25 and 1.26 (cgroup support and Green Tea GC)
  • The constraints to watch out for in current Go have shifted from GC pause time to allocation volume, live heap, and goroutine lifecycle
  • Test environment: Go 1.26.5 (linux/amd64). Version-dependent statements are called out each time

1. What Does “the Go Runtime” Refer To?

A Go executable statically links the implementation of the runtime package in addition to the code the user wrote. This is what is generally called the Go runtime; it’s not a separate-process virtual machine like the JVM. Its scope of responsibility is as follows.

ComponentResponsibilitySymptoms it mainly affects
Scheduler (G-P-M)Assigns goroutines to OS threadsLatency variance, inability to use up the CPU
Garbage collectorReclaims unreachable objectsCPU utilization, memory usage, tail latency
Memory allocatorHeap acquisition by size classAllocation speed, fragmentation, RSS
Stack managementVariable-length stack per goroutinegoroutine creation cost, stack overflow
netpoller and timersI/O multiplexing and time eventsConnection count scaling, timer precision
Execution tracing and metricsVisibility into internal stateEase of incident analysis

From the perspective of someone writing Go, these are things that get taken care of automatically β€” but the range of what gets taken care of has a specification and limits. Following how those limits shifted with each version is the subject of this article.

2. From Compilation to Execution

Grasping first when the runtime enters the binary and in what order it starts running makes the rest connect more easily. Select a stage and it shows what the runtime is doing in that step, along with the command to check it yourself.

This figure is drawn with JavaScript. Enable JavaScript to explore it interactively.

There are three key points.

1. The compiler and the runtime are designed as a pair

Syntax like go f() and m[k] = v is converted into calls to the corresponding runtime functions. The GC’s write barrier and the stack-growth check are also embedded into each function by the compiler. You can actually confirm this.

1
$ go tool objdump -s "main\.spawn$" ./app | grep -oE "runtime\.[a-zA-Z0-9_]+" | sort -u
1
2
runtime.morestack_noctxt
runtime.newproc

runtime.newproc is the substance of go f(), and runtime.morestack_noctxt is the stack-growth check inserted at the head of every function. Calls appear even though you wrote not a single line of them.

2. The runtime is statically bundled into the binary

A Go executable becomes a single ELF combining the user’s code, the standard library, and the runtime. This is why no separately installed language implementation is needed and why you can build a container image FROM scratch.

At the same time, this is also why binaries get large. Looking at the section layout under “executable” in the figure above, you can see that .gopclntab is larger than .text. Stack traces, line numbers on panic, and the GC’s stack scan all consult this table.

3. By the time you arrive at main.main, the runtime is already running

Note that the entry point is not main.main. It proceeds from _rt0_amd64_linux through runtime.rt0_go, runtime.schedinit, and runtime.main, runs the packages’ init(), and only then is main.main called. During this, determining GOMAXPROCS, reserving heap arenas, and starting the sysmon thread have all finished.

1
$ GODEBUG=inittrace=1 ./app 2>&1 | head -3
1
2
3
init internal/bytealg @0 ms, 0 ms clock, 0 bytes, 0 allocs
init internal/runtime/gc/scan @0.009 ms, 0 ms clock, 0 bytes, 0 allocs
init runtime @0.014 ms, 0.10 ms clock, 0 bytes, 0 allocs

3. The Overall Picture of the Evolution

First, let’s view the whole thing as a timeline. Clicking a version displays the changes that landed in that release and their impact on the person writing code. Filtering by category lets you follow just the scheduler, or just the GC.

This figure is drawn with JavaScript. Enable JavaScript to explore it interactively.

The broad flow can be divided into five periods.

  1. The C runtime era (1.0–1.3): the runtime core is in C. Stop-the-world GC and segmented stacks
  2. The Go-ification and precise GC era (1.4–1.5): rewrote the runtime in Go, concurrent GC, and automatic GOMAXPROCS
  3. The low-latency era (1.6–1.13): the hybrid write barrier brought GC pauses down to microseconds
  4. The predictability era (1.14–1.21): asynchronous preemption, GOMEMLIMIT, PGO
  5. The container optimization era (1.22–1.27): Swiss Tables, cgroup support, Green Tea GC

From here we dig in component by component.

4. The Scheduler (G-P-M)

4.1 The Structure of the Model

Go’s scheduler is composed of three entities.

  • G (goroutine): the unit of execution. Has a stack and a program counter
  • M (machine): an OS thread. What actually runs on the CPU
  • P (processor): the right to execute Go code. There are as many as GOMAXPROCS, each with a local run queue

For a G to run on the CPU, an M must hold a P. This is the crux of understanding Go’s concurrency. The number of Ps is the upper bound on how many Go code paths can run simultaneously, and the number of Ms grows independently of that. Ms are added by as much as is blocked in syscalls.

You can confirm this by playing with the figure below. Stacking goroutines with go f(), work stealing where a P whose local queue has emptied steals half from another P, the detaching of a P (handoff) when an M is stuck on a blocking syscall, and asynchronous preemption against for{} β€” each logs its reason the moment it occurs.

This figure is drawn with JavaScript. Enable JavaScript to explore it interactively.

The actual runtime behaviors reproduced in the figure are as follows.

  • Local queues hold up to 256. On overflow, half move to the global queue (the figure shrinks this to 16)
  • Once every 61 times it looks at the global queue first. Otherwise, continuing to look only at the local queue would starve goroutines in the global queue
  • Work stealing takes half from another P’s local queue
  • A P is detached from an M blocked in a syscall. Even if one goroutine stops on file I/O, the others keep running
  • Network I/O is handled by the netpoller, so it doesn’t block an M in the first place

4.2 The Evolution of the Scheduler

VersionChangeImpact
1.0GOMAXPROCS defaults to 1Without setting it explicitly, there was no parallel execution
1.1Introduced the G-P-M model and work stealingGlobal lock contention resolved. The prototype of today’s design
1.2Cooperative preemption check on function callsLoops with no function calls still couldn’t be preempted
1.5GOMAXPROCS defaults to the number of logical CPUsMulticore is used without doing anything
1.14Asynchronous preemption (SIGURG), internal timer overhaulThe problem of loops stalling the GC was resolved
1.23Overhaul of Timer and TickerTimers that are no longer referenced are reclaimed immediately
1.24New implementation of runtime-internal mutexesBehavior under high contention improved
1.25GOMAXPROCS recognizes cgroup CPU limitsNo more misreading the logical CPU count in containers
1.26cgo call overhead reduced by about 30%The floor dropped for code that uses cgo heavily

4.3 Constraints for the Person Writing Code

Constraint 1: goroutines don’t stop by themselves, and don’t finish by themselves

The problem of not yielding the CPU was solved by asynchronous preemption, but goroutines that never finish are a different story. A goroutine blocked forever on a channel receive keeps holding memory.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
// bad: this goroutine doesn't return even when ctx is done
go func() {
    for v := range ch { // lives forever unless ch is closed
        process(v)
    }
}()

// good: always provide an exit path
go func() {
    for {
        select {
        case <-ctx.Done():
            return
        case v, ok := <-ch:
            if !ok {
                return
            }
            process(v)
        }
    }
}()

In Go 1.27, a goroutineleak profile that detects goroutines blocked on unreachable synchronization primitives is expected to be promoted to an official feature. Because it uses the GC’s reachability analysis, false positives are unlikely. Until then, substitute monitoring of the goroutine count.

1
2
# watch how the goroutine count moves (usable on current versions too)
curl -s http://localhost:6060/debug/pprof/goroutine?debug=1 | head -20

Constraint 2: GOMAXPROCS can be misread in containers

Before Go 1.24, even with a CPU limit imposed by cgroup, it looked at the host’s logical CPU count. Running a Pod with cpu limit: 2 on a 32-core node gave GOMAXPROCS=32, creating excess Ps and increasing throttling and context switches.

1
2
3
# check the actual values
$ go version
$ GODEBUG=containermaxprocs=0 ./app   # to restore the old behavior on Go 1.25+

From Go 1.25 onward it reads the cgroup CPU bandwidth limit and sets this automatically, and it also follows changes made while running. If you continue to use Go 1.24 or earlier, you need a countermeasure equivalent to automaxprocs. Note that setting it explicitly via an environment variable or runtime.GOMAXPROCS disables the automatic following, so explicit settings should be reserved for when they’re genuinely needed.

Constraint 3: cgo and syscalls leave the scheduler

For the duration of a cgo call, that M leaves the Go scheduler’s management. Making long cgo calls from a large number of goroutines makes Ms keep increasing and the thread count balloon. The default limit is 10000 threads, and exceeding it crashes with runtime: program exceeds 10000-thread limit.

1
2
3
4
// throttle concurrency where you handle cgo or blocking syscalls
sem := make(chan struct{}, 16)
sem <- struct{}{}
defer func() { <-sem }()

5. The Garbage Collector

5.1 The Tri-Color Marking Algorithm

Go’s GC is concurrent mark & sweep, explained with the tri-color abstraction. Tri-color refers to classifying heap objects into the following three states.

  • White: not yet reached. If still white when marking ends, it’s a candidate for reclamation
  • Grey: reached, but the pointers this object holds haven’t been followed yet
  • Black: reached and finished being followed. Confirmed live

The algorithm itself just repeats the following until the grey set (the work list) is empty.

  1. Make the roots (global variables and each goroutine’s stack) grey and push them onto the work list
  2. Take one from the work list, and among the objects it references, make the white ones grey and push them onto the work list
  3. Make the object you took out itself black
  4. Once the work list is empty, reclaim what remains white

The important thing here is that what Go’s GC judges is not “is it referenced?” but “is it reachable from a root?” A group of objects that merely reference each other can’t be reclaimed by reference counting, but judged by reachability they can all be reclaimed together.

The figure below visualizes this search itself. Pressing “advance one step” advances steps 2 and 3 above once and changes the contents of the work list. The part enclosed in a dashed line on the right is garbage in a reference cycle, and it remains white even after the search finishes.

This figure is drawn with JavaScript. Enable JavaScript to explore it interactively.

5.2 Why a Write Barrier Is Needed

The problem lies in the fact that marking runs concurrently with the application. If an already-scanned black object is rewritten to point at an unreached white object, the black won’t be rescanned, so a live white object gets reclaimed without anyone following it.

The conditions for this situation are the following two, and breaking either one makes it safe.

  1. A reference from black to white is created
  2. The path from grey leading to that white is lost

What prevents this is the write barrier, a small hook the runtime inserts on every pointer write. Dijkstra’s scheme (make the write target grey) crushes condition 1, and Yuasa’s scheme (make the old, overwritten referent grey) crushes condition 2; from Go 1.8 onward Go uses a hybrid write barrier combining both.

Pressing “rewrite a pointer during marking” in the figure above draws a red arrow from black to white and lets you watch that white being lifted to grey. Switching to “compare with Go 1.4 and earlier (stop-the-world GC)” lets you confirm that under the stop-the-world scheme nothing could be rewritten concurrently in the first place so no write barrier was needed, and that in exchange the pause time was long.

The practical significance of the hybrid write barrier is that it removed the need to rescan every goroutine’s stack at the end of marking. Before Go 1.7 the pause time grew in proportion to the number of goroutines and the depth of their stacks; from Go 1.8 onward the pause time is roughly constant even with hundreds of thousands of goroutines.

Measured, the mark-termination pause is indeed extremely short.

1
$ GODEBUG=gctrace=1 go run .
1
gc 12 @0.168s 4%: 0.071+1.0+0.091 ms clock, 0.28+0.042/0.67/1.1+0.36 ms cpu, 4->5->1 MB, 5 MB goal, 0 MB stacks, 0 MB globals, 4 P

Here’s how to read it.

  • 0.071+1.0+0.091 ms clock: mark-start STW, concurrent mark, mark-termination STW. What actually stops is only the 0.07ms and 0.09ms at either end
  • 4->5->1 MB: heap at GC start, heap at end, live heap
  • 5 MB goal: the target heap size at which the next GC starts
  • 4%: cumulatively since program start, the GC is using 4% of CPU time

5.3 GOGC and GOMEMLIMIT

When the GC runs is determined by GOGC. The default is 100, meaning “start the next GC once the heap has grown by 100% of what survived the previous GC.”

1
next GC target heap = live heap Γ— (1 + GOGC/100)

If the live heap is 100MiB, by default the next GC starts once it has grown to 200MiB. Raising GOGC reduces the number of GCs but increases memory; lowering it does the opposite.

The problem is that this formula only determines a bound as a ratio against the live heap. If the live heap grows beyond expectations, the target heap grows proportionally, punching through the container’s memory limit and getting OOM-killed. What solved this is Go 1.19’s GOMEMLIMIT, which puts a bound on total memory including stacks and runtime-internal structures, not just the heap.

Moving the sliders on the “GOGC and GOMEMLIMIT” tab of this section’s figure shows how the two determine the target heap. Pressing “keep increasing the live heap” also reproduces the death-spiral state where it pins to the limit and the GC keeps spinning.

The two standard settings in practice are these.

1
2
3
4
5
# Pattern A: leave GOGC at the default and only give a ceiling (for most servers)
GOGC=100 GOMEMLIMIT=1800MiB ./app

# Pattern B: effectively stop the GC and only run it near the ceiling (for CPU-first batch jobs)
GOGC=off GOMEMLIMIT=3GiB ./app

Against a container’s memory limit, set a value with a little headroom, not the limit itself. Beyond Go’s heap, the executable itself, cgo’s memory, and the OS page cache also ride on it.

1
2
3
4
5
6
7
8
// example targeting roughly 90% of the container limit
import "runtime/debug"

func init() {
    if limit := readCgroupMemoryLimit(); limit > 0 {
        debug.SetMemoryLimit(int64(float64(limit) * 0.9))
    }
}

5.4 Green Tea GC (Experimental in Go 1.25, Default in Go 1.26)

Green Tea GC, made the default in Go 1.26, is a scheme that processes the marking of small objects at the granularity of a chunk of memory (a span) rather than per object. Previously, cache misses were likely each time a pointer was followed, but scanning the objects within the same span together improves locality.

Official guidance says GC overhead is reduced by 10–40% in GC-heavy applications, and on newer amd64 (Intel Ice Lake, AMD Zen 4 and later) vector-instruction scanning adds roughly another 10%.

Points to watch when migrating are as follows.

  • GOEXPERIMENT=nogreenteagc reverts to the old GC, but this escape hatch is said to be scheduled for removal in Go 1.27
  • The effect is highly workload-dependent. The more small, pointer-heavy objects you handle in bulk, the more it helps
  • Judge by comparing /gc/cycles/total:gc-cycles and /cpu/classes/gc/total:cpu-seconds before and after
1
2
3
4
5
6
// take GC CPU time via runtime/metrics
samples := []metrics.Sample{
    {Name: "/cpu/classes/gc/total:cpu-seconds"},
    {Name: "/gc/heap/live:bytes"},
}
metrics.Read(samples)

5.5 Constraints for the Person Writing Code

Constraint 4: the GC’s cost is determined by allocation volume, not pause time

Since Go 1.8, pause times are on the order of microseconds and are a non-issue for most uses. What matters now is the CPU time the GC consumes, which is roughly proportional to the number of allocations and the size of the live heap.

In other words, the target of optimization becomes reducing allocations rather than tuning the GC.

1
2
3
# identify the allocation-heavy spots
$ go test -bench=. -benchmem ./...
$ go tool pprof -alloc_objects http://localhost:6060/debug/pprof/allocs

Constraint 5: don’t rely on runtime.SetFinalizer for cleanup

Finalizers have no guaranteed execution timing, and aren’t called when there’s a reference cycle. runtime.AddCleanup, added in Go 1.24, has fewer constraints, but either way a design that explicitly calls Close() is the right answer.

Constraint 6: use sync.Pool on the premise that the GC empties it

The contents of a sync.Pool are thrown away on every GC. Go 1.13 added a victim cache so one cycle’s grace is granted, but you can’t store things long-term. What goes into a pool is a buffer you want to reuse, not a cache.

6. Stacks and Escape Analysis

6.1 Why goroutines Are Lightweight

A goroutine’s initial stack is 2KiB (Go 1.4 onward). Considering that an OS thread’s default stack is several MiB, that’s three orders of magnitude smaller. This is the basis for saying you can stand up 100,000 goroutines and be fine.

What happens when it runs out? From Go 1.3 onward it uses contiguous stacks: allocate a new region twice the size and copy the existing frames wholesale, rewriting pointers on the stack too. This operation is called morestack.

Pressing “call a function” a few times on the left side of the figure below lets you watch a copy occur the moment capacity is exceeded. Pressing “shrink the stack via GC” lets you conversely watch the GC shrinking the stack.

This figure is drawn with JavaScript. Enable JavaScript to explore it interactively.

Since Go 1.19 the initial stack size is determined from the historical average, so workloads that routinely use deep stacks see fewer copies.

6.2 Escape Analysis

Whether a variable lands on the stack or moves to the heap is decided by the compiler’s escape analysis. The decision can be checked with go build -gcflags=-m.

1
$ go build -gcflags=-m . 2>&1 | grep -E "escapes|moved to heap|does not escape"
1
2
3
4
5
6
./main.go:9:26: moved to heap: p
./main.go:11:33: n escapes to heap
./main.go:13:32: make([]byte, 64) does not escape
./main.go:15:35: make([]byte, n) escapes to heap
./main.go:18:2: moved to heap: n
./main.go:19:9: func literal escapes to heap

On the right side of the figure above you can switch between representative cases and check them. The tendencies of the decision are as follows.

CaseResultReason
Return a local valueStackDoesn’t leak outside the function
Return a pointerHeapThe address goes outside the function
Convert to interface{}May become heapMust be held along with dynamic type information
make([]byte, 64)StackSize is fixed at compile time
make([]byte, n)HeapSize is determined at runtime
Captured by a closureHeapThe closure carries the variable forward

6.3 Constraints for the Person Writing Code

Constraint 7: constructors returning pointers produce allocations

func New() *T always results in a heap allocation. If you create many on a hot path, consider a form returning a value (func New() T), or having the caller allocate a slice in bulk. However, this is an optimization to do after measuring; there’s no reason to hurt readability up front.

Constraint 8: slices whose size is determined at runtime escape

When you know the upper bound, taking a fixed-size array and slicing it can sometimes avoid the allocation.

1
2
3
4
5
6
// escapes
buf := make([]byte, n)

// may not escape if you know the upper bound
var arr [256]byte
buf := arr[:n]

Constraint 9: stacks have an upper limit

One goroutine’s stack limit defaults to 1GiB on 64-bit. Exceeding it with deep recursion crashes the process with goroutine stack exceeds 1000000000-byte limit. You can change it with debug.SetMaxStack, but rewriting the recursion as iteration is the more correct first move.

7. The Memory Allocator and Other Pieces

7.1 The Structure of the Allocator

Go’s allocator is a TCMalloc-family design with size classes.

  • mcache: a per-P cache. Allocation without locks
  • mcentral: a shared pool per size class
  • mheap: management of pages obtained from the OS

Small objects (under 32KiB) are rounded to a size class. This is where internal fragmentation like a 17-byte struct consuming 24 bytes arises. It’s also why tidying up struct field order to reduce padding sometimes helps.

1
2
# check struct alignment
$ go run honnef.co/go/tools/cmd/staticcheck@latest ./...

Go 1.26 added a size-specialized fast path for allocation, making small allocations under 512 bytes up to 30% faster. The more your code creates lots of small objects, the more likely you are to see a benefit from upgrading the version.

7.2 The map Implementation Change (Go 1.24)

Go 1.24 replaced the builtin map with a Swiss Tables-based implementation. Lookups and insertions on large maps got faster, and memory efficiency improved as well.

The thing to watch out for is iteration order. Go’s map was already undefined and randomized in order, but with the implementation change the actual ordering changes too. Tests that happen to depend on the incidental order will break with this change.

7.3 Timers and the netpoller

In Go 1.14 the internal timers moved to per-P management, reducing lock contention and context switches. Go 1.23 further overhauled Timer and Ticker, so timers that are no longer referenced are reclaimed immediately without calling Stop().

1
2
3
// from Go 1.23 on, forgetting Stop is less likely to leak. Still, being explicit is recommended
t := time.NewTimer(time.Second)
defer t.Stop()

However, since the timer channel became unbuffered, code depending on the old behavior may need GODEBUG=asynctimerchan=1. This behavioral difference is controlled by the go directive in go.mod.

7.4 Returning Memory to the OS

“I freed memory but RSS doesn’t go down” is a classic inquiry. The cause differs by version.

VersionSchemeSymptom
1.11 and earlierMADV_DONTNEEDSlow return
1.12–1.15MADV_FREERSS doesn’t drop until the kernel reclaims
1.16 onwardMADV_DONTNEEDRSS reflects reality

On Go 1.16 or later this usually isn’t a problem, but it still isn’t returned instantly. If you want to force a return there’s debug.FreeOSMemory(), but it involves a STW so avoid routine use.

8. The Boundary Between the Runtime and the OS/Hardware

So far we’ve looked inside the runtime, but the runtime ultimately runs on top of an OS. The Go runtime’s design philosophy can be summarized as “avoid system calls as much as possible, and where unavoidable, confine their impact to one thread.”

8.1 How Far Down Each Operation Goes

Here’s a diagram of, for one line of Go code, which part of the runtime it passes through and which kernel facility it descends to. Selecting an operation lights up the path. A path that stops in grey means it completes in user space without reaching a system call.

This figure is drawn with JavaScript. Enable JavaScript to explore it interactively.

Tabulating the correspondences gives the following. The facilities used differ by platform, but the roles are the same.

Runtime facilityLinuxmacOS / BSDWindows
Creating an OS threadclone(2)pthread_createCreateThread
Waiting and waking threadsfutex(2)pthread_cond / __ulockEvent objects
I/O multiplexing (netpoller)epoll(7)kqueue(2)IOCP
Reserving and returning virtual memorymmap / madvisemmap / madviseVirtualAlloc
Preemptiontgkill + SIGURGpthread_kill + SIGURGSuspendThread
Getting the timevDSO clock_gettimecommpageQueryPerformanceCounter

8.2 What It Does Well

1. It shifted the unit of blocking from the thread to the goroutine

Waiting on a channel or a mutex merely removes the G from the run queue via gopark, and the M goes on to run a different G. From the kernel’s point of view the thread simply keeps running, and the state of “waiting” doesn’t exist. The cost of waiting fits into a few function calls rather than a context switch.

2. It consolidates the waiting point into one place

The netpoller’s epoll_wait handles waiting on network I/O and waiting on timers at the same time. By passing the time until the next timer fires as epoll_wait’s timeout, it avoids adding a dedicated timer thread or timerfd.

3. It borrows in bulk from the kernel and hands out in small pieces

What it reserves via mmap are arenas in 64MiB units, and actual allocation is carved out of the per-P mcache. Because the number of allocations is decoupled from the number of system calls, even millions of allocations per second don’t enter the kernel.

4. It uses the “syscall-avoidance routes” the kernel prepared

time.Now() calls clock_gettime via the vDSO. The vDSO is a shared code region the kernel maps into each process’s address space, letting you read the time without switching to kernel mode. That sync.Mutex completes with atomic instructions alone when uncontended and only falls to futex when contended is the same idea.

5. It confines unavoidable blocking to one thread

Ordinary file I/O can’t be waited on with epoll, so it actually blocks the whole thread in read(2). So the runtime detaches the P from the M via entersyscall before entering the syscall. Since sysmon hands the detached P to another M, only one thread stops.

8.3 Coordination with Hardware

Descending below the OS to physical resources makes the reasons behind the runtime’s design decisions one step clearer. Everything from goroutines to CPU cores, RAM, storage, and NIC connects vertically. Selecting a piece of hardware displays the path to that resource and how the runtime uses it.

This figure is drawn with JavaScript. Enable JavaScript to explore it interactively.

Summarizing the four pillars in a line each gives the following.

ResourceThe runtime’s responsibilityThe central idea
CPUScheduler (G-P-M)Split run queues and caches per P, reducing state shared between cores
MemoryGC and allocatorReserve virtual addresses in large chunks; allocate physical pages when touched
StorageFile I/OMinimize the pages read at startup via static linking and mmap
NetworknetpollerDecouple connection count from thread count via non-blocking fds and epoll

What they share is the policy of stopping sharing and localizing per resource. The per-P run queue, the per-P mcache, and the per-P timer heap all come from the same idea, and exist to avoid inter-core synchronization and fights over cache lines. Green Tea GC, made the default in Go 1.26, was likewise a change that aligned the scan order with the memory layout to raise cache locality.

8.4 Constraints for the Person Writing Code

Constraint 10: operations that pin OS threads collide with the scheduler

runtime.LockOSThread pins the current G to a specific M. It’s necessary when handling per-thread state, such as an OpenGL context or setns, but that M can no longer run other Gs. If the goroutine finishes while still pinned, that thread is destroyed.

1
2
3
4
5
func mustRunOnOneThread() {
    runtime.LockOSThread()
    defer runtime.UnlockOSThread()
    // for the duration of this block, the G is pinned to the same OS thread
}

Constraint 11: signals are shared with the runtime

The runtime internally uses SIGURG (asynchronous preemption), SIGPROF (CPU profiling), SIGSEGV (detecting nil pointer dereferences), and others. If a library loaded via cgo overwrites these handlers, preemption may stop working or the program may crash.

1
2
# turn off asynchronous preemption to isolate the problem (don't make this a permanent fix)
GODEBUG=asyncpreemptoff=1 ./app

Constraint 12: there are thread count limits on the OS side too

Go’s default limit is 10000 threads, but you may hit cgroup’s pids.max or ulimit -u before that. In designs that make heavy use of blocking syscalls or cgo, it’s safer to explicitly throttle concurrency.

Constraint 13: GOMAXPROCS is not a cap on CPU time

The number of Ps is how many Go code paths can run simultaneously; it’s distinct from the process’s total thread count and it isn’t a CPU quota. Because Ms blocked in syscalls continue to exist without holding a P, even with GOMAXPROCS=2 the thread count can reach dozens. This distinction is needed when looking at cgroup CPU throttling.

1
2
# see the scheduler's actual state once per second
$ GODEBUG=schedtrace=1000 ./app
1
SCHED 1002ms: gomaxprocs=4 idleprocs=0 threads=5 spinningthreads=0 needspinning=1 idlethreads=0 runqueue=30 [2 1 1 2]

threads=5 is the actual OS thread count, runqueue=30 is the global run queue length, and [2 1 1 2] are the per-P local run queue lengths. If runqueue stays large all the time, there aren’t enough Ps; if idleprocs is large, you aren’t using up the available parallelism.

9. How to Observe

The means of viewing runtime state have accumulated generation by generation. For new implementations, prefer the newer ones.

MeansUseNotes
GODEBUG=gctrace=1Raw log of GC cyclesStandard error. Don’t use routinely in production
GODEBUG=schedtrace=1000Print scheduler state once per secondShows per-P queue lengths
runtime/metricsObtaining numbers via a stable APIGo 1.16 onward. Preferred over runtime.MemStats
net/http/pprofHeap, CPU, and goroutine profilesCan be collected in production too
runtime/traceExecution tracesOverhead reduced in Go 1.23
runtime/trace.FlightRecorderContinuously retains the most recent traceGo 1.25 onward. Dump only on anomalies
testing/synctestTesting time-dependent concurrent codePromoted to an official feature in Go 1.25
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
// minimal example of runtime/metrics
import "runtime/metrics"

func snapshot() []metrics.Sample {
    samples := []metrics.Sample{
        {Name: "/sched/goroutines:goroutines"},
        {Name: "/gc/heap/live:bytes"},
        {Name: "/cpu/classes/gc/total:cpu-seconds"},
    }
    metrics.Read(samples)
    return samples // when reading, determine the type via s.Value.Kind()
}

10. A Per-Version Checklist

A quick reference for what to watch out for on the Go version you’re currently using.

Version in usePoints to watch especially
1.13 and earlierLoops with no function calls stall the GC. GOMAXPROCS is the host CPU count
1.14–1.18GOGC is the only means of controlling the memory ceiling. Watch for OOM in containers
1.19–1.21Set GOMEMLIMIT. The loop-variable capture bug still exists
1.22–1.23Loop variable behavior changes when go.mod’s go line is 1.22 or higher. Check the impact of the timer overhaul
1.24Tests depending on map iteration order may break
1.25 onwardWatch for double application of automatic GOMAXPROCS and automaxprocs
1.26Green Tea GC is the default. Compare GC metrics before and after migration

11. Caveats

  • Just upgrading the version often makes things faster. Rather than optimizing locally, raising the Go version and measuring first has better cost-effectiveness
  • Watch the GODEBUG compatibility mechanism. From Go 1.21 onward, behavior-changing changes are controlled by the go directive in go.mod. Even after upgrading the toolchain, behavior may stay old until you raise the go line
  • Don’t design around GOEXPERIMENT. Some features, like arenas, remain experimental and are never generally released
  • Take benchmarks with the CPU count fixed. Now that GOMAXPROCS follows the environment automatically, environmental differences show up in results more easily
  • Always measure before changing production GC settings. Raising GOGC reduces CPU but increases memory. Which one is your constraint depends on the workload

References

Mostly primary sources. Please confirm dates and figures in the official release notes.

Share


See also