Ch22.2: Sysroot

1. The Rule

Rule. A toolchain needs a C library for its target. You either provide one in a sysroot, or the target is freestanding and the toolchain must be configured as such. Everything else — the compiler runtime, the C++ standard library, the sanitizers — is provided by GCC or by LLVM.

This is the part that is missing from most cross-compilation tutorials. They tell you to install a compiler and pass --target, and then a link fails with cannot find crt1.o or cannot find -lc. The compiler is not supposed to supply those files. A compiler is a code generator plus a driver; the operating system's ABI lives in the C library and in the start-up objects that come with it.

There is a stronger form of the rule, and it is the one that matters when you bootstrap a new platform: a C/C++ sysroot is the minimum required to build a compiler at all. GCC and Clang are C++ programs. Before either can be built, the build machine needs a working C++ compiler, the C++ standard library, libc, and the platform headers. The compiler you are building may target a freestanding board, but the compiler binary itself still runs on a hosted machine and still links a C++ runtime.

The floor. Every other language toolchain on the platform sits on the C/C++ sysroot. CPython's interpreter and its extension modules are C and link libc; Go's cgo and platform packages use the C ABI and the platform's syscall layer; Rust's standard library links libc and invokes a C linker; Node.js and V8 are C++. None of them defines a second sysroot. If the C/C++ sysroot is missing or wrong, they cannot be built for that platform either.

That is why the first thing a new operating-system port needs is not a Python port or a Rust port. It is a C/C++ toolchain and a sysroot. Once those exist, the other runtimes are ordinary programs built against them, and "porting Python" becomes a build of CPython rather than a new platform definition. The C/C++ layer is the platform contract; everything else is a consumer.

In a Canadian build the same rule applies twice. The compiler binary runs on the host, so the host sysroot must be complete enough to link a C++ program: libc, the C++ standard library, and the C++ headers. The target sysroot must provide the C library and headers that the target runtime libraries are built against. Both are C/C++ sysroots; they are just for different triples.

The split is clean once you see it:

Component	Provided by	Examples
C library	The sysroot	glibc, musl, bionic, newlib, mingw-w64 CRT, Apple libSystem
Start-up objects	The sysroot (from libc)	`crt1.o`, `crti.o`, `crtn.o`, `Scrt1.o`
Kernel headers	The sysroot	`linux/`, `asm/`, `asm-generic/`
Dynamic loader	The sysroot (from libc)	`ld-linux-x86-64.so.2`, `ld-musl-aarch64.so.1`
Compiler runtime	GCC or LLVM	`libgcc`, `libgcc_eh`, `compiler-rt` builtins
C++ standard library	GCC or LLVM for the target; the host's copy is part of the host C/C++ environment	`libstdc++`, `libc++`, `libc++abi`, `libunwind`
Compiler headers	GCC or LLVM	`stddef.h`, `stdarg.h`, `stdint.h`, `limits.h`

The C++ standard library depends on the C library, which is why it is built after the libc and against the sysroot. But it is still part of the compiler distribution: you install libstdc++ with GCC and libc++ with LLVM, not with the operating system. On a Canadian build, that distinction decides which artifact must be built for the host and which for the target.

The host side is the exception that proves the rule. The C++ compiler and C++ standard library used to build the new compiler are part of the build machine's existing C/C++ environment. They are not produced by the compiler you are building. This is why the first requirement on a new build machine is a working C/C++ toolchain, not a copy of the new compiler.

2. What a Sysroot Looks Like

A sysroot is a directory tree that looks like the target's root filesystem. GCC and Clang use the same idea: point the compiler at one directory, and the driver resolves its default header, library and program search paths inside it. The usual tree is just:

/path/to/sysroot/
  bin/       target programs and helper tools
  include/   C and C++ headers
  lib/       libraries, startup objects and the dynamic loader
  share/     target data files

Some systems merge /usr and put the same directories under usr/, so the compiler may search usr/include and usr/lib instead of include and lib. That is a path detail, not a different sysroot format. A target may also add lib64, libexec or a multiarch subdirectory. The root is still one directory.

GCC and Clang use the same root layout; both accept --sysroot. Their drivers may have different default search paths, but that is a driver detail, not a different sysroot format. A GCC cross driver often defaults its sysroot to $PREFIX/$TARGET; Clang is often invoked with an explicit --sysroot= pointing at an SDK. Both are pointing at a root tree. If the compiler's search path and the files on disk disagree, the link fails; the fix is to inspect the actual search path, not to rename the tree.

A sysroot is not a "GCC layout" or a "Clang layout". It is a target root. The compiler's default paths are a property of the driver and the target configuration; --sysroot only changes the root against which those paths are resolved.

The start-up objects are the usual first failure. A dynamically linked C program on Linux needs crt1.o, crti.o and crtn.o from libc, plus crtbegin.o and crtend.o from the compiler runtime, plus libc.so. A static program needs libc.a, libm.a and the static variants of the start-up objects. If any one is missing, the link fails before your first line of code is involved.

3. Hosted or Freestanding

There are two valid answers to "where is libc", and only two:

Hosted. The sysroot contains a real C library for the target: x86_64-linux-gnu with glibc, aarch64-linux-musl with musl, aarch64-linux-android30 with bionic, x86_64-w64-mingw32 with the mingw-w64 CRT, wasm32-wasip1 with wasi-libc. The compiler is built and configured for a hosted target, and __STDC_HOSTED__ is 1.
Freestanding. There is no libc, and there is not supposed to be one: x86_64-elf, arm-none-eabi, wasm32-unknown-unknown, riscv64-none-elf. The program supplies its own start-up code, its own memcpy and friends if the compiler emits calls to them, and its own I/O. The compiler is built without a hosted C++ library, with --without-headers and --disable-hosted-libstdcxx, and the result is a freestanding toolchain. Ch20 is about writing the program side of this.

The mistake is mixing the two. A triple that says -linux-gnu is a promise that the target has glibc and a Linux ABI. If you build a compiler for that triple without a sysroot, the build may succeed and every link will fail. Conversely, a triple that says -elf or -none- has no libc by definition; adding a Linux sysroot to it does not make it Linux, it only hides the mismatch until something calls a syscall that the target does not have.

Some embedded targets use newlib, which is a small C library with a porting layer. That is a hosted target in the C sense, but it is not POSIX and it is not glibc. It still belongs in the sysroot, and the compiler still provides only its own runtime. "Freestanding" means no libc at all, not "a small libc".

4. Where the Sysroot Comes From

There are three practical sources, and they fail in different ways:

Build the libc from source. For a Linux target this is the normal path: install kernel headers, build glibc or musl with the stage-1 cross compiler, install it into the sysroot, then rebuild GCC against it. The sysroot is then exactly the ABI you chose, at the cost of a long build.
Extract it from an SDK or an OS image. Android's NDK ships a sysroot with bionic headers and stub libraries. A Windows SDK ships the MSVC CRT and import libraries. A macOS SDK ships libSystem and the frameworks. This is faster but ties you to the vendor's layout and license.
Reuse another toolchain's sysroot. If a GCC cross toolchain already exists, clang can use it with --gcc-toolchain=$PREFIX --sysroot=$SYSROOT. This is how most people get a working clang cross compiler without rebuilding the world. It is also how the LLVM page in this chapter works.

For a Canadian build, the sysroot must be built for the target, and it must be visible at the path the compiler will record. GCC's --with-sysroot=$SYSROOT bakes that path into the compiler. If you later move the toolchain to another machine and the path does not exist there, the compiler starts and then fails to find its headers. Either install the sysroot at the same path on the host, or pass --sysroot explicitly at every invocation.

During the build there is a second path: --with-build-sysroot. It tells the build system where the target headers and libraries are on the build machine while the compiler is being produced. It is not recorded in the compiler. Confusing the two is a common reason a cross compiler works during make and fails after installation.

5. The Compiler's Own Files

It is worth being precise about what "GCC provides" and "LLVM provides" means, because the files are installed in different places:

GCC	LLVM	Built for
`libgcc.a`, `libgcc_s.so`	`libclang_rt.builtins.a`	Target
`libstdc++.so`, `libstdc++.a`	`libc++.so`, `libc++.a`	Target
`crtbegin.o`, `crtend.o`	`crtbegin.o`, `crtend.o` (where compiler-rt provides them)	Target
`libatomic`, `libgomp`, `libquadmath`	`libomp`, compiler-rt sanitizers	Target, optional
`cc1`, `cc1plus`, `collect2`, `lto-wrapper`	`clang`, `lld`, `llvm-ar`, `llvm-objcopy`	Host
`libgcc` for the compiler itself	LLVM/Clang libraries linked into the tools	Host

This table is the reason a Canadian build is not one build. The host-side tools (cc1plus, clang, ld) must run on the host. The target-side runtimes (libgcc, libstdc++) must run on the target. The sysroot's libc must match the target. A single make cannot do all three at once on one machine; the build has to keep the triples straight.

6. Verifying a Sysroot

Before blaming the compiler, ask it what it thinks it is:

$TARGET-gcc -print-sysroot
$TARGET-gcc -print-search-dirs
$TARGET-gcc -v -E -x c /dev/null

clang --target=$TARGET --sysroot=$SYSROOT -print-search-dirs
clang --target=$TARGET --sysroot=$SYSROOT -v -E -x c /dev/null

The -v -E output lists the include search path. Check that the first entries are inside the sysroot and that they contain the headers you expect. Then compile a trivial file with -nostdlib -Wl,--verbose to see which start-up objects and which libc the linker finds. If -print-sysroot is empty, the compiler is using its default prefix layout, which may still be correct; if it prints a path that does not exist on the current machine, that is your bug.

For a freestanding toolchain, the same commands should show no libc and no sysroot, and a link should succeed only when you pass -nostdlib and provide your own start-up code. If a freestanding toolchain silently finds the build machine's glibc, it will produce binaries that link and then cannot run.