# Building & Debugging a Custom ARM64 Linux Kernel — Yocto, QEMU, GDB

You can rebuild an ARM64 Linux kernel, boot it, and single-step through `start_kernel` in GDB without owning a single piece of ARM hardware. It runs on your laptop. The recipe is the easy part and it's well-trodden; what actually eats an afternoon the first time is a quieter problem — GDB attaches, your breakpoint hits, and then it tells you it can't find the source file. This walks the whole loop and spends its time on that part.

%[https://www.youtube.com/watch?v=t34iHB195y0] 

## The loop, four steps

The workflow in the video is four steps, and each maps to one tool:

1.  **Change the kernel config** and capture it as a *config fragment* (not a hand-edited `.config` you'll lose on the next build).
    
2.  **Rebuild** the kernel and root filesystem with `bitbake`.
    
3.  **Boot** the image under QEMU, with the GDB stub enabled.
    
4.  **Attach** GDB to the stub and debug the running kernel.
    

The reason to do it this way — Yocto for the build rather than a raw `make` — is reproducibility: the config fragment and the recipe are the source of truth, so the debug kernel you built today is the debug kernel you get next month.

## Step 1 — a debug config, as a fragment

Open the kernel config through Yocto rather than poking the tree directly:

```bash
bitbake -c menuconfig virtual/kernel
```

The settings that matter for debugging aren't about features, they're about *keeping the information GDB needs*:

```plaintext
CONFIG_DEBUG_KERNEL=y
CONFIG_DEBUG_INFO=y
CONFIG_DEBUG_INFO_REDUCED=n     # reduced info drops what you need for inline unwinding
CONFIG_GDB_SCRIPTS=y            # brings in the vmlinux-gdb.py helpers
CONFIG_RANDOMIZE_BASE=n         # turn KASLR off so symbol addresses are stable
```

That last one is the difference between a productive session and confusion: with KASLR on, the kernel's runtime addresses are randomized and won't line up with the symbols GDB reads from `vmlinux`. Turn it off for debugging (or pass `nokaslr` on the kernel command line). Save these as a fragment and wire it into the kernel recipe (`SRC_URI += "file://debug.cfg"`) so it survives rebuilds instead of living in your shell history.

## Step 2 & 3 — build, then boot with the stub open

```bash
bitbake core-image-minimal
runqemu qemuarm64 nographic qemuparams="-s -S"
```

The two QEMU flags are the whole trick: `-s` opens the GDB stub on TCP `:1234`, and `-S` freezes the machine at reset so nothing runs until you attach. Without `-S` the kernel is already past early boot before GDB connects, and `start_kernel` breakpoints never fire.

## Step 4 — attach, and the part that actually trips people

Point the cross-GDB at the `vmlinux` with symbols (the one from the build tree, not the stripped image that boots) and connect:

```gdb
aarch64-linux-gnu-gdb vmlinux
(gdb) target remote :1234
(gdb) break start_kernel
(gdb) continue
```

The breakpoint hits — and GDB prints something like *"No such file or directory"* for the source line. This is the moment the tutorials skip, and it's exactly where the video slows down. The symbols are fine; the addresses are fine. What's wrong is that the debug info records the *build-time* source path — some long `/usr/src/kernel/...` or Yocto `tmp/work/...` path from the build host — and that path doesn't exist where you're now running GDB. GDB is looking in the right conceptual place and the wrong literal one.

The fix is to tell GDB how to translate the build path to your actual source tree:

```gdb
(gdb) set substitute-path /usr/src/kernel /home/you/yocto/.../linux-source
(gdb) list start_kernel
```

Now the source resolves and `list`, `step`, and inline frames all work. Because you'll do this every session, put the connect-and-substitute sequence in a `.gdbinit` (or a `-x` script) so a single `gdb -x debug.gdb vmlinux` gets you to a live, source-resolved breakpoint every time. That small bit of automation is what turns "I got it working once" into a debugging loop you'll actually use.

## Why bother, if you ship Go and not kernels

Most backend and infra engineers never look below the syscall boundary. They see goroutine yields, container CPU throttling, and latency spikes they can't explain, reach for `pprof`, and when that runs out, blame "the cluster." But a lot of what shows up as p99 lives in the kernel scheduler: which core your thread runs on, how often it migrates, whether the kernel preempted you at a CFS slice boundary or you yielded. You can't reason confidently about any of that from user space alone.

Being able to break on `schedule()` in a running kernel and watch it decide is what lets you make a claim about where time goes instead of guessing. For AI infrastructure the same logic is sharper: every inference call is a stack of userspace→kernel→userspace round trips — file I/O, network, GPU driver entry — and the latency variance you're tempted to pin on the model is often kernel-side scheduling and syscall cost. A kernel you can stop and inspect is the instrument that settles those arguments.

You won't build a Yocto image at work. But having done it once — and knowing why GDB couldn't find the source, and how to make it — is the difference between treating the kernel as a black box and treating it as something you can open.

## Related

*   **Video:** [Build & Debug a Custom ARM64 Linux Kernel with Yocto, QEMU, GDB](https://www.youtube.com/watch?v=t34iHB195y0)
    
*   **Sibling walkthrough (x86):** [Debugging the Ubuntu 6.8 x86-64 kernel with GDB + QEMU](https://www.youtube.com/watch?v=XFJx_3u6Gx8)
    
*   **GitHub:** [CoreTracer](https://github.com/harrison001/CoreTracer) — separate low-level experiments (scheduling, cache, lock-free) in the same "open the box" spirit, not the Yocto build above.