Over the last several updates, Rainbow-OS went on a diet. One by one, the
things that had been compiled into the kernel — both games, the text
editor, the BASIC interpreter, even the C compiler — moved out and became
ordinary ring-3 programs loaded from disk. The kernel shrank from around 122 KB
to 49 KB and started to look like what a kernel is supposed to be: the thing
that runs programs, not the thing that is the programs.
One subsystem refused to leave: the debugger. And after looking closely at
what it would take, the right answer turned out to be — it shouldn’t. Here’s
why, and what it would actually take to change that.
The debugger isn’t a program. It’s a privilege.
The other subsystems were easy to evict because they only ever asked
the kernel for things: draw this, read that file, give me a keystroke. Wrap each
request in a syscall and the code runs perfectly well unprivileged.
The debugger is different in kind. It doesn’t ask the kernel for things —
it is a piece of the kernel’s machinery. When you set a
breakpoint, the debugger writes a 0xCC byte into the target program.
When the CPU hits it, it raises a breakpoint exception that
vectors straight into an interrupt handler. Single-stepping works by setting a
flag in the CPU’s status register so that every instruction raises a
debug exception. Those exceptions are delivered by the hardware,
in ring 0, to handlers only the kernel can install. The debugger’s entire
command prompt runs inside those handlers, reading and rewriting the
suspended program’s saved registers directly.
A ring-3 program cannot install an interrupt handler. It cannot receive a CPU
exception. It cannot reach into another program’s registers. Everything that
makes the debugger a debugger is, by definition, privileged. You can’t syscall
your way out of that.
And even if you could, where would it live?
There’s a second, more concrete problem. Rainbow-OS currently runs with a
single address space: there is one slot for a user program, at a
fixed address (2 MB), and exactly one program lives there at a time. The C
compiler bakes that address into every binary it produces.
A debugger and the program it debugs have to be in memory at the same
time. But both of them want to live at 2 MB. They’d land on top of each
other. In the current model there is simply nowhere to put two ring-3 programs at
once.
What it would actually take
So a userland debugger isn’t a port — it’s two real milestones of new
operating-system infrastructure:
- Per-process address spaces. Each program gets its own page
table, so the debugger’s „2 MB“ and the target’s „2 MB“ map to different
physical memory. This is the same machinery you need for multitasking in general,
and it’s the bigger of the two jobs. - A ptrace-style syscall interface. The kernel keeps the trap
handlers, but instead of running a prompt inside them, it freezes the target and
hands control back to the debugger program with a description of what happened.
The debugger then drives everything through syscalls: load a target under
control, wait for it to stop, read and write its memory, read and write its
registers, set a breakpoint, single-step, continue. If that list looks
familiar, it’s because it’s essentially Unix’sptrace— the
exact API real systems invented for exactly this problem.
Notice the dependency: the syscall API is useless without the address spaces,
because without them the two programs can’t coexist. So it’s address spaces
first, then ptrace, then the debugger UI moves to userland and the kernel keeps
only a small, well-defined control core.
Why staying in the kernel is the right answer (for now)
Here’s the thing: even on Linux, the debugging facility lives in the kernel.
ptrace is a system call; the kernel is what stops the traced process,
reads its registers, and single-steps it. Userland tools like gdb
are just clients of that kernel service. A debugger has to be partly privileged
— that’s not a wart, it’s the nature of the task.
So Rainbow-OS’s debugger staying in the kernel isn’t unfinished business. It’s
the one subsystem that genuinely belongs there, and it marks a clean line: the
kernel now contains the OS core — drivers, filesystem, memory, the syscall
layer, and the privileged debug facility — and nothing else. Everything a
user could reasonably run as a program, now is one.
The day Rainbow-OS grows per-process address spaces — which it will want
anyway, the moment it wants to run two things at once — the debugger can
take the same trip the editor and the compiler already took. Until then, it’s
exactly where it should be.