$ cd /home/martin/workspace/prototypes/boot-to-claude
git commit -m "$(cat <<'EOF'
Phase T8: self-hosting bootstrap (Stage1/2/3, byte-identical)
tools/bootstrap_selfhost.sh runs the full chain: Stage1 (host gcc, the
only invocation anywhere in the build) compiles toolchain/{as,cc,ld} +
the new toolchain/hostlib/, Stage2 compiles that same source with
Stage1, Stage3 compiles it again with Stage2. Stage2 and Stage3 are
byte-identical for all three tools - the fixed-point proof CC genuinely
compiles itself. The Makefile's toolchain-bootstrap rule now calls this
script directly, so `make` builds the entire OS with the verified
Stage2/3 result.
Getting AS/CC/LD's own source through CC at all (never attempted before
- Phase T7 only ever compiled kernel/host-tool source) needed two real
CC features: variadic functions (va_start/va_arg/va_end, desugared to
plain pointer arithmetic since the preprocessor has no function-like
macros) and correct multi-dimensional array support. It also needed a
minimal libc - toolchain/hostlib/ implements just enough of stdio/
stdlib/string/ctype (unbuffered FILE* I/O, a bump allocator, a small
printf subset) over two new builtins (__sys_brk, __sys_chmod) - since
AS/CC/LD's own source uses fopen/malloc/strcmp freely, unlike the
already-libc-free mkfs_tinyfs.
Getting the self-hosted result to actually boot surfaced eleven further
real bugs, found via the same two-tier verification this phase's plan
called for (tools/run_toolchain_tests.sh, then the full OS boot/shell
suite against a disk image built end-to-end by the self-hosted result):
- elf_format.h's ELF32_ST_INFO/R_SYM/R_TYPE/R_INFO were function-like
macros, unsupported by CC's preprocessor - converted to static inline
functions
- three compound literals in AS's own source (never covered by Phase
T3's compound-literal refactor, which only touched kernel/ and
tools/) - replaced with memset + field assignment
- parse_param_list's own declaration (char names[][MAX_NAME]) exposed
its array-parameter decay bug: every bracket group wrapped in another
pointer instead of only the first, mistyping it as char** instead of
char(*)[MAX_NAME]
- codegen_addr had no case for a struct-returning ND_CALL used directly
as an lvalue (line_peek(ln).kind - AS's own parser.c)
- AS's .bss handling wrote real zero bytes into a fixed 64KB buffer;
AS's own ctx global alone needs ~20MB of .bss
- LD's per-file symbol/relocation limits (512) were too small for
encode.c's own object file (800+ symbols)
- a short/partial local brace-init (line_t ln = {0};) left the rest of
the aggregate as stack garbage, sending AS's own line-tokenizing loop
into a multi-million-iteration spin that looked like a hang
- a string-literal initializer for a pointer-typed struct field (AS's
own REGS[] register table) emitted a null pointer instead of a
relocation
- pointer-minus-pointer (elf_writer.c's own `s - ctx->syms`) was
compiled as pointer arithmetic instead of an integer element count,
corrupting a relocation's symbol index
- chmod(path, 0755) silently meant decimal 755 - CC's lexer has no
octal-literal support
- AS's own `.dq` directive computed its high 32 bits via `v >> 32`,
which is a no-op under x86's 32-bit shift-count masking once int64_t
is CC's 32-bit alias - corrupted boot.S's disk-address-packet LBA
field, the one bug invisible to every toolchain test (none use dq)
and only caught by the full OS boot suite
Full writeup in docs/toolchain.md's "Bootstrap (Phase T8 -
self-hosting)" section.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
EOF
)"
git status --short[main acff0a5] Phase T8: self-hosting bootstrap (Stage1/2/3, byte-identical) 30 files changed, 1131 insertions(+), 79 deletions(-) create mode 100644 toolchain/hostlib/ctype.c create mode 100644 toolchain/hostlib/stat.c create mode 100644 toolchain/hostlib/stdio.c create mode 100644 toolchain/hostlib/stdlib.c create mode 100644 toolchain/hostlib/string.c create mode 100644 toolchain/include/ctype.h create mode 100644 toolchain/include/hostsys.h create mode 100644 toolchain/include/stdarg.h create mode 100644 toolchain/include/stdio.h create mode 100644 toolchain/include/stdlib.h create mode 100644 toolchain/include/string.h create mode 100644 toolchain/include/sys/stat.h create mode 100755 tools/bootstrap_selfhost.sh
$ cd /home/martin/workspace/prototypes/boot-to-claude ls *.md README* 2>/dev/null find . -maxdepth 1 -iname "readme*"
README.md README.md ./README.md
1 # boot-to-claude
2
3 A from-scratch, single-tasking x86 OS for QEMU/KVM: a custom two-stage
4 boot loader, a monolithic kernel with its own tiny filesystem (tinyfs), an
5 interactive shell, a dynamic scripting language, and a small C-like
6 compiler with its own bytecode VM.
7
8 Every line of OS/filesystem/language source in this repo is hand-written
9 for this project - `nasm`, `gcc`, `ld`, `objcopy`, and `qemu` are used
10 purely as build/run infrastructure, the same way any project uses a
11 compiler.
12
13 ## Quickstart
14
15 ```
16 make # builds build/disk.img
17 make run # boots it in QEMU (headless, serial on stdio)
18 ```
19
20 `make run` boots into the shell:
21
22 ```
23 tiny-os shell. type 'help' for commands.
24 > ls
25 hello.txt 15
26 hello.tsh 281
27 hello.tc 272
28 > cat hello.txt
29 Hello, tinyfs!
30 > run hello.tsh
31 hello from tinyfs script
32 x = 14
33 ...
34 > build hello.tc hello.tbc
35 build: ok
36 > exec hello.tbc
37 14
38 7
39 55
40 10
41 ```
42
43 That one session exercises every layer: tinyfs (`ls`/`cat`), the
44 scripting language (`run`, including its own `fread`/`fwrite` calls into
45 tinyfs), and the compiler + bytecode VM (`build` + `exec`, including a
46 recursive function call). `tools/run_tests.sh` automates exactly this
47 against a headless QEMU instance - see [Testing](#testing).
48
49 Requires `nasm`, `gcc` (with `-m32` support), `ld`, `objcopy`, `dd`, and
50 `qemu-system-i386` on the host. No cross-compiler needed.
51
52 ## Architecture
53
54 ```
55 BIOS -> boot/boot.S (MBR, 512B) real mode
56 -> boot/stage2.S real mode -> protected mode
57 -> kernel/entry.S -> kmain.c 32-bit protected mode, flat, no paging
58 gdt.c / idt.c kernel-owned GDT; IDT covers CPU
59 exceptions only - no PIC/IRQs, ever
60 console.c / serial.c VGA text + COM1, dual-output throughout
61 memory.c free-list kmalloc/kfree over a 1MB heap
62 ata.c -> fs/tinyfs.c polled ATA PIO -> tinyfs filesystem
63 keyboard.c -> shell/shell.c polled PS/2 -> command REPL
64 lang/lexer.c (shared)
65 -> lang/script/{parser,ast,eval}.c scripting language (tree-walking)
66 -> lang/compiler/{parser,codegen}.c compiler -> lang/vm.c (bytecode VM)
67 ```
68
69 Deliberately single-tasking: no scheduler, no preemption, no interrupts
70 enabled after boot, no process/syscall boundary. The shell and both
71 language runtimes execute directly in kernel context. This keeps the OS
72 layer small so the interesting work is in the filesystem and language
73 implementations - see [`docs/`](docs/) for their designs:
74
75 - [`docs/tinyfs.md`](docs/tinyfs.md) - on-disk filesystem format
76 - [`docs/scripting-language.md`](docs/scripting-language.md) - the `tsh` language
77 - [`docs/bytecode-isa.md`](docs/bytecode-isa.md) - the compiled language + VM ISA
78
79 ## Repo layout
80
81 ```
82 boot/ stage1 (MBR) + stage2 boot loaders
83 kernel/ the kernel itself (see architecture diagram above)
84 kernel/fs/ tinyfs (shared: freestanding in the kernel, native in mkfs_tinyfs)
85 kernel/lang/ shared lexer + the two language implementations
86 kernel/shell/ interactive command REPL
87 tools/ host-side build/test infrastructure (not OS code):
88 mkfs_tinyfs.c formats + seeds build/disk.img at build time
89 run_tests.sh headless QEMU test harness
90 qemu_type.py scripts keystrokes into QEMU via its HMP monitor
91 tests/ fixtures (*.tsh, *.tc) + tests/expected/ serial-output fixtures
92 docs/ design docs, listed above
93 ```
94
95 ## Testing
96
97 There's no hardware and no display in CI, so the serial port is the test
98 oracle throughout: `kernel/selftest.c` runs a boot-time self-test (string
99 ops, the allocator including coalescing, and a tinyfs create/write/read/
100 delete roundtrip) and prints `SELFTEST PASS`/`FAIL`.
101
102 ```
103 tools/run_tests.sh
104 ```
105
106 boots `build/disk.img` headless (`-display none -serial file:...`), waits
107 for `SELFTEST PASS`, then drives the shell by scripting keystrokes into
108 QEMU's HMP monitor over a unix socket (`tools/qemu_type.py sendkey ...`)
109 - the only way to feed the kernel's polled PS/2 driver without real
110 hardware - and asserts on the resulting serial output for every shell
111 command, `run hello.tsh`, and `build`+`exec` on `hello.tc`.
112
113 ## Boot budget
114
115 The kernel image is capped at a fixed number of disk sectors (`boot/
116 stage2.S`'s `KERNEL_SECTORS`, currently 512 = 256KB), read via BIOS
117 INT13h in bounded chunks (SeaBIOS silently misbehaves on very large
118 single extended-read requests). `kernel/fs/tinyfs.h`'s
119 `TINYFS_FS_START_LBA` reserves exactly that much room before the
120 filesystem region begins. The Makefile fails the build loudly if the
121 kernel image ever grows past its budget - if you hit that, bump
122 `KERNEL_SECTORS`, the Makefile's `KERNEL_MAX_SECTORS`, and
123 `TINYFS_FS_START_LBA` together.
124 $ grep -n '^## ' /home/martin/workspace/prototypes/boot-to-claude/docs/toolchain.md
9:## AS (assembler) 129:## CC (compiler) 259:## Kernel source adaptation (Phase T3) 293:## LD (linker, Phase T5) 423:## Full switch-over (Phase T7) 599:## Bootstrap (Phase T8 - self-hosting)
1 # Self-hosted toolchain 2 3 `toolchain/` replaces `nasm`/`gcc`/`ld`/`objcopy` with our own tools, so 4 the OS build depends on external infrastructure only once - to bootstrap 5 the toolchain itself from source (see [Bootstrap](#bootstrap) below). 6 `qemu-system-i386` stays external; it's the machine we run on, not part 7 of what builds for it. 8 9 ## AS (assembler) 10
- Every line of OS/filesystem/language source in this repo is hand-written - for this project - `nasm`, `gcc`, `ld`, `objcopy`, and `qemu` are used - purely as build/run infrastructure, the same way any project uses a - compiler. + Every line of OS/filesystem/language source in this repo is hand-written + for this project - including the toolchain that builds it. `toolchain/` + is a self-hosted assembler, C compiler, and linker (see + [`docs/toolchain.md`](docs/toolchain.md)) that replaces `nasm`/`gcc`/ + `ld`/`objcopy` entirely: the host's real `gcc` is used exactly once, to + bootstrap the toolchain from source on a fresh checkout, and never + touches OS code. `qemu-system-i386` stays external - it's the machine we + run on, not part of what builds for it.
The file /home/martin/workspace/prototypes/boot-to-claude/README.md has been updated successfully. (file state is current in your context — no need to Read it back)
- Requires `nasm`, `gcc` (with `-m32` support), `ld`, `objcopy`, `dd`, and
- `qemu-system-i386` on the host. No cross-compiler needed.
+ Requires `dd` and `qemu-system-i386` on the host, plus (one-time only,
+ to bootstrap `toolchain/` from source - see below) a host C compiler,
+ `gcc` by default. That bootstrap step never runs again after the first
+ `build/toolchain/{as,cc,ld}` exist: `nasm`/`ld`/`objcopy` aren't needed
+ at all, and `gcc` never touches OS code, only `toolchain/`'s own source.
+ No cross-compiler needed.The file /home/martin/workspace/prototypes/boot-to-claude/README.md has been updated successfully. (file state is current in your context — no need to Read it back)
- - [`docs/tinyfs.md`](docs/tinyfs.md) - on-disk filesystem format - - [`docs/scripting-language.md`](docs/scripting-language.md) - the `tsh` language - - [`docs/bytecode-isa.md`](docs/bytecode-isa.md) - the compiled language + VM ISA + - [`docs/tinyfs.md`](docs/tinyfs.md) - on-disk filesystem format + - [`docs/scripting-language.md`](docs/scripting-language.md) - the `tsh` language + - [`docs/bytecode-isa.md`](docs/bytecode-isa.md) - the compiled language + VM ISA + - [`docs/toolchain.md`](docs/toolchain.md) - the self-hosted assembler/compiler/linker
The file /home/martin/workspace/prototypes/boot-to-claude/README.md has been updated successfully. (file state is current in your context — no need to Read it back)
- ``` - boot/ stage1 (MBR) + stage2 boot loaders - kernel/ the kernel itself (see architecture diagram above) - kernel/fs/ tinyfs (shared: freestanding in the kernel, native in mkfs_tinyfs) - kernel/lang/ shared lexer + the two language implementations - kernel/shell/ interactive command REPL - tools/ host-side build/test infrastructure (not OS code): - mkfs_tinyfs.c formats + seeds build/disk.img at build time - run_tests.sh headless QEMU test harness - qemu_type.py scripts keystrokes into QEMU via its HMP monitor - tests/ fixtures (*.tsh, *.tc) + tests/expected/ serial-output fixtures - docs/ design docs, listed above - ``` + ``` + boot/ stage1 (MBR) + stage2 boot loaders + kernel/ the kernel itself (see architecture diagram above) + kernel/fs/ tinyfs (shared: freestanding in the kernel, native in mkfs_tinyfs) + kernel/lang/ shared lexer + the two language implementations + kernel/shell/ interactive command REPL + toolchain/ self-hosted AS/CC/LD (see docs/toolchain.md) - host-side + build infrastructure, not OS code, but self-authored + like everything else in this repo + tools/ host-side build/test infrastructure (not OS code): + mkfs_tinyfs.c formats + seeds build/disk.img at build time + bootstrap_selfhost.sh builds+verifies toolchain/ (Stage1/2/3) + run_tests.sh headless QEMU test harness + run_toolchain_tests.sh host-side AS/CC/LD test harness + qemu_type.py scripts keystrokes into QEMU via its HMP monitor + tests/ fixtures (*.tsh, *.tc) + tests/expected/ serial-output fixtures + tests/toolchain/ AS/CC/LD test fixtures (see run_toolchain_tests.sh) + docs/ design docs, listed above + ```
The file /home/martin/workspace/prototypes/boot-to-claude/README.md has been updated successfully. (file state is current in your context — no need to Read it back)
$ cd /home/martin/workspace/prototypes/boot-to-claude rm -rf build timeout 300 make 2>&1 | tail -5 echo "=== make exit=$? ===" timeout 60 ./tools/run_tests.sh 2>&1 | tail -5 echo "=== run_tests exit ===" timeout 90 ./tools/run_toolchain_tests.sh 2>&1 | tail -5
dd if=build/stage2.bin of=build/disk.img bs=512 seek=1 conv=notrunc status=none dd if=build/kernel.bin of=build/disk.img bs=512 seek=5 conv=notrunc status=none build/mkfs_tinyfs build/disk.img tests/hello.tsh:hello.tsh tests/hello.tc:hello.tc mkfs_tinyfs: done rm build/console.S build/shell/shell.S build/idt.S build/lang/script/parser.S build/lang/script/ast.S build/lang/script/eval.S build/lang/vm.S build/lang/compiler/codegen.S build/lang/compiler/parser.S build/kmain.S build/gdt.S build/string.S build/fs/tinyfs.S build/selftest.S build/serial.S build/lang/lexer.S build/fs/tinyfs_disk.S build/ata.S build/keyboard.S build/memory.S === make exit=0 === run_tests: PASS qemu-system-i386: terminating on signal 15 from pid 1555308 (bash) === run_tests exit === PASS t12_local_array_size (4) PASS t13_struct_copy (212) PASS t14_struct_return (44) PASS t15_escapes (158) run_toolchain_tests: PASS
$ cd /home/martin/workspace/prototypes/boot-to-claude ls build/toolchain-stage2 build/toolchain-stage3 2>&1 cmp build/toolchain-stage2/as build/toolchain-stage3/as && echo "as: identical" cmp build/toolchain-stage2/cc build/toolchain-stage3/cc && echo "cc: identical" cmp build/toolchain-stage2/ld build/toolchain-stage3/ld && echo "ld: identical"
build/toolchain-stage2: as cc ld build/toolchain-stage3: as cc ld as: identical cc: identical ld: identical
$ cd /home/martin/workspace/prototypes/boot-to-claude git diff README.md
diff --git a/README.md b/README.md
index d749636..f0f78c5 100644
--- a/README.md
+++ b/README.md
@@ -6,9 +6,13 @@ interactive shell, a dynamic scripting language, and a small C-like
compiler with its own bytecode VM.
Every line of OS/filesystem/language source in this repo is hand-written
-for this project - `nasm`, `gcc`, `ld`, `objcopy`, and `qemu` are used
-purely as build/run infrastructure, the same way any project uses a
-compiler.
+for this project - including the toolchain that builds it. `toolchain/`
+is a self-hosted assembler, C compiler, and linker (see
+[`docs/toolchain.md`](docs/toolchain.md)) that replaces `nasm`/`gcc`/
+`ld`/`objcopy` entirely: the host's real `gcc` is used exactly once, to
+bootstrap the toolchain from source on a fresh checkout, and never
+touches OS code. `qemu-system-i386` stays external - it's the machine we
+run on, not part of what builds for it.
## Quickstart
@@ -46,8 +50,12 @@ tinyfs), and the compiler + bytecode VM (`build` + `exec`, including a
recursive function call). `tools/run_tests.sh` automates exactly this
against a headless QEMU instance - see [Testing](#testing).
-Requires `nasm`, `gcc` (with `-m32` support), `ld`, `objcopy`, `dd`, and
-`qemu-system-i386` on the host. No cross-compiler needed.
+Requires `dd` and `qemu-system-i386` on the host, plus (one-time only,
+to bootstrap `toolchain/` from source - see below) a host C compiler,
+`gcc` by default. That bootstrap step never runs again after the first
+`build/toolchain/{as,cc,ld}` exist: `nasm`/`ld`/`objcopy` aren't needed
+at all, and `gcc` never touches OS code, only `toolchain/`'s own source.
+No cross-compiler needed.
## Architecture
@@ -75,6 +83,7 @@ implementations - see [`docs/`](docs/) for their designs:
- [`docs/tinyfs.md`](docs/tinyfs.md) - on-disk filesystem format
- [`docs/scripting-language.md`](docs/scripting-language.md) - the `tsh` language
- [`docs/bytecode-isa.md`](docs/bytecode-isa.md) - the compiled language + VM ISA
+- [`docs/toolchain.md`](docs/toolchain.md) - the self-hosted assembler/compiler/linker
## Repo layout
@@ -84,11 +93,17 @@ kernel/ the kernel itself (see architecture diagram above)
kernel/fs/ tinyfs (shared: freestanding in the kernel, native in mkfs_tinyfs)
kernel/lang/ shared lexer + the two language implementations
kernel/shell/ interactive command REPL
+toolchain/ self-hosted AS/CC/LD (see docs/toolchain.md) - host-side
+ build infrastructure, not OS code, but self-authored
+ like everything else in this repo
tools/ host-side build/test infrastructure (not OS code):
- mkfs_tinyfs.c formats + seeds build/disk.img at build time
- run_tests.sh headless QEMU test harness
- qemu_type.py scripts keystrokes into QEMU via its HMP monitor
+ mkfs_tinyfs.c formats + seeds build/disk.img at build time
+ bootstrap_selfhost.sh builds+verifies toolchain/ (Stage1/2/3)
+ run_tests.sh headless QEMU test harness
+ run_toolchain_tests.sh host-side AS/CC/LD test harness
+ qemu_type.py scripts keystrokes into QEMU via its HMP monitor
tests/ fixtures (*.tsh, *.tc) + tests/expected/ serial-output fixtures
+tests/toolchain/ AS/CC/LD test fixtures (see run_toolchain_tests.sh)
docs/ design docs, listed above
```228 re-running the full `tools/run_tests.sh` OS boot/shell suite against a 229 disk image built end-to-end by the self-hosted result, exactly the 230 two-tier verification this phase's plan called for. Full list in 231 `docs/toolchain.md`'s "Bootstrap (Phase T8 - self-hosting)" section. 232 233 **Phase T9 - Docs + cleanup** 234 `docs/toolchain.md` finalized (instruction set, C dialect + builtin 235 list, linker behavior, the 3-stage bootstrap procedure and how to 236 re-run it). README's toolchain-requirements section updated: host 237 gcc/nasm/ld are a one-time bootstrap dependency, not a build dependency 238 (`objcopy` isn't needed even for bootstrap). Final full-repo 239 `tools/run_tests.sh` pass plus the Stage2==Stage3 check, both green. 240 241 ## Verification strategy 242 243 Every phase has a concrete, automated pass/fail gate before moving on: 244 byte-diff against the reference tool (T1), parse/compile of the real 245 codebase (T2, T3), a growing fixture battery through the new pipeline 246 (T4), functional QEMU/serial-log equivalence (T5, T7, reusing 247 `tools/run_tests.sh`'s existing pattern), and byte-identical fixed-point
- **Phase T9 - Docs + cleanup** - `docs/toolchain.md` finalized (instruction set, C dialect + builtin - list, linker behavior, the 3-stage bootstrap procedure and how to - re-run it). README's toolchain-requirements section updated: host - gcc/nasm/ld are a one-time bootstrap dependency, not a build dependency - (`objcopy` isn't needed even for bootstrap). Final full-repo - `tools/run_tests.sh` pass plus the Stage2==Stage3 check, both green. + **Phase T9 - Docs + cleanup** - done + `docs/toolchain.md` was already finalized incrementally across T1-T8 + (instruction set, C dialect + builtin list, linker behavior, the + 3-stage bootstrap procedure); this phase's own work was the README: + its toolchain-requirements section now says what's actually true - + `nasm`/`ld`/`objcopy` aren't needed at all, and host `gcc` is a + one-time-only bootstrap dependency that never touches OS code - plus a + new `docs/toolchain.md` link, a `toolchain/` entry in the repo-layout + table, and `tools/bootstrap_selfhost.sh`/`tests/toolchain/` added + alongside the existing tools/tests entries. Verified: a full clean + `make` (no build/ directory present beforehand), `tools/run_tests.sh`, + `tools/run_toolchain_tests.sh`, and an explicit Stage2==Stage3 `cmp` for + all three tools - all green from this one build. + + This closes out the whole self-hosted-toolchain project: every line of + this OS, and now the toolchain that builds it, is Claude-authored, with + host `gcc` touching nothing but `toolchain/`'s own source, exactly once, + on a fresh checkout.
The file /home/martin/.claude/plans/swirling-mixing-bee.md has been updated successfully. (file state is current in your context — no need to Read it back)
$ cd /home/martin/workspace/prototypes/boot-to-claude git add README.md git status --short
M README.md