Phase 7: README, docs, final check

Bash · Jul 17, 19:38:08 · toolu_01Q8xxZPAi1XgGw5rNDS4wCt
Commit Phase 6 compiler and bytecode VM
$ git commit -m "$(cat <<'EOF'
Phase 6: tiny compiler + bytecode VM

Adds a second language path alongside the Phase 5 interpreter: a
statically-scoped, int-only C-like language compiled to a small stack-
machine bytecode (docs/bytecode-isa.md) and executed by an in-kernel
VM. Reuses the shared lexer (kernel/lang/lexer.c) and the scripting
language's generic AST node type (kernel/lang/script/ast.h) - the
compiler's own parser (kernel/lang/compiler/parser.c) builds the same
tree shape minus string literals, since a full second AST would just
duplicate kernel/lang/script/ast.h for no benefit.

codegen.c does a two-pass compile: register all top-level function
names first (so forward/recursive calls resolve), then compile each
function body and the top-level statements into a flat bytecode
buffer plus a function table (entry address/param count/local count).
Every compiled expression leaves exactly one value on the VM's value
stack, including assignment (`<value> DUP STORE`), so callers can
treat all expressions uniformly.

vm.c executes the resulting .tbc file: separate value stack, a
locals stack sliced into per-call frames (supporting real recursion -
verified via a recursive fib(10) in the test fixture), a call stack
of {return_pc, locals_base}, and a fixed globals array. Functions
cannot see globals or an enclosing scope, matching the Phase 5
interpreter's "no closures" rule for consistency between the two
languages.

Shell gains `build <src> <out>` (parse + codegen, writes a .tbc file)
and `exec <file>` (runs it through the VM). tests/hello.tc exercises
arithmetic, a plain function call, recursion, and a while loop end-to-
end via tools/run_tests.sh.
EOF
)"
result
[main 3c37ed4] Phase 6: tiny compiler + bytecode VM
 13 files changed, 1248 insertions(+), 5 deletions(-)
 create mode 100644 docs/bytecode-isa.md
 create mode 100644 kernel/lang/bytecode.h
 create mode 100644 kernel/lang/compiler/codegen.c
 create mode 100644 kernel/lang/compiler/codegen.h
 create mode 100644 kernel/lang/compiler/parser.c
 create mode 100644 kernel/lang/compiler/parser.h
 create mode 100644 kernel/lang/vm.c
 create mode 100644 kernel/lang/vm.h
 create mode 100644 tests/expected/hello.tc.txt
 create mode 100644 tests/hello.tc
agent · Jul 17, 19:38:09
You've hit your session limit · resets 10:10pm (Europe/Berlin)
human · Jul 17, 22:10:17
go
TaskUpdate · Jul 17, 22:10:26 · toolu_01DyVu7LsWGRrLGoM3wiskWH
{
 "taskId": "7",
 "status": "completed"
}
result
Updated task #7 status
TaskUpdate · Jul 17, 22:10:26 · toolu_01SVzQFL4RHchyNN2QfJfSm9
{
 "taskId": "8",
 "status": "in_progress"
}
result
Updated task #8 status
agent · Jul 17, 22:10:39
Now Phase 7: README, the scripting-language doc, and a final cleanliness check.
Write · Jul 17, 22:10:48 · toolu_01BzAF7tUbZQuLfaSHMuLDMR
/home/martin/workspace/prototypes/boot-to-claude/docs/scripting-language.md
59 lines
# tsh scripting language

The dynamic, tree-walking counterpart to the [tiny compiler](bytecode-isa.md).
Run with the shell's `run <file>` command; see `tests/hello.tsh` for a
worked example.

## Syntax

C-like statements, no semicolons after blocks:

```
x = 2 + 3 * 4;
print("x =", x);

func add(a, b) {
    return a + b;
}
print("add(3,4) =", add(3, 4));

i = 0;
while (i < 3) {
    print("loop", i);
    i = i + 1;
}
```

- Types: 32-bit integers and strings (`"..."`). No arrays, no booleans -
  `0`/empty-string are falsy, everything else truthy.
- Variables: no declarations: first assignment creates them. All
  variables in a script share one global scope, *except* inside a
  function body, which gets its own isolated local scope - functions
  cannot see the caller's variables or each other's (no closures).
- Control flow: `if (...) {...} else {...}` (including `else if` chains),
  `while (...) {...}`.
- Functions: `func name(params) {...}`, `return expr;` (or bare `return;`
  / falling off the end, both give `0`).
- Operators: `+ - * / %`, `== != < <= > >=`, `&& ||` (short-circuit),
  unary `- !`. `+` on any string operand concatenates (numbers are
  stringified).
- Comments: `# to end of line`.

## Builtins

| call | effect |
|---|---|
| `print(a, b, ...)` | writes args space-separated + newline to console and serial |
| `fread(name)` | returns a file's contents as a string, or `""` if missing |
| `fwrite(name, contents)` | creates/overwrites a tinyfs file, returns `1`/`0` |
| `flist()` | prints every file name (one per line), returns the count |

## Implementation

`kernel/lang/lexer.c` (shared with the compiler) tokenizes;
`kernel/lang/script/parser.c` builds an AST (`ast.h`) via recursive
descent; `kernel/lang/script/eval.c` tree-walks it directly - there is no
intermediate bytecode, unlike the compiled language. AST nodes and string
literals are allocated from the kernel heap and never freed; fine for a
single `run` of a short script, since the heap resets on reboot.
result
File created successfully at: /home/martin/workspace/prototypes/boot-to-claude/docs/scripting-language.md (file state is current in your context — no need to Read it back)
Bash · Jul 17, 22:10:56 · toolu_014K9rJYhWizATKj5jWoKcyd
List all tracked files for an accurate README tree
$ find . -path ./build -prune -o -path ./.git -prune -o -type f -print | sort
result
57 lines
./.gitignore
./Makefile
./boot/boot.S
./boot/stage2.S
./docs/bytecode-isa.md
./docs/scripting-language.md
./docs/tinyfs.md
./kernel/ata.c
./kernel/ata.h
./kernel/console.c
./kernel/console.h
./kernel/entry.S
./kernel/fs/tinyfs.c
./kernel/fs/tinyfs.h
./kernel/fs/tinyfs_disk.c
./kernel/fs/tinyfs_disk.h
./kernel/gdt.c
./kernel/gdt.h
./kernel/idt.c
./kernel/idt.h
./kernel/idt_stubs.S
./kernel/keyboard.c
./kernel/keyboard.h
./kernel/kmain.c
./kernel/lang/bytecode.h
./kernel/lang/compiler/codegen.c
./kernel/lang/compiler/codegen.h
./kernel/lang/compiler/parser.c
./kernel/lang/compiler/parser.h
./kernel/lang/lexer.c
./kernel/lang/lexer.h
./kernel/lang/script/ast.c
./kernel/lang/script/ast.h
./kernel/lang/script/eval.c
./kernel/lang/script/eval.h
./kernel/lang/script/parser.c
./kernel/lang/script/parser.h
./kernel/lang/vm.c
./kernel/lang/vm.h
./kernel/link.ld
./kernel/memory.c
./kernel/memory.h
./kernel/selftest.c
./kernel/selftest.h
./kernel/serial.c
./kernel/serial.h
./kernel/shell/shell.c
./kernel/shell/shell.h
./kernel/string.c
./kernel/string.h
./tests/expected/hello.tc.txt
./tests/expected/hello.tsh.txt
./tests/hello.tc
./tests/hello.tsh
./tools/mkfs_tinyfs.c
./tools/qemu_type.py
./tools/run_tests.sh
Write · Jul 17, 22:11:21 · toolu_01HBkiJnwqaYwJRahQyHdmtS
/home/martin/workspace/prototypes/boot-to-claude/README.md
124 lines
# boot-to-claude

A from-scratch, single-tasking x86 OS for QEMU/KVM: a custom two-stage
boot loader, a monolithic kernel with its own tiny filesystem (tinyfs), an
interactive shell, a dynamic scripting language, and a small C-like
compiler with its own bytecode VM.

Every line of OS/filesystem/language source in this repo is hand-written
for this project - `nasm`, `gcc`, `ld`, `objcopy`, and `qemu` are used
purely as build/run infrastructure, the same way any project uses a
compiler.

## Quickstart

```
make        # builds build/disk.img
make run    # boots it in QEMU (headless, serial on stdio)
```

`make run` boots into the shell:

```
tiny-os shell. type 'help' for commands.
> ls
hello.txt  15
hello.tsh  281
hello.tc   206
> cat hello.txt
Hello, tinyfs!
> run hello.tsh
hello from tinyfs script
x = 14
...
> build hello.tc hello.tbc
build: ok
> exec hello.tbc
14
7
55
10
```

That one session exercises every layer: tinyfs (`ls`/`cat`), the
scripting language (`run`, including its own `fread`/`fwrite` calls into
tinyfs), and the compiler + bytecode VM (`build` + `exec`, including a
recursive function call). `tools/run_tests.sh` automates exactly this
against a headless QEMU instance - see [Testing](#testing).

Requires `nasm`, `gcc` (with `-m32` support), `ld`, `objcopy`, `dd`, and
`qemu-system-i386` on the host. No cross-compiler needed.

## Architecture

```
BIOS -> boot/boot.S (MBR, 512B)         real mode
     -> boot/stage2.S                    real mode -> protected mode
     -> kernel/entry.S -> kmain.c        32-bit protected mode, flat, no paging
          gdt.c / idt.c                  kernel-owned GDT; IDT covers CPU
                                          exceptions only - no PIC/IRQs, ever
          console.c / serial.c           VGA text + COM1, dual-output throughout
          memory.c                       free-list kmalloc/kfree over a 1MB heap
          ata.c -> fs/tinyfs.c           polled ATA PIO -> tinyfs filesystem
          keyboard.c -> shell/shell.c    polled PS/2 -> command REPL
          lang/lexer.c (shared)
            -> lang/script/{parser,ast,eval}.c   scripting language (tree-walking)
            -> lang/compiler/{parser,codegen}.c  compiler -> lang/vm.c (bytecode VM)
```

Deliberately single-tasking: no scheduler, no preemption, no interrupts
enabled after boot, no process/syscall boundary. The shell and both
language runtimes execute directly in kernel context. This keeps the OS
layer small so the interesting work is in the filesystem and language
implementations - see [`docs/`](docs/) for their designs:

- [`docs/tinyfs.md`](docs/tinyfs.md) - on-disk filesystem format
- [`docs/scripting-language.md`](docs/scripting-language.md) - the `tsh` language
- [`docs/bytecode-isa.md`](docs/bytecode-isa.md) - the compiled language + VM ISA

## Repo layout

```
boot/            stage1 (MBR) + stage2 boot loaders
kernel/          the kernel itself (see architecture diagram above)
kernel/fs/       tinyfs (shared: freestanding in the kernel, native in mkfs_tinyfs)
kernel/lang/     shared lexer + the two language implementations
kernel/shell/    interactive command REPL
tools/           host-side build/test infrastructure (not OS code):
                   mkfs_tinyfs.c   formats + seeds build/disk.img at build time
                   run_tests.sh    headless QEMU test harness
                   qemu_type.py    scripts keystrokes into QEMU via its HMP monitor
tests/           fixtures (*.tsh, *.tc) + tests/expected/ serial-output fixtures
docs/            design docs, listed above
```

## Testing

There's no hardware and no display in CI, so the serial port is the test
oracle throughout: `kernel/selftest.c` runs a boot-time self-test (string
ops, the allocator including coalescing, and a tinyfs create/write/read/
delete roundtrip) and prints `SELFTEST PASS`/`FAIL`.

```
tools/run_tests.sh
```

boots `build/disk.img` headless (`-display none -serial file:...`), waits
for `SELFTEST PASS`, then drives the shell by scripting keystrokes into
QEMU's HMP monitor over a unix socket (`tools/qemu_type.py sendkey ...`)
- the only way to feed the kernel's polled PS/2 driver without real
hardware - and asserts on the resulting serial output for every shell
command, `run hello.tsh`, and `build`+`exec` on `hello.tc`.

## Boot budget

The kernel image is capped at a fixed number of disk sectors (`boot/
stage2.S`'s `KERNEL_SECTORS`, currently 512 = 256KB), read via BIOS
INT13h in bounded chunks (SeaBIOS silently misbehaves on very large
single extended-read requests). `kernel/fs/tinyfs.h`'s
`TINYFS_FS_START_LBA` reserves exactly that much room before the
filesystem region begins. The Makefile fails the build loudly if the
kernel image ever grows past its budget - if you hit that, bump
`KERNEL_SECTORS`, the Makefile's `KERNEL_MAX_SECTORS`, and
`TINYFS_FS_START_LBA` together.
result
File created successfully at: /home/martin/workspace/prototypes/boot-to-claude/README.md (file state is current in your context — no need to Read it back)
agent · Jul 17, 22:11:24
Let me verify the exact `ls` output and demo transcript match reality before finalizing the README.
Bash · Jul 17, 22:11:27 · toolu_01Gp8YzywEQ7cnsTC5Jbz5cY
Boot and check exact ls/cat output for README accuracy
$ cd /home/martin/workspace/prototypes/boot-to-claude
rm -f build/mon.sock build/serial.log
timeout 8 qemu-system-i386 -drive file=build/disk.img,format=raw -display none -serial file:build/serial.log -monitor unix:build/mon.sock,server,nowait -no-reboot &
QPID=$!
for i in $(seq 1 25); do [ -S build/mon.sock ] && break; sleep 0.2; done
sleep 0.3
python3 tools/qemu_type.py build/mon.sock $'ls\n'
sleep 0.3
python3 tools/qemu_type.py build/mon.sock $'cat hello.txt\n'
sleep 0.3
wait $QPID 2>/dev/null
cat build/serial.log
result
qemu-system-i386: terminating on signal 15 from pid 2113803 (timeout)
tiny-os: kernel entered
selftest: running
SELFTEST PASS
tiny-os shell. type 'help' for commands.
> ls
hello.txt  15
hello.tsh  281
hello.tc  272
script_out.txt  17
hello.tbc  184
> cat hello.txt
Hello, tinyfs!
>
Bash · Jul 17, 22:11:44 · toolu_01EorFXG4g64eqqYj4Yth2Lx
Fresh disk image boot to get accurate ls output for README
$ cd /home/martin/workspace/prototypes/boot-to-claude
make clean >/dev/null && make >/dev/null 2>&1
rm -f build/mon.sock build/serial.log
timeout 6 qemu-system-i386 -drive file=build/disk.img,format=raw -display none -serial file:build/serial.log -monitor unix:build/mon.sock,server,nowait -no-reboot &
QPID=$!
for i in $(seq 1 25); do [ -S build/mon.sock ] && break; sleep 0.2; done
sleep 0.3
python3 tools/qemu_type.py build/mon.sock $'ls\n'
sleep 0.3
python3 tools/qemu_type.py build/mon.sock $'cat hello.txt\n'
sleep 0.3
wait $QPID 2>/dev/null
cat build/serial.log
result
qemu-system-i386: terminating on signal 15 from pid 2114108 (timeout)
tiny-os: kernel entered
selftest: running
SELFTEST PASS
tiny-os shell. type 'help' for commands.
> ls
hello.txt  15
hello.tsh  281
hello.tc  272
> cat hello.txt
Hello, tinyfs!
>
agent · Jul 17, 22:11:55
Good, that matches the README's illustration closely enough — updating the exact sizes (281/272 vs my placeholder 281/206):
Edit · Jul 17, 22:11:57 · toolu_018nE4aPcUX6W8YRxgYeKLye
/home/martin/workspace/prototypes/boot-to-claude/README.md
- > ls
- hello.txt  15
- hello.tsh  281
- hello.tc   206
- > cat hello.txt
+ > ls
+ hello.txt  15
+ hello.tsh  281
+ hello.tc   272
+ > cat hello.txt
result
The file /home/martin/workspace/prototypes/boot-to-claude/README.md has been updated successfully. (file state is current in your context — no need to Read it back)
Bash · Jul 17, 22:12:02 · toolu_01KAb25XyX3rMfp9zMghcnqY
Final clean-build full test run
$ cd /home/martin/workspace/prototypes/boot-to-claude
pgrep -af qemu-system-i386 || echo "no stray qemu"
make clean >/dev/null && ./tools/run_tests.sh 2>&1 | tail -5
result
2114264 /bin/bash -c source /home/martin/.claude/shell-snapshots/snapshot-bash-1784307599835-hlkzgk.sh 2>/dev/null || true && shopt -u extglob 2>/dev/null || true && eval 'cd /home/martin/workspace/prototypes/boot-to-claude pgrep -af qemu-system-i386 || echo "no stray qemu" make clean >/dev/null && ./tools/run_tests.sh 2>&1 | tail -5' < /dev/null && pwd -P >| /tmp/claude-383a-cwd
ld: warning: build/kernel.elf has a LOAD segment with RWX permissions
run_tests: PASS
qemu-system-i386: terminating on signal 15 from pid 2114269 (bash)
agent · Jul 17, 22:12:17
Clean build from scratch, full test suite green. Now the final cleanliness check on `.gitignore` and git history:
Bash · Jul 17, 22:12:18 · toolu_01YL7toEW7S4g6uxsz27g9TX
Verify git status is clean and .gitignore covers all build artifacts
$ cd /home/martin/workspace/prototypes/boot-to-claude
git status --short
echo "---"
git status --short --ignored | grep '^!!' | head -20
echo "---clean check for stray untracked non-ignored files---"
git clean -ndx
result
?? README.md
?? docs/scripting-language.md
---
!! build/
---clean check for stray untracked non-ignored files---
Would remove README.md
Would remove build/
Would remove docs/scripting-language.md
agent · Jul 17, 22:12:23
`.gitignore` correctly covers `build/`; only the two new doc files are untracked. Committing Phase 7.
Bash · Jul 17, 22:12:23 · toolu_01ACb7Xg842pjHZeKKnXjeu1
Stage Phase 7 files
$ cd /home/martin/workspace/prototypes/boot-to-claude
git add README.md docs/scripting-language.md
git status --short
result
A  README.md
A  docs/scripting-language.md