Understand Syscall ABI: Registers, Numbers, Kernel Entry

Crossing the User-Kernel Boundary

The syscall ABI defines how CPU state crosses privilege rings. On x86-64 Linux, user code loads rax with the syscall number, places arguments in designated registers, executes syscall, and reads the return from rax.

Register Map

x86-64 Linux passes syscall arguments in fixed registers:

• Syscall number: rax
• Arg 1: rdi, Arg 2: rsi, Arg 3: rdx
• Arg 4: r10, Arg 5: r8, Arg 6: r9

For example, read(3, buf, 1024) maps to NR 0, fd in rdi, buffer in rsi, count in rdx.

Why This Matters in Production

The r10-instead-of-rcx quirk you'll learn here exists because the x86-64 syscall instruction itself clobbers rcx and r11 to hold the return address and flags, a detail glibc's syscall wrappers and Go's runtime assembly both have to account for by hand. This ABI is exactly what objdump -d reveals when reverse-engineering a stripped binary, and mismatching it is a classic bug in hand-written syscall stubs for new libc ports.

Debugging Checklist

Before you call the implementation done, walk failure modes on purpose. Test empty structures, single-element edge cases, maximum concurrency, and errno paths that must not crash the program. OS code usually fails in production when happy-path tests pass but invariants break under contention or memory pressure.

Implementation Notes

Keep structures small and name fields after kernel counterparts when possible. That lets you read man pages and kernel source side by side while you work. Print observable events during development; remove noisy logs once tests pass reliably.

For This Exercise

You will document the ABI for three syscalls and write a tiny assembly stub that invokes one. This exercise requires explaining why rcx and r11 are clobbered by the syscall instruction itself.