Add Error Recovery: Skip Invalid Characters

Recovery Keeps the Compiler Useful

When a lexer hits an illegal character, quitting leaves the user with one error and no further analysis. Clang, Rust, and TypeScript all recover: skip the bad input, emit an error token, and keep scanning so the parser can report multiple issues in one compile.

Skip and Continue

On error, record the location and message, advance past the offending character (or run of garbage), and resume normal scanning. Some lexers insert a sentinel ERROR token so the parser can synchronize at a statement boundary.

cCopy
Loading…

• Never loop forever on the same position
• Report line and column with every recovery action
• Optionally skip to next whitespace or delimiter
• Let the parser decide whether to abort or continue

Connecting to real compilers

Production compilers embed this step inside a longer pipeline. GCC flows through cpp, cc1, assembly, and ld; Clang uses the driver, Sema, LLVM IR passes, and a target backend. LLVM bitcode, JVM bytecode, and WASM are other familiar IRs at the same layer. The exercise isolates one pass so you can test it alone before chaining it to the next stage.

Why for this exercise

You will add error recovery to your lexer so invalid characters are skipped after reporting. This exercise asks you to continue producing tokens after errors, enabling multi-error diagnostics in later compiler phases.