Lexer Error Handling

Lexers Must Not Panic on Bad Input

Real compilers encounter garbage characters, unterminated strings, and malformed numeric literals constantly. Clang's lexer reports precise diagnostics and recovers so parsing can continue. A lexer that aborts on the first unexpected byte makes the compiler useless on incomplete or generated code.

Error Reporting Strategy

When the lexer sees an invalid character, it should record an error with line and column, skip or consume the offending input, and return either an error token or continue scanning. Never read past the buffer end. Pair every error with enough context that the user can fix the source.

cCopy
Loading…

• Emit TOK_ERROR or a dedicated invalid token for the parser to handle
• Advance past the bad character to avoid infinite loops
• Track line/column so diagnostics point at the right spot
• Distinguish errors (recoverable) from fatal internal failures

Connecting to real compilers

Production compilers embed this step inside a longer pipeline. GCC flows through cpp, cc1, assembly, and ld; Clang uses the driver, Sema, LLVM IR passes, and a target backend. LLVM bitcode, JVM bytecode, and WASM are other familiar IRs at the same layer. The exercise isolates one pass so you can test it alone before chaining it to the next stage.

Why for this exercise

You will add error detection and reporting to your lexer so invalid input produces clear messages instead of silent mis-tokenization. This exercise requires handling unexpected characters and malformed literals while keeping the scan loop running.