Token Struct Definition

Tokens Are the Lexer's Contract

Every production compiler starts by carving source text into tokens. Clang's Token class, GCC's cpp_token, and CPython's tokenizer all carry the same core fields: a kind, the original lexeme, and a source location. Get this struct wrong and every later phase pays for it.

What Belongs in a Token

A token is the smallest meaningful unit the parser consumes. For int x = 42; the lexer must emit distinct tokens for the keyword, identifier, operator, literal, and semicolon. Each token needs enough metadata to drive parsing and diagnostics.

cCopy
Loading…

• KW_INT with lexeme int
• IDENT with lexeme x
• INT_LIT with value 42
• SEMI with lexeme ;

Line and column numbers are what let Rust and Clang print caret diagnostics instead of vague file-level errors.

Connecting to real compilers

Production compilers embed this step inside a longer pipeline. GCC flows through cpp, cc1, assembly, and ld; Clang uses the driver, Sema, LLVM IR passes, and a target backend. LLVM bitcode, JVM bytecode, and WASM are other familiar IRs at the same layer. The exercise isolates one pass so you can test it alone before chaining it to the next stage.

Why for this exercise

You will implement the Token and TokenType definitions that every subsequent lexer function returns. This exercise asks you to nail down the data contract before writing scanning logic: enum variants for keywords, identifiers, literals, and punctuation, plus a struct wide enough to hold literal values and source positions.