Identifier and Keyword Recognition

Identifiers vs Keywords

Identifiers and keywords look identical in the source: a letter followed by letters and digits. The lexer reads the full spelling, then checks a keyword table. Clang and GCC both use hash tables or trie structures for this lookup; CPython caches interned identifier strings.

Recognition Algorithm

Start at the first alphabetic character (or underscore). Consume while characters are alphanumeric or underscore. Buffer the spelling. If the spelling matches a reserved word in the keyword table, emit a keyword token; otherwise emit an identifier token carrying the name.

cCopy
Loading…

• Keywords are a fixed set defined by the language grammar
• Identifiers carry the spelling for the symbol table later
• Case sensitivity matters: C treats While and while differently
• Underscore-prefixed names are identifiers, not keywords

Connecting to real compilers

Production compilers embed this step inside a longer pipeline. GCC flows through cpp, cc1, assembly, and ld; Clang uses the driver, Sema, LLVM IR passes, and a target backend. LLVM bitcode, JVM bytecode, and WASM are other familiar IRs at the same layer. The exercise isolates one pass so you can test it alone before chaining it to the next stage.

Why for this exercise

You will implement identifier scanning and keyword disambiguation in your lexer. The exercise asks you to read alphanumeric sequences, consult a keyword table, and emit the correct token type so the parser can distinguish int the keyword from int as a variable name.