Comment and escape sequence (such as string literal) are very exceptional from regular symbolic representation.
It's hard to understand for me how does regular lexical analyzers tokenize them.
How do lexical analyzers like lex
, flex
, or etc.. handle this kind of symbols? Is there a generic method? Or just case by case for each language?