d988991f9f
The compiler presently tokenizes the bodies of only function-like macro definitions from the command line, and does so crudely. Tokenize keyword-like macros too, get character literals right, and handle numeric constants correctly. (Also delete two needless functions noticed in characters.h.) Fixes https://github.com/llvm/llvm-project/issues/168077.