This module implements the token recognizer when it is not in the mode that recognizes here-documents, and in cases where the recognition of tokens is independent from the parsing context, as specified by:
http://pubs.opengroup.org/onlinepubs/9699919799/ 2.3 Token Recognition
val token :
PrelexerState.t ->
Lexing.lexbuf ->
(Pretoken.t * Lexing.position * Lexing.position) listtoken b l advances in the lexbuf l, and produces a list of pretokens from l, using a queue of symbols in the buffer b
val single_quotes :
Buffer.t ->
PrelexerState.t ->
Lexing.lexbuf ->
PrelexerState.tsingle_quotes b l advances in the lexing buffer l to recognize a word between single quotes. This is used by here-document recognition.