This module implements the token recognizer when it is not in the mode that recognizes here-documents, and in cases where the recognition of tokens is independent from the parsing context, as specified by:
http://pubs.opengroup.org/onlinepubs/9699919799/ 2.3 Token Recognition
val token :
PrelexerState.t ->
Stdlib.Lexing.lexbuf ->
(Pretoken.t * Stdlib.Lexing.position * Stdlib.Lexing.position) listtoken b l advances in the lexbuf l, and produces a list of pretokens from l, using a queue of symbols in the buffer b
val single_quotes : PrelexerState.t -> Stdlib.Lexing.lexbuf -> PrelexerState.tsingle_quotes b l advances in the lexing buffer l to recognize a word between single quotes. This is used by here-document recognition.