BloombergLP::bdljsn::Tokenizer::Tokenizer

Create a Tokenizer object. Optionally specify a basicAllocator used to supply memory. If basicAllocator is 0, the currently installed default allocator is used. By default, the conformanceMode is e_RELAXED and the value of the Tokenizer options are: ` allowConsecutiveSeparators() == true; allowFormFeedAsWhitespace() == true; allowHeterogeneousArrays() == true; allowNonUtf8StringLiterals() == true; allowStandAloneValues() == true; allowTrailingTopLevelComma() == true; allowUnescapedControlCharacters() == true; ` The reset method must be called before any calls to advanceToNextToken or resetStreamBufGetPointer.