[#BloombergLP-bdljsn-Tokenizer-2constructor]
= xref:BloombergLP.adoc[BloombergLP]::xref:BloombergLP/bdljsn.adoc[bdljsn]::xref:BloombergLP/bdljsn/Tokenizer.adoc[Tokenizer]::Tokenizer
:relfileprefix: ../../../
:mrdocs:

Create a `Tokenizer` object. Optionally specify a `basicAllocator` used to supply memory. If `basicAllocator` is 0, the currently installed default allocator is used. By default, the `conformanceMode` is `e_RELAXED` and the value of the `Tokenizer` options are: ` allowConsecutiveSeparators() == true; allowFormFeedAsWhitespace() == true; allowHeterogeneousArrays() == true; allowNonUtf8StringLiterals() == true; allowStandAloneValues() == true; allowTrailingTopLevelComma() == true; allowUnescapedControlCharacters() == true; ` The `reset` method must be called before any calls to `advanceToNextToken` or `resetStreamBufGetPointer`.

== Synopsis

Declared in `<bdljsn_tokenizer.h>`

[source,cpp,subs="verbatim,replacements,macros,-callouts"]
----
explicit
Tokenizer(xref:BloombergLP/bslma/Allocator.adoc[bslma::Allocator]* basicAllocator = 0);
----


[.small]#Created with https://www.mrdocs.com[MrDocs]#