[#BloombergLP-bdljsn-Tokenizer-2constructor]
= xref:BloombergLP.adoc[BloombergLP]::xref:BloombergLP/bdljsn.adoc[bdljsn]::xref:BloombergLP/bdljsn/Tokenizer.adoc[Tokenizer]::Tokenizer
:relfileprefix: ../../../
:mrdocs:

Create a `Tokenizer` object&period; Optionally specify a `basicAllocator` used to supply memory&period; If `basicAllocator` is 0, the currently installed default allocator is used&period; By default, the `conformanceMode` is `e&lowbar;RELAXED` and the value of the `Tokenizer` options are&colon; &grave; allowConsecutiveSeparators() &equals;&equals; true&semi; allowFormFeedAsWhitespace() &equals;&equals; true&semi; allowHeterogeneousArrays() &equals;&equals; true&semi; allowNonUtf8StringLiterals() &equals;&equals; true&semi; allowStandAloneValues() &equals;&equals; true&semi; allowTrailingTopLevelComma() &equals;&equals; true&semi; allowUnescapedControlCharacters() &equals;&equals; true&semi; &grave; The `reset` method must be called before any calls to `advanceToNextToken` or `resetStreamBufGetPointer`&period;

== Synopsis

Declared in `&lt;bdljsn&lowbar;tokenizer&period;h&gt;`

[source,cpp,subs="verbatim,replacements,macros,-callouts"]
----
explicit
Tokenizer(xref:BloombergLP/bslma/Allocator.adoc[bslma::Allocator]* basicAllocator = 0);
----


[.small]#Created with https://www.mrdocs.com[MrDocs]#
