[#BloombergLP-bdljsn-Tokenizer]
= xref:BloombergLP.adoc[BloombergLP]::xref:BloombergLP/bdljsn.adoc[bdljsn]::Tokenizer
:relfileprefix: ../../
:mrdocs:

This `class` provides a mechanism for traversing JSON data stored in a `bsl&colon;&colon;streambuf` one node at a time and allows clients to access the data associated with that node, including its type and data value&period;

== Synopsis

Declared in `&lt;bdljsn&lowbar;tokenizer&period;h&gt;`

[source,cpp,subs="verbatim,replacements,macros,-callouts"]
----
class Tokenizer;
----

== Type Aliases

[cols="1,4"]
|===
| Name| Description
| xref:BloombergLP/bdljsn/Tokenizer/IntPtr.adoc[`IntPtr`] 
| Alias for `bsls&colon;&colon;Types&colon;&colon;IntPtr`&period;
| xref:BloombergLP/bdljsn/Tokenizer/Uint64.adoc[`Uint64`] 
| Alias for `bsls&colon;&colon;Types&colon;&colon;Uint64`&period;
|===


== Enums

[cols="1,4"]
|===
| Name| Description
| xref:BloombergLP/bdljsn/Tokenizer/_04enum.adoc[`Unnamed enum`] 
| Tokenizer constants&period;
| xref:BloombergLP/bdljsn/Tokenizer/ConformanceMode-03.adoc[`ConformanceMode`] 
| JSON conformance modes supported by this tokenizer&period;
| xref:BloombergLP/bdljsn/Tokenizer/TokenType-01.adoc[`TokenType`] 
| Token types returned by the tokenizer&period;
|===


== Member Functions

[cols="1,4"]
|===
| Name| Description
| xref:BloombergLP/bdljsn/Tokenizer/2constructor.adoc[`Tokenizer`]         [.small]#[constructor]#
| Create a `Tokenizer` object&period; Optionally specify a `basicAllocator` used to supply memory&period; If `basicAllocator` is 0, the currently installed default allocator is used&period; By default, the `conformanceMode` is `e&lowbar;RELAXED` and the value of the `Tokenizer` options are&colon; &grave; allowConsecutiveSeparators() &equals;&equals; true&semi; allowFormFeedAsWhitespace() &equals;&equals; true&semi; allowHeterogeneousArrays() &equals;&equals; true&semi; allowNonUtf8StringLiterals() &equals;&equals; true&semi; allowStandAloneValues() &equals;&equals; true&semi; allowTrailingTopLevelComma() &equals;&equals; true&semi; allowUnescapedControlCharacters() &equals;&equals; true&semi; &grave; The `reset` method must be called before any calls to `advanceToNextToken` or `resetStreamBufGetPointer`&period;
| xref:BloombergLP/bdljsn/Tokenizer/2destructor.adoc[`&#126;Tokenizer`] [.small]#[destructor]#
| Destroy this object&period;
| xref:BloombergLP/bdljsn/Tokenizer/advanceToNextToken.adoc[`advanceToNextToken`] 
| Move to the next token in the data steam&period; Return 0 on success and a non&hyphen;zero value otherwise&period; Each call to `advanceToNextToken` invalidates the string references returned by the `value` accessor for prior nodes&period; This function _may_ fail to move to the next token if doing so would advanced past a character sequence that is not valid JSON, and is guaranteed to do so (fail to move) if `e&lowbar;RELAXED !&equals; conformanceMode()`&period; The behavior is undefined unless `reset` has been called&period;
| xref:BloombergLP/bdljsn/Tokenizer/allowConsecutiveSeparators.adoc[`allowConsecutiveSeparators`] 
| Return the value of the `allowConsecutiveSeparators` option of this tokenizer&period;
| xref:BloombergLP/bdljsn/Tokenizer/allowFormFeedAsWhitespace.adoc[`allowFormFeedAsWhitespace`] 
| Return the value of the `allowFormFeedAsWhitespace` option of this tokenizer&period;
| xref:BloombergLP/bdljsn/Tokenizer/allowHeterogenousArrays.adoc[`allowHeterogenousArrays`] 
| Return the value of the `allowHeterogenousArrays` option of this tokenizer&period;
| xref:BloombergLP/bdljsn/Tokenizer/allowNonUtf8StringLiterals.adoc[`allowNonUtf8StringLiterals`] 
| Return the value of the `allowNonUtf8StringLiterals` option of this tokenizer&period;
| xref:BloombergLP/bdljsn/Tokenizer/allowStandAloneValues.adoc[`allowStandAloneValues`] 
| Return the value of the `allowStandAloneValues` option of this tokenizer&period;
| xref:BloombergLP/bdljsn/Tokenizer/allowTrailingTopLevelComma.adoc[`allowTrailingTopLevelComma`] 
| Return the value of the `allowTrailingTopLevelComma` option of this tokenizer&period;
| xref:BloombergLP/bdljsn/Tokenizer/allowUnescapedControlCharacters.adoc[`allowUnescapedControlCharacters`] 
| Return the value of the `allowUnescapedControlCharacters` option of this tokenizer&period;
| xref:BloombergLP/bdljsn/Tokenizer/conformanceMode-0e.adoc[`conformanceMode`] 
| Return the `conformanceMode` of this tokenizer&period;
| xref:BloombergLP/bdljsn/Tokenizer/currentPosition.adoc[`currentPosition`] 
| Return the offset of the current octet being tokenized in the stream supplied to `reset`, or if an error occurred, the position where the failed attempt to tokenize a token occurred&period; Note that this operation is intended to provide additional information in the case of an error&period;
| xref:BloombergLP/bdljsn/Tokenizer/readOffset.adoc[`readOffset`] 
| Return the last read position relative to when `reset` was called&period; Note that `readOffset() &gt;&equals; currentPosition()` &hyphen;&hyphen; the `readOffset` is the offset of the last octet read from the stream supplied to `reset`, and is at or beyond the current position being tokenized&period;
| xref:BloombergLP/bdljsn/Tokenizer/readStatus.adoc[`readStatus`] 
| Return the status of the last call to `reloadStringBuffer()`&colon; &ast; 0 if `reloadStringBuffer()` has not been called or if a token was successfully read&period; &ast; `k&lowbar;EOF` (which is positive) if no data could be read before reaching EOF&period; &ast; a negative value if the `allowNonUtf8StringLiterals` option is `false` and a UTF&hyphen;8 error occurred&period; The specific value returned will be one of the enumerators of the `bdlde&colon;&colon;Utf8Util&colon;&colon;ErrorStatus` `enum` type indicating the nature of the UTF&hyphen;8 error&period;
| xref:BloombergLP/bdljsn/Tokenizer/reset.adoc[`reset`] 
| Reset this tokenizer to read data from the specified `streambuf`&period; Note that the reader will not be on a valid node until `advanceToNextToken` is called&period; Note that this function does not change the the `conformanceMode` nor the values of any of the individual token options&colon; &ast; `allowConsecutiveSeparators` &ast; `allowFormFeedAsWhitespace` &ast; `allowHeterogenousArrays` &ast; `allowNonUtf8StringLiterals` &ast; `allowStandAloneValues` &ast; `allowTrailingTopLevelComma` &ast; `allowUnescapedControlCharacters`
| xref:BloombergLP/bdljsn/Tokenizer/resetStreamBufGetPointer.adoc[`resetStreamBufGetPointer`] 
| Reset the get pointer of the `streambuf` held by this object to refer to the byte following the last processed byte, if the held `streambuf` supports seeking, and return an error otherwise leaving this object unchanged&period; Return 0 on success, and a non&hyphen;zero value otherwise&period; The behavior is undefined unless `reset` has been called&period; Note that after a successful function return users can read data from the `streambuf` that was specified during `reset` from where this object stopped&period; Also note that this call implies the end of processing for this object and any subsequent methods invoked on this object should only be done after calling `reset` and specifying a new `streambuf`&period;
| xref:BloombergLP/bdljsn/Tokenizer/setAllowConsecutiveSeparators.adoc[`setAllowConsecutiveSeparators`] 
| Set the `allowConsecutiveSeparators` option to the specified `value` and return a non&hyphen;`const` reference to this tokenizer&period; JSON defines two separator tokens&colon; the colon (`&colon;`) and the comma (`,`)&period; If the `allowConsecutiveSeparators` value is `true` this tokenizer will accept multiple consecutive sequences of a given separator (e&period;g&period;, `&quot;a&quot;&colon;&colon;b, &quot;c&quot;&colon;&colon;&colon;d` and `&quot;a&quot;&colon;b,, &quot;c&quot;&colon;d`, ,, &quot;e&quot;&colon;f&apos;) as if a single separator had appeared (i&period;e&period;, `&quot;a&quot;&colon;b, &quot;c&quot;&colon;d` and `&quot;a&quot;&colon;b, &quot;c&quot;&colon;d`, &quot;e&quot;&colon;f&apos;, respectively)&period; Otherwise the tokenizer returns an error when multiple consecutive colons are found&period; By default, the value of the `allowConsecutiveSeparators` option is `true`&period; The behavior is undefined unless `e&lowbar;RELAXED &equals;&equals; conformanceMode()`&period; Note that consecutive sequences using both tokens (e&period;g&period;, `&colon;&colon;,,&colon;&colon;`) is always an error&period;
| xref:BloombergLP/bdljsn/Tokenizer/setAllowFormFeedAsWhitespace.adoc[`setAllowFormFeedAsWhitespace`] 
| Set the `allowFormFeedAsWhitespace` option to the specified value and return a non&hyphen;`const` reference to this tokenizer&period; If the `allowFormFeedAsWhitespace` value is `true` the formfeed character (`f`) is recognized as a whitespace character in addition to &grave;
&grave;, `t`, `r`, and `v`&period; Otherwise, formfeed is disallowed as a whitespace character&period;
| xref:BloombergLP/bdljsn/Tokenizer/setAllowHeterogenousArrays.adoc[`setAllowHeterogenousArrays`] 
| Set the `allowHeterogenousArrays` option to the specified `value` and return a non&hyphen;`const` reference to this tokenizer&period; If the `allowHeterogenousArrays` value is `true` this tokenizer will successfully tokenize heterogeneous values within an array&period; If the option&apos;s value is `false` then the tokenizer will return an error for arrays having heterogeneous values&period; By default, the value of the `allowHeterogenousArrays` option is `true`&period; The behavior is undefined unless `e&lowbar;RELAXED &equals;&equals; conformanceMode()`&period;
| xref:BloombergLP/bdljsn/Tokenizer/setAllowNonUtf8StringLiterals.adoc[`setAllowNonUtf8StringLiterals`] 
| Set the `allowNonUtf8StringLiterals` option to the specified `value` and return a non&hyphen;`const` reference to this tokenizer&period; If the `allowNonUtf8StringLiterals` value is `false` this tokenizer will check string literal tokens for invalid UTF&hyphen;8, enter an error mode if it encounters a string literal token that has any content that is not UTF&hyphen;8, and fail to advance to subsequent tokens until `reset` is called&period; By default, the value of the `allowNonUtf8StringLiterals` option is `true`&period; The behavior is undefined unless `e&lowbar;RELAXED &equals;&equals; conformanceMode()`&period;
| xref:BloombergLP/bdljsn/Tokenizer/setAllowStandAloneValues.adoc[`setAllowStandAloneValues`] 
| Set the `allowStandAloneValues` option to the specified `value` and return a non&hyphen;`const` reference to this tokenizer&period; If the `allowStandAloneValues` value is `true` this tokenizer will successfully tokenize JSON values (strings and numbers)&period; If the option&apos;s value is `false` then the tokenizer will only tokenize complete JSON documents (JSON objects and arrays) and return an error for stand alone JSON values&period; By default, the value of the `allowStandAloneValues` option is `true`&period; The behavior is undefined unless `e&lowbar;RELAXED &equals;&equals; conformanceMode()`&period;
| xref:BloombergLP/bdljsn/Tokenizer/setAllowTrailingTopLevelComma.adoc[`setAllowTrailingTopLevelComma`] 
| Set the `allowTrailingTopLevelComma` option to the specified `value` and return a non&hyphen;`const` reference to this tokenizer&period; If the `allowTrailingTopLevelComma` value is `true` this tokenizer will successfully tokenize JSON values where a comma follows the top&hyphen;level JSON element&period; If the option&apos;s value is `false` then the tokenizer will reject documents with such trailing commas, such as `&lcub;&rcub;,`&period; By default, the value of the `allowTrailingTopLevelComma` option is `true` for backwards compatibility&period; Note that a document without any JSON elements is invalid whether or not it contains commas&period; The behavior is undefined unless `e&lowbar;RELAXED &equals;&equals; conformanceMode()`&period;
| xref:BloombergLP/bdljsn/Tokenizer/setAllowUnescapedControlCharacters.adoc[`setAllowUnescapedControlCharacters`] 
| Set the `allowUnescapedControlCharacters` option of this tokenizer to the specified `value`&period; If `true`, characters in the range `&lsqb; 0x00 &period;&period; 0x1F &rsqb;` are allowed in JSON strings&period; If the option is `false`, these characters must be represented by their six byte escape sequences `&lsqb; u0000 &period;&period; u001F &rsqb;`&period; Several values in that range are also (conveniently) represented by two byte sequences&colon; &grave; &quot; quotation mark reverse solidus &sol; solidus *backspace* f form feed line feed r carriage return t tab &grave; The `DEL` control character (`0x7F`) is accepted even in strict mode&period;
| xref:BloombergLP/bdljsn/Tokenizer/setConformanceMode.adoc[`setConformanceMode`] 
| Set the `conformanceMode` of this tokenizer to the specified `mode` and return a non&hyphen;`const` reference to this tokenizer&period; If `mode` is `e&lowbar;STRICT&lowbar;20240119` the option values of this tokenizer are set to be fully compliant with RFC8259 (see https&colon;&sol;&sol;www&period;rfc&hyphen;editor&period;org&sol;rfc&sol;rfc8259)
| xref:BloombergLP/bdljsn/Tokenizer/tokenType-00.adoc[`tokenType`] 
| Return the token type of the current token&period;
| xref:BloombergLP/bdljsn/Tokenizer/value.adoc[`value`] 
| Load into the specified `data` the value of the specified token if the current token&apos;s type is `e&lowbar;ELEMENT&lowbar;NAME` or `e&lowbar;ELEMENT&lowbar;VALUE` or leave `data` unmodified otherwise&period; Return 0 on success and a non&hyphen;zero value otherwise&period; Note that the returned `data` is only valid until the next manipulator call on this object&period;
|===



[.small]#Created with https://www.mrdocs.com[MrDocs]#
