[#BloombergLP-bdljsn-Tokenizer]
= xref:BloombergLP.adoc[BloombergLP]::xref:BloombergLP/bdljsn.adoc[bdljsn]::Tokenizer
:relfileprefix: ../../
:mrdocs:

Mechanism for traversing JSON data one node at a time.

== Synopsis

Declared in `<bdljsn_tokenizer.h>`

[source,cpp,subs="verbatim,replacements,macros,-callouts"]
----
class Tokenizer;
----

== Description

This `class` provides a mechanism for traversing JSON data stored in a `bsl::streambuf` one node at a time and allows clients to access the data associated with that node, including its type and data value.

== Type Aliases

[cols="1,4"]
|===
| Name| Description
| xref:BloombergLP/bdljsn/Tokenizer/IntPtr.adoc[`IntPtr`] 
| Alias for `bsls::Types::IntPtr`.
| xref:BloombergLP/bdljsn/Tokenizer/Uint64.adoc[`Uint64`] 
| Alias for `bsls::Types::Uint64`.
|===


== Enums

[cols="1,4"]
|===
| Name| Description
| xref:BloombergLP/bdljsn/Tokenizer/_04enum.adoc[`Unnamed enum`] 
| Tokenizer constants.
| xref:BloombergLP/bdljsn/Tokenizer/ConformanceMode-03.adoc[`ConformanceMode`] 
| JSON conformance modes supported by this tokenizer.
| xref:BloombergLP/bdljsn/Tokenizer/TokenType-01.adoc[`TokenType`] 
| Token types returned by the tokenizer.
|===


== Member Functions

[cols="1,4"]
|===
| Name| Description
| xref:BloombergLP/bdljsn/Tokenizer/2constructor.adoc[`Tokenizer`]         [.small]#[constructor]#
| Create a `Tokenizer` object.
| xref:BloombergLP/bdljsn/Tokenizer/2destructor.adoc[`~Tokenizer`] [.small]#[destructor]#
| Destroy this object.
| xref:BloombergLP/bdljsn/Tokenizer/advanceToNextToken.adoc[`advanceToNextToken`] 
| Move to the next token in the data stream.
| xref:BloombergLP/bdljsn/Tokenizer/allowConsecutiveSeparators.adoc[`allowConsecutiveSeparators`] 
| Return the value of the `allowConsecutiveSeparators` option of this tokenizer.
| xref:BloombergLP/bdljsn/Tokenizer/allowFormFeedAsWhitespace.adoc[`allowFormFeedAsWhitespace`] 
| Return the value of the `allowFormFeedAsWhitespace` option of this tokenizer.
| xref:BloombergLP/bdljsn/Tokenizer/allowHeterogenousArrays.adoc[`allowHeterogenousArrays`] 
| Return the value of the `allowHeterogenousArrays` option of this tokenizer.
| xref:BloombergLP/bdljsn/Tokenizer/allowNonUtf8StringLiterals.adoc[`allowNonUtf8StringLiterals`] 
| Return the value of the `allowNonUtf8StringLiterals` option of this tokenizer.
| xref:BloombergLP/bdljsn/Tokenizer/allowStandAloneValues.adoc[`allowStandAloneValues`] 
| Return the value of the `allowStandAloneValues` option of this tokenizer.
| xref:BloombergLP/bdljsn/Tokenizer/allowTrailingTopLevelComma.adoc[`allowTrailingTopLevelComma`] 
| Return the value of the `allowTrailingTopLevelComma` option of this tokenizer.
| xref:BloombergLP/bdljsn/Tokenizer/allowUnescapedControlCharacters.adoc[`allowUnescapedControlCharacters`] 
| Return the value of the `allowUnescapedControlCharacters` option of this tokenizer.
| xref:BloombergLP/bdljsn/Tokenizer/conformanceMode-0e.adoc[`conformanceMode`] 
| Return the `conformanceMode` of this tokenizer.
| xref:BloombergLP/bdljsn/Tokenizer/currentPosition.adoc[`currentPosition`] 
| Return the offset of the current octet being tokenized.
| xref:BloombergLP/bdljsn/Tokenizer/readOffset.adoc[`readOffset`] 
| Return the last read position relative to when `reset` was called.
| xref:BloombergLP/bdljsn/Tokenizer/readStatus.adoc[`readStatus`] 
| Return the status of the last call to `reloadStringBuffer()`.
| xref:BloombergLP/bdljsn/Tokenizer/reset.adoc[`reset`] 
| Reset this tokenizer to read data from the specified `streambuf`.
| xref:BloombergLP/bdljsn/Tokenizer/resetStreamBufGetPointer.adoc[`resetStreamBufGetPointer`] 
| Reset the held `streambuf` get pointer past the last processed byte.
| xref:BloombergLP/bdljsn/Tokenizer/setAllowConsecutiveSeparators.adoc[`setAllowConsecutiveSeparators`] 
| Set the `allowConsecutiveSeparators` option to the specified `value`.
| xref:BloombergLP/bdljsn/Tokenizer/setAllowFormFeedAsWhitespace.adoc[`setAllowFormFeedAsWhitespace`] 
| Set the `allowFormFeedAsWhitespace` option to the specified `value`.
| xref:BloombergLP/bdljsn/Tokenizer/setAllowHeterogenousArrays.adoc[`setAllowHeterogenousArrays`] 
| Set the `allowHeterogenousArrays` option to the specified `value`.
| xref:BloombergLP/bdljsn/Tokenizer/setAllowNonUtf8StringLiterals.adoc[`setAllowNonUtf8StringLiterals`] 
| Set the `allowNonUtf8StringLiterals` option to the specified `value`.
| xref:BloombergLP/bdljsn/Tokenizer/setAllowStandAloneValues.adoc[`setAllowStandAloneValues`] 
| Set the `allowStandAloneValues` option to the specified `value`.
| xref:BloombergLP/bdljsn/Tokenizer/setAllowTrailingTopLevelComma.adoc[`setAllowTrailingTopLevelComma`] 
| Set the `allowTrailingTopLevelComma` option to the specified `value`.
| xref:BloombergLP/bdljsn/Tokenizer/setAllowUnescapedControlCharacters.adoc[`setAllowUnescapedControlCharacters`] 
| Set the `allowUnescapedControlCharacters` option to the specified `value`.
| xref:BloombergLP/bdljsn/Tokenizer/setConformanceMode.adoc[`setConformanceMode`] 
| Set the `conformanceMode` of this tokenizer to the specified `mode`.
| xref:BloombergLP/bdljsn/Tokenizer/tokenType-00.adoc[`tokenType`] 
| Return the token type of the current token.
| xref:BloombergLP/bdljsn/Tokenizer/value.adoc[`value`] 
| Load the current token value into `data` when applicable.
|===



[.small]#Created with https://www.mrdocs.com[MrDocs]#