[#BloombergLP-bdlb-Tokenizer_Data]
= xref:BloombergLP.adoc[BloombergLP]::xref:BloombergLP/bdlb.adoc[bdlb]::Tokenizer_Data
:relfileprefix: ../../
:mrdocs:

This component‐private class is used to hold delimiter information. Each `Tokenizer` object will have, as a private data member, an object of this class, and will pass the address of that member to the (private) constructor of each `TokenizerIterator` object it issues: ` +‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐+ | ,‐‐‐‐‐‐‐‐‐‐‐‐‐‐. | | ( Tokenizer_Data ) | | `‐‐‐‐‐‐‐‐‐‐‐‐‐‐'\ | | | | | | ,‐‐‐‐*‐‐‐‐‐‐‐‐‐‐‐‐. | | | ( TokenizerIterator ) | | | /`‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐' | | | / | | ,‐‐‐‐*‐‐o‐. | | ( Tokenizer ) | | `‐‐‐‐‐‐‐‐‐' | +‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐+ bdlb_tokenizer `

== Synopsis

Declared in `<bdlb_tokenizer.h>`

[source,cpp,subs="verbatim,replacements,macros,-callouts"]
----
class Tokenizer_Data;
----

== Member Functions

[cols="1,4"]
|===
| Name| Description
| xref:BloombergLP/bdlb/Tokenizer_Data/2constructor-07.adoc[`Tokenizer_Data`]         [.small]#[constructor]#
| Constructors
| xref:BloombergLP/bdlb/Tokenizer_Data/inputType.adoc[`inputType`] 
| Return the input type of the specified `character`: 0 for token, 1 for soft delimiter, 2 for hard delimiter.
|===



[.small]#Created with https://www.mrdocs.com[MrDocs]#