[#BloombergLP-bdlb-Tokenizer]
= xref:BloombergLP.adoc[BloombergLP]::xref:BloombergLP/bdlb.adoc[bdlb]::Tokenizer
:relfileprefix: ../../
:mrdocs:

Provides sequential read&hyphen;only access to tokens with soft and hard delimiters&period;

== Synopsis

Declared in `&lt;bdlb&lowbar;tokenizer&period;h&gt;`

[source,cpp,subs="verbatim,replacements,macros,-callouts"]
----
class Tokenizer;
----

== Description

This class provides (read&hyphen;only) sequential access to tokens delimited by two user&hyphen;supplied character sets consisting, respectively, of _soft_ and _hard_ delimiters characters&period; Access to the previous and current (trailing) delimiter, as well as to the current token itself, is provided efficiently via `bslstl&colon;&colon;StringRef`&period;

== Type Aliases

[cols="1,4"]
|===
| Name| Description
| xref:BloombergLP/bdlb/Tokenizer/iterator.adoc[`iterator`] 
| Input&hyphen;iterator type over tokens in the bound input string&period;
|===


== Member Functions

[cols="1,4"]
|===
| Name| Description
| xref:BloombergLP/bdlb/Tokenizer/2constructor-08.adoc[`Tokenizer`]         [.small]#[constructor]#
| Constructors
| xref:BloombergLP/bdlb/Tokenizer/2destructor.adoc[`&#126;Tokenizer`] [.small]#[destructor]#
| Destroy this object&period;
| xref:BloombergLP/bdlb/Tokenizer/begin.adoc[`begin`] 
| Return an iterator referring to the first token in the input string&period;
| xref:BloombergLP/bdlb/Tokenizer/end.adoc[`end`] 
| Return an iterator referring past the last token in the input string&period;
| xref:BloombergLP/bdlb/Tokenizer/hasPreviousSoft.adoc[`hasPreviousSoft`] 
| Return `true` if the previous delimiter contains a soft delimiter&period;
| xref:BloombergLP/bdlb/Tokenizer/hasTrailingSoft.adoc[`hasTrailingSoft`] 
| Return `true` if the trailing delimiter contains a soft delimiter&period;
| xref:BloombergLP/bdlb/Tokenizer/isPreviousHard.adoc[`isPreviousHard`] 
| Return `true` if the previous delimiter contains a hard delimiter&period;
| xref:BloombergLP/bdlb/Tokenizer/isTrailingHard.adoc[`isTrailingHard`] 
| Return `true` if the trailing delimiter contains a hard delimiter&period;
| xref:BloombergLP/bdlb/Tokenizer/isValid.adoc[`isValid`] 
| Return `true` if this object&apos;s iteration state is valid&period;
| xref:BloombergLP/bdlb/Tokenizer/operator_inc-02.adoc[`operator&plus;&plus;`] 
| Advance to the next token and return a reference to this object&period;
| xref:BloombergLP/bdlb/Tokenizer/previousDelimiter.adoc[`previousDelimiter`] 
| Return a reference to the non&hyphen;modifiable previous delimiter&period;
| xref:BloombergLP/bdlb/Tokenizer/reset-04.adoc[`reset`] 
| Rebind this object to refer to the specified sequence of `input` characters&period;
| xref:BloombergLP/bdlb/Tokenizer/token.adoc[`token`] 
| Return a reference to the non&hyphen;modifiable current token&period;
| xref:BloombergLP/bdlb/Tokenizer/trailingDelimiter.adoc[`trailingDelimiter`] 
| Return a reference to the non&hyphen;modifiable current trailing delimiter&period;
|===



[.small]#Created with https://www.mrdocs.com[MrDocs]#
