[#BloombergLP-bdlb-Tokenizer]
= xref:BloombergLP.adoc[BloombergLP]::xref:BloombergLP/bdlb.adoc[bdlb]::Tokenizer
:relfileprefix: ../../
:mrdocs:

Provides sequential read‐only access to tokens with soft and hard delimiters.

== Synopsis

Declared in `<bdlb_tokenizer.h>`

[source,cpp,subs="verbatim,replacements,macros,-callouts"]
----
class Tokenizer;
----

== Description

This class provides (read‐only) sequential access to tokens delimited by two user‐supplied character sets consisting, respectively, of _soft_ and _hard_ delimiters characters. Access to the previous and current (trailing) delimiter, as well as to the current token itself, is provided efficiently via `bslstl::StringRef`.

== Type Aliases

[cols="1,4"]
|===
| Name| Description
| xref:BloombergLP/bdlb/Tokenizer/iterator.adoc[`iterator`] 
| Input‐iterator type over tokens in the bound input string.
|===


== Member Functions

[cols="1,4"]
|===
| Name| Description
| xref:BloombergLP/bdlb/Tokenizer/2constructor-08.adoc[`Tokenizer`]         [.small]#[constructor]#
| Constructors
| xref:BloombergLP/bdlb/Tokenizer/2destructor.adoc[`~Tokenizer`] [.small]#[destructor]#
| Destroy this object.
| xref:BloombergLP/bdlb/Tokenizer/begin.adoc[`begin`] 
| Return an iterator referring to the first token in the input string.
| xref:BloombergLP/bdlb/Tokenizer/end.adoc[`end`] 
| Return an iterator referring past the last token in the input string.
| xref:BloombergLP/bdlb/Tokenizer/hasPreviousSoft.adoc[`hasPreviousSoft`] 
| Return `true` if the previous delimiter contains a soft delimiter.
| xref:BloombergLP/bdlb/Tokenizer/hasTrailingSoft.adoc[`hasTrailingSoft`] 
| Return `true` if the trailing delimiter contains a soft delimiter.
| xref:BloombergLP/bdlb/Tokenizer/isPreviousHard.adoc[`isPreviousHard`] 
| Return `true` if the previous delimiter contains a hard delimiter.
| xref:BloombergLP/bdlb/Tokenizer/isTrailingHard.adoc[`isTrailingHard`] 
| Return `true` if the trailing delimiter contains a hard delimiter.
| xref:BloombergLP/bdlb/Tokenizer/isValid.adoc[`isValid`] 
| Return `true` if this object's iteration state is valid.
| xref:BloombergLP/bdlb/Tokenizer/operator_inc-02.adoc[`operator++`] 
| Advance to the next token and return a reference to this object.
| xref:BloombergLP/bdlb/Tokenizer/previousDelimiter.adoc[`previousDelimiter`] 
| Return a reference to the non‐modifiable previous delimiter.
| xref:BloombergLP/bdlb/Tokenizer/reset-04.adoc[`reset`] 
| Rebind this object to refer to the specified sequence of `input` characters.
| xref:BloombergLP/bdlb/Tokenizer/token.adoc[`token`] 
| Return a reference to the non‐modifiable current token.
| xref:BloombergLP/bdlb/Tokenizer/trailingDelimiter.adoc[`trailingDelimiter`] 
| Return a reference to the non‐modifiable current trailing delimiter.
|===



[.small]#Created with https://www.mrdocs.com[MrDocs]#