[#BloombergLP-bdlb-TokenizerIterator]
= xref:BloombergLP.adoc[BloombergLP]::xref:BloombergLP/bdlb.adoc[bdlb]::TokenizerIterator
:relfileprefix: ../../
:mrdocs:

This class provides a C++‐standards‐conforming input iterator over the tokens in the input string suppled at construction (along with the designation of _soft_ and _hard_ delimiter characters) to a `Tokenizer` object. Tokens are returned, using a `bslstl::StringRef` ‐‐ by value ‐‐ that means the iterated references remain valid until the underlying input string itself is modified or destroyed. Note that all iterators are invalidated whenever the input string in the parent `Tokenizer` change.

== Synopsis

Declared in `<bdlb_tokenizer.h>`

[source,cpp,subs="verbatim,replacements,macros,-callouts"]
----
class TokenizerIterator;
----

== Type Aliases

[cols="1,4"]
|===
| Name| Description
| xref:BloombergLP/bdlb/TokenizerIterator/difference_type.adoc[`difference_type`] 
| Signed type used to represent distances between iterators.
| xref:BloombergLP/bdlb/TokenizerIterator/iterator_category.adoc[`iterator_category`] 
| Tag identifying this type as an input iterator.
| xref:BloombergLP/bdlb/TokenizerIterator/pointer.adoc[`pointer`] 
| Proxy type returned by `operator‐>`.
| xref:BloombergLP/bdlb/TokenizerIterator/reference.adoc[`reference`] 
| Reference type returned by `operator*`.
| xref:BloombergLP/bdlb/TokenizerIterator/value_type.adoc[`value_type`] 
| Element type yielded by this iterator (`bslstl::StringRef`).
|===


== Member Functions

[cols="1,4"]
|===
| Name| Description
| xref:BloombergLP/bdlb/TokenizerIterator/2constructor-0e.adoc[`TokenizerIterator`]         [.small]#[constructor]#
| Create a `TokenizerIterator` object having the value of the specified `origin` iterator.
| xref:BloombergLP/bdlb/TokenizerIterator/operator_assign.adoc[`operator=`] 
| Assign to this object the value of the specified `rhs` iterator, and return a reference providing modifiable access to this object.
| xref:BloombergLP/bdlb/TokenizerIterator/operator_star.adoc[`operator*`] 
| Return a reference to the non‐modifiable current token (i.e., maximal sequence of non‐delimiter characters) in the input string. The returned reference remains valid so long as the underlying input is not modified or destroyed ‐‐ irrespective of the state (or existence) of this object. The behavior is undefined unless the iteration state of this object is initially valid, or if the underlying input has been modified or destroyed since this object was created.
| xref:BloombergLP/bdlb/TokenizerIterator/operator_inc.adoc[`operator++`] 
| Advance the iteration state of this object to refer to the next token in the underlying input sequence, and return a reference providing modifiable access to this object. The behavior is undefined unless the iteration state of this object is initially valid, or if the underlying input has been modified or destroyed since this object was created.
| xref:BloombergLP/bdlb/TokenizerIterator/operator_ptr.adoc[`operator‐>`] 
| Return a proxy object containing the non‐modifiable current token (i.e., maximal sequence of non‐delimiter characters) in the input string. The returned proxy remains valid so long as the underlying input is not modified or destroyed ‐‐ irrespective of the state (or existence) of this object. The behavior is undefined unless the iteration state of this object is initially valid, or if the underlying input has been modified or destroyed since this object was created.
|===


== Friends

[cols="1,4"]
|===
| Name| Description
| `xref:BloombergLP/bdlb/operator_not_eq-098.adoc[BloombergLP::bdlb::operator!=]`
| Return `true` if the specified `lhs` and `rhs` objects do not have the same value, and `false` otherwise. The behaviour is undefined unless the iterators returned by the same `Tokenizer` object, or if the underlying input has been modified or destroyed since any of those objects were created.
| `xref:BloombergLP/bdlb/operator_eq-026.adoc[BloombergLP::bdlb::operator==]`
| Return `true` if the specified `lhs` and `rhs` objects have the same value, and `false` otherwise. Two `TokenizerIterator` objects have the same value if both of them are pointing to the same token within the same tokenized string or if they both point past the tokenized string. The behaviour is undefined unless the iterators returned by the same `Tokenizer` object, or if the underlying input has been modified or destroyed since any of those objects were created.
| `xref:BloombergLP/bdlb/Tokenizer.adoc[BloombergLP::bdlb::Tokenizer]`
| This class provides (read‐only) sequential access to tokens delimited by two user‐supplied character sets consisting, respectively, of _soft_ and _hard_ delimiters characters. Access to the previous and current (trailing) delimiter, as well as to the current token itself, is provided efficiently via `bslstl::StringRef`.
|===

== Non-Member Functions

[cols="1,4"]
|===
| Name| Description
| xref:BloombergLP/bdlb/operator_inc-07.adoc[`operator++`]
| Advance the iteration state of the specified `object` to refer to the next token in the underlying input sequence, and return a copy of this object prior advancing the iteration state. The behavior is undefined unless the iteration state of this object is initially valid, or if the underlying input has been modified or destroyed since this object was created.
|===


[.small]#Created with https://www.mrdocs.com[MrDocs]#