[#BloombergLP-bdlb-Tokenizer-2constructor-0a]
= xref:BloombergLP.adoc[BloombergLP]::xref:BloombergLP/bdlb.adoc[bdlb]::xref:BloombergLP/bdlb/Tokenizer.adoc[Tokenizer]::Tokenizer
:relfileprefix: ../../../
:mrdocs:

Create a `Tokenizer` object bound to the specified sequence of `input` characters having the specified set of (unique) `soft` delimiter characters to be used to separate _tokens_ (i.e., maximal sequence of non‐delimiter characters) in `input`. Optionally specify a disjoint set of (unique) `hard` delimiter characters to be used to explicitly terminate tokens. Delimiters within `input` consist of a maximal sequence of one or more delimiter characters, at most one of which may be _hard_; when there is a contiguous sequence of delimiter characters containing two or more _hard_ delimiter characters in `input`, any intervening _soft_ delimiter characters are associated with the previous (_hard_) delimiter. Any leading soft delimiter characters ‐‐ i.e., those preceding the first _token_ or _hard_ delimiter character (referred to as the _leader_) ‐‐ are available immediately after construction via the `previousDelimiter` method. The behavior is undefined unless all supplied delimiter characters are unique. Note that the behavior is also undefined if this object is used in any way (other than to reset or destroy it) after its underlying `input` string is modified. Also note that the current token and (trailing) delimiter may be accessed only while this object is in the valid state; however, the previous delimiter (or _leader_) is always accessible. Also note that all token and delimiter strings are returned as references into the underlying `input` string, and hence remain valid so long as that string is not modified or destroyed ‐‐ irrespective of the state (or even the existence) of this object. Finally note that supplying a default constructed `string_view` is equivalent to supplying an empty c‐string (i.e., "").

== Synopsis

Declared in `<bdlb_tokenizer.h>`

[source,cpp,subs="verbatim,replacements,macros,-callouts"]
----
Tokenizer(
    char const* input,
    std::string_view const& soft);
----


[.small]#Created with https://www.mrdocs.com[MrDocs]#