[#BloombergLP-bdlb-Tokenizer-2constructor-0a]
= xref:BloombergLP.adoc[BloombergLP]::xref:BloombergLP/bdlb.adoc[bdlb]::xref:BloombergLP/bdlb/Tokenizer.adoc[Tokenizer]::Tokenizer
:relfileprefix: ../../../
:mrdocs:

Create a `Tokenizer` bound to `input` using the specified soft delimiters&period;

== Synopsis

Declared in `&lt;bdlb&lowbar;tokenizer&period;h&gt;`

[source,cpp,subs="verbatim,replacements,macros,-callouts"]
----
Tokenizer(
    char const* input,
    std::string&lowbar;view const& soft);
----

== Description

Create a `Tokenizer` object bound to the specified sequence of `input` characters having the specified set of (unique) `soft` delimiter characters to be used to separate _tokens_ (i&period;e&period;, maximal sequence of non&hyphen;delimiter characters) in `input`&period; Optionally specify a disjoint set of (unique) `hard` delimiter characters to be used to explicitly terminate tokens&period; Delimiters within `input` consist of a maximal sequence of one or more delimiter characters, at most one of which may be _hard_&semi; when there is a contiguous sequence of delimiter characters containing two or more _hard_ delimiter characters in `input`, any intervening _soft_ delimiter characters are associated with the previous (_hard_) delimiter&period; Any leading soft delimiter characters &hyphen;&hyphen; i&period;e&period;, those preceding the first _token_ or _hard_ delimiter character (referred to as the _leader_) &hyphen;&hyphen; are available immediately after construction via the `previousDelimiter` method&period; The behavior is undefined unless all supplied delimiter characters are unique&period; Note that the behavior is also undefined if this object is used in any way (other than to reset or destroy it) after its underlying `input` string is modified&period; Also note that the current token and (trailing) delimiter may be accessed only while this object is in the valid state&semi; however, the previous delimiter (or _leader_) is always accessible&period; Also note that all token and delimiter strings are returned as references into the underlying `input` string, and hence remain valid so long as that string is not modified or destroyed &hyphen;&hyphen; irrespective of the state (or even the existence) of this object&period; Finally note that supplying a default constructed `string&lowbar;view` is equivalent to supplying an empty c&hyphen;string (i&period;e&period;, &quot;&quot;)&period;

== Parameters

[cols="1,4"]
|===
| Name| Description
| *input*
| null&hyphen;terminated character sequence to tokenize
| *soft*
| set of unique soft delimiter characters
|===


[.small]#Created with https://www.mrdocs.com[MrDocs]#
