[#BloombergLP-bdlb-Tokenizer_Data-2constructor-04]
= xref:BloombergLP.adoc[BloombergLP]::xref:BloombergLP/bdlb.adoc[bdlb]::xref:BloombergLP/bdlb/Tokenizer_Data.adoc[Tokenizer_Data]::Tokenizer_Data
:relfileprefix: ../../../
:mrdocs:

Create a `Tokenizer_Data` object and load the `d_charTypes` data member such that it has the same value _as_ _if_ this (overly prescriptive) algorithm were used: (I) initialize each entry in `d_charTypes` array to a value indicating that the character having that `index` as its (e.g., ASCII) representation is a _token_ character; (II) then, for each character in the specified `softDelimiters` sequence, overwrite the element at the corresponding index in `d_charTypes` with a value that indicates that the character is a _soft_ delimiter character; (III) finally, for each character in the specified `hardDelimiters` sequence, overwrite the element at the corresponding index with a distinct value that indicates the character is a _hard_ delimiter* character. Note that duplicate delimiter characters in the respective inputs are naturally ignored, and that a character that appears in both sets would naturally be considered _hard_. Also note that it is entirely reasonable to state, in any public interface, that the behavior is undefined unless the characters in the union of the two delimiter sequences are unique.

== Synopsis

Declared in `<bdlb_tokenizer.h>`

[source,cpp,subs="verbatim,replacements,macros,-callouts"]
----
explicit
Tokenizer_Data(std::string_view const& softDelimiters);
----


[.small]#Created with https://www.mrdocs.com[MrDocs]#