[#BloombergLP-bdlb-Tokenizer_Data-2constructor-00]
= xref:BloombergLP.adoc[BloombergLP]::xref:BloombergLP/bdlb.adoc[bdlb]::xref:BloombergLP/bdlb/Tokenizer_Data.adoc[Tokenizer_Data]::Tokenizer_Data
:relfileprefix: ../../../
:mrdocs:

Create a `Tokenizer_Data` from soft and hard delimiter sets.

== Synopsis

Declared in `<bdlb_tokenizer.h>`

[source,cpp,subs="verbatim,replacements,macros,-callouts"]
----
Tokenizer_Data(
    std::string_view const& softDelimiters,
    std::string_view const& hardDelimiters);
----

== Description

Same as the preceding overload, also marking characters in the specified `hardDelimiters` as hard delimiter characters.

== Parameters

[cols="1,4"]
|===
| Name| Description
| *softDelimiters*
| characters treated as soft delimiters
| *hardDelimiters*
| characters treated as hard delimiters
|===


[.small]#Created with https://www.mrdocs.com[MrDocs]#