Create a Tokenizer_Data from soft and hard delimiter sets.
Declared in <bdlb_tokenizer.h>
Tokenizer_Data(
std::string_view const& softDelimiters,
std::string_view const& hardDelimiters);Same as the preceding overload, also marking characters in the specified hardDelimiters as hard delimiter characters.
| Name | Description |
|---|---|
| softDelimiters | characters treated as soft delimiters |
| hardDelimiters | characters treated as hard delimiters |