BloombergLP::bdlb::Tokenizer_Data::Tokenizer

Create a Tokenizer_Data from soft and hard delimiter sets.

Synopsis

Declared in <bdlb_tokenizer.h>

Tokenizer_Data(
    std::string_view const& softDelimiters,
    std::string_view const& hardDelimiters);

Same as the preceding overload, also marking characters in the specified hardDelimiters as hard delimiter characters.

Name

Description

softDelimiters

characters treated as soft delimiters

hardDelimiters

characters treated as hard delimiters

Created with MrDocs