BloombergLP::bdlb::Tokenizer::Tokenizer

Create a Tokenizer object bound to the specified sequence of input characters having the specified set of (unique) soft delimiter characters to be used to separate tokens (i.e., maximal sequence of non‐delimiter characters) in input. Optionally specify a disjoint set of (unique) hard delimiter characters to be used to explicitly terminate tokens. Delimiters within input consist of a maximal sequence of one or more delimiter characters, at most one of which may be hard; when there is a contiguous sequence of delimiter characters containing two or more hard delimiter characters in input, any intervening soft delimiter characters are associated with the previous (hard) delimiter. Any leading soft delimiter characters ‐‐ i.e., those preceding the first token or hard delimiter character (referred to as the leader) ‐‐ are available immediately after construction via the previousDelimiter method. The behavior is undefined unless all supplied delimiter characters are unique. Note that the behavior is also undefined if this object is used in any way (other than to reset or destroy it) after its underlying input string is modified. Also note that the current token and (trailing) delimiter may be accessed only while this object is in the valid state; however, the previous delimiter (or leader) is always accessible. Also note that all token and delimiter strings are returned as references into the underlying input string, and hence remain valid so long as that string is not modified or destroyed ‐‐ irrespective of the state (or even the existence) of this object. Finally note that supplying a default constructed string_view is equivalent to supplying an empty c‐string (i.e., "").