Provides sequential read‐only access to tokens with soft and hard delimiters.

Synopsis

Declared in <bdlb_tokenizer.h>

class Tokenizer;

Description

This class provides (read‐only) sequential access to tokens delimited by two user‐supplied character sets consisting, respectively, of soft and hard delimiters characters. Access to the previous and current (trailing) delimiter, as well as to the current token itself, is provided efficiently via bslstl::StringRef.

Type Aliases

Name

Description

iterator

Input‐iterator type over tokens in the bound input string.

Member Functions

Name

Description

Tokenizer [constructor]

Constructors

~Tokenizer [destructor]

Destroy this object.

begin

Return an iterator referring to the first token in the input string.

end

Return an iterator referring past the last token in the input string.

hasPreviousSoft

Return true if the previous delimiter contains a soft delimiter.

hasTrailingSoft

Return true if the trailing delimiter contains a soft delimiter.

isPreviousHard

Return true if the previous delimiter contains a hard delimiter.

isTrailingHard

Return true if the trailing delimiter contains a hard delimiter.

isValid

Return true if this object's iteration state is valid.

operator++

Advance to the next token and return a reference to this object.

previousDelimiter

Return a reference to the non‐modifiable previous delimiter.

reset

Rebind this object to refer to the specified sequence of input characters.

token

Return a reference to the non‐modifiable current token.

trailingDelimiter

Return a reference to the non‐modifiable current trailing delimiter.

Created with MrDocs