Provides sequential read-only access to tokens with soft and hard delimiters.
Declared in <bdlb_tokenizer.h>
class Tokenizer;This class provides (read-only) sequential access to tokens delimited by two user-supplied character sets consisting, respectively, of soft and hard delimiters characters. Access to the previous and current (trailing) delimiter, as well as to the current token itself, is provided efficiently via bslstl::StringRef.
| Name | Description |
|---|---|
iterator | Input-iterator type over tokens in the bound input string. |
| Name | Description |
|---|---|
Tokenizer [constructor] | Constructors |
~Tokenizer [destructor] | Destroy this object. |
begin | Return an iterator referring to the first token in the input string. |
end | Return an iterator referring past the last token in the input string. |
hasPreviousSoft | Return true if the previous delimiter contains a soft delimiter. |
hasTrailingSoft | Return true if the trailing delimiter contains a soft delimiter. |
isPreviousHard | Return true if the previous delimiter contains a hard delimiter. |
isTrailingHard | Return true if the trailing delimiter contains a hard delimiter. |
isValid | Return true if this object's iteration state is valid. |
operator++ | Advance to the next token and return a reference to this object. |
previousDelimiter | Return a reference to the non-modifiable previous delimiter. |
reset | Rebind this object to refer to the specified sequence of input characters. |
token | Return a reference to the non-modifiable current token. |
trailingDelimiter | Return a reference to the non-modifiable current trailing delimiter. |