BloombergLP::bdlb::Tokenizer

Synopsis

Declared in <bdlb_tokenizer.h>

class Tokenizer;

Type Aliases

Name	Description
`iterator`	Input-iterator type over tokens in the bound input string.

Member Functions

Name	Description
`Tokenizer` [constructor]	Constructors
`~Tokenizer` [destructor]	Destroy this object.
`begin`	Return an iterator referring to the first token in this object's input string (the past-the-end iterator if this object iteration state is initially invalid). This reference remains valid as long as the underlying input has not been modified or destroyed since this object was most recently reset (or created).
`end`	Return an iterator referring to position beyond the last token in this object's input string. This reference remains valid as long as the underlying input has not been modified or destroyed since this object was most recently reset (or created).
`hasPreviousSoft`	Return `true` if the previous delimiter (or leader) contains a soft delimiter character, and `false` otherwise. The behavior is undefined if the underlying input itself has been modified or destroyed since this object was most recently reset (or created).
`hasTrailingSoft`	Return `true` if the current (trailing) delimiter contains a soft delimiter character, and `false` otherwise. The behavior is undefined if the iteration state of this object is initially invalid, or if the underlying input itself has been modified or destroyed since this object was most recently reset (or created).
`isPreviousHard`	Return `true` if the previous delimiter contains a hard-delimiter character, and `false` otherwise. The behavior is undefined if the underlying input itself has been modified or destroyed since this object was most recently reset (or created).
`isTrailingHard`	Return `true` if the current (trailing) delimiter contains a hard delimiter character, and `false` otherwise. The behavior is undefined if the iteration state of this object is initially invalid, or if the underlying input itself has been modified or destroyed since this object was most recently reset (or created).
`isValid`	Return `true` if the iteration state of this object is valid, and `false` otherwise. Note that the behavior of advancing the iteration state as well as accessing the current token or (trailing) delimiter is undefined unless the current iteration state of this object is valid.
`operator++`	Advance the iteration state of this object to refer to the next sequence of previous delimiter, current token, and current (trailing) delimiter in the underlying input sequence, and return a reference providing modifiable access to this object. The current delimiter reference becomes the previous one. If there is another token remaining in the input, the current token and delimiter are updated to refer to the respective new token and (trailing) delimiter values -- either of which (but not both) might be empty. If there are no tokens remaining in the input, the iteration state of this object becomes invalid. The behavior is undefined unless the iteration state of this object is initially valid, or if the underlying input has been modified or destroyed since this object was most recently reset (or created).
`previousDelimiter`	Return a reference to the non-modifiable previous delimiter (or leader) in the input string. The behavior is undefined if the underlying input has been modified or destroyed since this object was most recently reset (or created).
`reset`	`reset` overloads
`token`	Return a reference to the non-modifiable current token (i.e., maximal sequence of non-delimiter characters) in the input string. The returned reference remains valid so long as the underlying input is not modified or destroyed -- irrespective of the state (or existence) of this object. The behavior is undefined unless the iteration state of this object is initially valid, or if the underlying input has been modified or destroyed since this object was most recently reset (or created).
`trailingDelimiter`	Return a reference to the non-modifiable current (trailing) delimiter (maximal sequence of one or more delimiter characters containing at most one hard delimiter character) in the input string. The returned reference remains valid so long as the underlying input is not modified or destroyed -- irrespective of the state (or existence) of this object. The behavior is undefined unless the iteration state of this object is initially valid, or if the underlying input has been modified or destroyed since this object was most recently reset (or created).