Delimiter type enumeration for unified token format.
Used in the new FieldToken format to indicate what follows the field value. This enables a more efficient token format where only field tokens are emitted.
Next token is a field (followed by field delimiter like comma)
Next token is a record delimiter (newline)
Delimiter type enumeration for unified token format.
Used in the new FieldToken format to indicate what follows the field value. This enables a more efficient token format where only field tokens are emitted.