Type Alias Token<TrackLocation>

Token: TrackLocation extends true ? TokenWithLocation : TokenNoLocation

Token type.

This is the optimized token format that reduces token count by 50%. Instead of emitting separate Field, FieldDelimiter, and RecordDelimiter tokens, only unified field tokens are emitted with the delimiter property indicating what delimiter follows.

Type Parameters

TrackLocation extends boolean = false
Whether to include location information (default: false)

Example: Without location tracking (default, fastest)

// CSV: "a,b,c\n"
// Tokens:
// { value: "a", delimiter: Delimiter.Field, delimiterLength: 1 }
// { value: "b", delimiter: Delimiter.Field, delimiterLength: 1 }
// { value: "c", delimiter: Delimiter.Record, delimiterLength: 1 }

Example: With CRLF

// CSV: "a,b\r\n"
// Tokens:
// { value: "a", delimiter: Delimiter.Field, delimiterLength: 1 }
// { value: "b", delimiter: Delimiter.Record, delimiterLength: 2 }  // CRLF = 2

Type Alias Token<TrackLocation>

Type Parameters

Example: Without location tracking (default, fastest)

Example: With CRLF

Settings