Interface TokenWithLocation

Token with location tracking.

This is the optimized token format where only field tokens are emitted, with location information included.

interface TokenWithLocation {
    delimiter: Delimiter;
    delimiterLength: number;
    location: TokenLocation;
    value: string;
}

Properties

delimiter: Delimiter

What delimiter follows this field

delimiterLength: number

Length of the delimiter in characters (1 for comma/LF/CR, 2 for CRLF, 0 for EOF)

Location information for error reporting

value: string

The field value