web-csv-toolbox
    Preparing search index...

    Class CSVLexerTransformer<Delimiter, Quotation>

    A transform stream that converts a stream of strings into a stream of tokens.

    CSV-specific options (delimiter, quotation, etc.)

    Strategy for the writable side (default: { highWaterMark: 65536, size: chunk => chunk.length, checkInterval: 100 })

    Strategy for the readable side (default: { highWaterMark: 1024, size: () => 1, checkInterval: 100 })

    Follows the Web Streams API pattern where queuing strategies are passed as constructor arguments, similar to the standard TransformStream.

    Default Queuing Strategy:

    • Writable side: Counts by string length (characters). Default highWaterMark is 65536 characters (≈64KB).
    • Readable side: Counts each token as 1. Default highWaterMark is 1024 tokens.

    Backpressure Handling: The transformer monitors controller.desiredSize and yields to the event loop when backpressure is detected (desiredSize ≤ 0). This prevents blocking the main thread during heavy processing and allows the downstream consumer to catch up.

    These defaults are starting points based on data flow characteristics, not empirical benchmarks. Optimal values depend on your runtime environment, data size, and performance requirements.

    new ReadableStream({
      start(controller) {
        controller.enqueue("name,age\r\n");
        controller.enqueue("Alice,20\r\n");
        controller.close();
      }
    })
      .pipeThrough(new CSVLexerTransformer())
      .pipeTo(new WritableStream({ write(token) {
        console.log(token);
      }}));
    // { type: Field, value: "name", location: {...} }
    // { type: FieldDelimiter, value: ",", location: {...} }
    // { type: Field, value: "age", location: {...} }
    // { type: RecordDelimiter, value: "\r\n", location: {...} }
    // { type: Field, value: "Alice", location: {...} }
    // { type: FieldDelimiter, value: ",", location: {...} }
    // { type: Field, value: "20" }
    // { type: RecordDelimiter, value: "\r\n", location: {...} }

    const transformer = new CSVLexerTransformer(
      { delimiter: ',' },
      {
        highWaterMark: 131072,         // 128KB of characters
        size: (chunk) => chunk.length, // Count by character length
        checkInterval: 200             // Check backpressure every 200 tokens
      },
      {
        highWaterMark: 2048,           // 2048 tokens
        size: () => 1,                 // Each token counts as 1
        checkInterval: 50              // Check backpressure every 50 tokens
      }
    );

    await fetch('large-file.csv')
      .then(res => res.body)
      .pipeThrough(new TextDecoderStream())
      .pipeThrough(transformer)
      .pipeTo(yourProcessor);

    Type Parameters

    • Delimiter extends string = DEFAULT_DELIMITER
    • Quotation extends string = DEFAULT_QUOTATION

    Hierarchy

    Index

    Constructors

    constructor

    Properties

    lexer readable writable

    Methods

    yieldToEventLoop

    Constructors

    Properties

    lexer: CSVLexer<Delimiter, Quotation>
    readable: ReadableStream<Token>

    The readable read-only property of the TransformStream interface returns the ReadableStream instance controlled by this TransformStream.

    MDN Reference

    writable: WritableStream<string>

    The writable read-only property of the TransformStream interface returns the WritableStream instance controlled by this TransformStream.

    MDN Reference

    Methods

    Settings

    Member Visibility

    On This Page

    Constructors
    Properties
    Methods