Streaming

A LexParseStream is a push-based parse session: feed it chunks as they arrive (sockets, file reads, editors), and consume one fresh LexTree per chunk through async iteration. Streams never throw and their iterators never reject.

Package: @lanexio/parser (re-exported from @lanexio/parser-core).

Create a stream

Through a parser handle (recommended — binds the grammar for you):

import { createParser } from '@lanexio/parser';
import { htmlGrammar } from '@lanexio/parser-grammar-html';

const parser = await createParser(htmlGrammar);
const stream = parser.createStream();

Or standalone, binding any parse function:

import { createParseStream } from '@lanexio/parser';
import { parseMarkdown } from '@lanexio/parser-grammar-markdown';

const stream = createParseStream(
  '@lanexio/parser-grammar-markdown',
  (bytes) => parseMarkdown(bytes, { gfm: true }),
);

Push, end, iterate

stream.push('# Title\n');          // strings are UTF-8 encoded for you
stream.push(moreBytes);            // Uint8Array chunks work too
stream.end();

for await (const tree of stream) {
  // One tree per push. Each tree covers ALL bytes accumulated up to
  // and including its corresponding push — not just the new chunk.
  console.log(tree.nodeCount, tree.source.byteLength);
}

The contract

Member	Behavior
push(chunk)	Appends the chunk, reparses the full accumulation, and enqueues one tree. Never throws. Calls after end() are silently ignored.
end()	Signals end-of-input. Idempotent. The iterator completes after draining already-queued trees. Never throws.
source()	Returns a copy of the accumulated bytes — useful for diagnostics; prefer tree.source on the trees themselves.
[Symbol.asyncIterator]()	Yields one tree per push, then completes after end(). Never rejects.
grammar	The stable npm package name the stream was bound to.

Three properties worth internalizing:

1. Each tree is a full document parse. The current implementation reparses the entire accumulation on every push, and the API contract guarantees the result is always byte-identical to a full parse of the accumulated bytes — future versions may reuse unaffected subtrees internally without changing what you observe.
2. The stream owns its copies. Uint8Array chunks are copied on entry, so you can safely reuse a scratch buffer (the common socket-read pattern) and mutate it after push without corrupting earlier or later trees.
3. Never-throw is layered. Even if the bound parse function violated its own contract and threw, the stream catches it and enqueues a single-node error tree for that push instead of dropping it or rejecting the iterator.

Backpressure and consumption patterns

The queue between producer and consumer is unbounded; if you push thousands of chunks before consuming, all intermediate trees are retained until iterated. For high-volume feeds, interleave production and consumption:

const stream = parser.createStream();

async function consume() {
  for await (const tree of stream) {
    render(tree);                 // use only the latest if you prefer:
  }                               // intermediate trees can simply be skipped
}
const consumer = consume();

for await (const chunk of socket) {
  stream.push(chunk);
}
stream.end();
await consumer;

One stream, one consumer

Concurrent for await loops on the same stream split the trees between them — each tree is delivered to exactly one consumer. If multiple parts of your program need every tree, fan the trees out yourself from a single consumer loop.

Cost model

Because every push reparses the accumulation, pushing N chunks costs O(total² / chunk) parse work in aggregate. For chunked network loads this is rarely an issue (parses are fast and chunks are large); for per-keystroke editor updates, prefer Incremental Parsing with reparse over a stream.