inikulin / parse5 / 39873271538

96%

push

github

feat: allow flushing buffered character tokens (#1854)

* feat: allow flushing buffered character tokens

Add flushCharacters() to ParserStream and Tokenizer instances. Character-token batching can leave trailing text out of the tree until more input arrives. Consumers implementing document.write(), such as jsdom, need to make that text synchronously observable without ending the input stream.

Flushing delivers buffered character tokens but leaves incomplete tags and character references pending. It does not resume suspended scripts, drain queued stream writes, or bypass the tree builder's pending table-character processing. Calls from parser callbacks are rejected to prevent reentrant token delivery.

Derive source boundaries from tokenizer state at flush time so ordinary parsing needs no per-character boundary tracking. Keep attribute locations independent of character locations, since flushing can occur while an attribute is incomplete. Correct source positions for lookahead emissions and surrogate pairs so offsets and columns agree.

Flushing splits character-token runs, so text-node source ranges can differ from parsing without flushing: subsequently ignored input, such as CDATA delimiters or the leading newline in a <pre> element, does not extend an already flushed run.

Co-Authored-By: codex <codex@openai.com>

* fix: preserve text offsets after flushing split CRLF

Exclude the ignored LF from the next character token's source range when its preceding CR was flushed. Text following the discarded leading newline in pre, textarea, and listing elements then starts at the actual character offset.

Co-Authored-By: codex <codex@openai.com>

* perf: avoid splitting flushed text into lines

Compute flushed character-token locations with indexOf() scans instead of allocating an array of line substrings. Keep the scans bounded to emitted text so large unfinished tags or attributes do not increase flush cost.

Co-Authored-By: codex <codex@openai.com>

--... (continued)

540 of 543 branches covered (99.45%)

78 of 78 new or added lines in 3 files covered. (100.0%)

3870 of 4049 relevant lines covered (95.58%)

44971.68 hits per line