AfterShip / clickhouse-sql-parser / 28727078583

53%

master: 54%

Build Type

Pull #280

github

Committed by git-hulk

Commit Message

Fix number lexing, unterminated comments and non-ASCII bytes

Four lexer-level fixes, each with regression tests:

- consumeNumber checked peekN(0)/peekN(1) for the 0x prefix after
  consuming a sign, so `WHERE x = -0xFF` failed to lex. The prefix
  check now starts at the position after the sign.
- consumeNumber accepted unlimited dots and stray exponents: `1.2.3`,
  `1..2`, `1e2.3` and `0x1.8` all lexed as single float tokens that
  ClickHouse rejects. A second dot, a dot after the exponent, or a dot
  in a hex literal is now an invalid number. The unused p/P exponent
  (only meaningful for hex floats, which are unsupported) is dropped,
  so `1p5` no longer lexes as an integer named 1p5.
- Scientific notation without a decimal point (`1e5`) lexed as
  TokenKindInt; it is now TokenKindFloat.
- An unterminated /* block comment was silently swallowed at EOF and
  the statement parsed successfully; it is now an error, matching
  ClickHouse. consumeSingleLineComment also no longer advances the
  offset one byte past EOF when a -- comment has no trailing newline.
- A bare non-ASCII byte produced a garbage one-byte token that split
  the UTF-8 sequence (errors printed "unexpected token kind: <?>" with
  a broken rune). The lexer now reports the whole rune:
  unexpected character '中'.

Note: mid-statement lexer errors are still discarded by the parser's
`_ = p.lexer.consumeToken()` call sites; surfacing them is a separate
change. The new tests therefore assert at the lexer level.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Pull Request Pull Request #280: Fix number lexing, unterminated block comments and non-ASCII bytes

Coverage Stats

35 of 36 new or added lines in 1 file covered. (97.22%)

8 existing lines in 1 file now uncovered.

8569 of 16183 relevant lines covered (52.95%)

3027.45 hits per line