grobidOrg / grobid / 29900893897

44%

master: 44%

Pull #1517

github

fix: split paragraphs after a callout on layout, not the sequence label

Re-addresses the real #1482 case (a paragraph that genuinely ends at a
trailing callout) on top of the clean 0.9.0 base restored by the revert,
without the isBeginning() predicate that made the previous attempt split
at nearly every callout.

The decision comes from the layout of the callout and the resumed text:

- Same line as the callout -> continue. The common case (a callout
  inside a paragraph), and it needs no further evidence.
- A line break alone proves nothing, since a paragraph interrupted near
  the right margin also wraps, so a break must also cross a block
  boundary - blocks being the unit pdfalto groups paragraphs into.
- A block boundary also falls inside a paragraph, so the resumed text
  must additionally read like the start of one (upper case or digit).
  This stops continuations, and the punctuation closing the interrupted
  sentence, from being split off.
- Text resuming higher up the page has wrapped to the top of the next
  column, and a page change makes the coordinates incomparable. Neither
  is evidence of a paragraph break.

Splitting requires positive evidence; anything unknown continues the
paragraph, so the worst case degrades to 0.9.0 rather than a new failure
mode. Only two overloads: the layout-aware one, and the 0.9.0-compatible
2-arg used by Table and external callers.

Two details, both found only by instrumenting. LayoutToken
.isNewLineAfter() cannot detect the line break: it is only populated by
enrichWithNewLineInfo(), which the fulltext path never calls, so it is
always false here - the break is read from the Y coordinates. And the
token before a cluster boundary is very often a synthesised space with
no coordinates (x = y = -1), so the comparison skips back to the last
positioned token or every callout looks like a line break.

Full corpus, 11,945 documents vs 0.9.0: <p> 1,170,384 -> 1,177,912
(+0.64%, master +85.5%); paragraphs opening ... (continued)

10152 of 25716 branches covered (39.48%)

Branch coverage included in aggregate %.

21245 of 46387 relevant lines covered (45.8%)

1.86 hits per line