berserkhmdvhb / agentic-scraper / 17369246166

87%

main: 88%

Build Type

push

github

Committed by berserkhmdvhb

Commit Message

refactor(rule_based): improve robustness of rule-based agent and add stricter validation

- Enhanced `guess_price`:
  - Added fallback regexes for formats like `19,99 €` and `€ 19.99`.
  - Implemented candidate scoring to prefer decimal-containing matches.
  - Added synthesis logic for separated integer/fractional groups.
  - Appended fractional part from trailing text when missing.
  - Cleaned extracted strings to normalize separators (comma vs dot).

- Improved `guess_description`:
  - Added `_strip_trailing_price_lines` helper to drop price-only lines from candidate paragraphs.
  - Ensured returned description excludes misleading price artifacts.

- Updated `extract_structured_data`:
  - Now returns `None` early if no informative fields are extracted (title, description, price).
  - Added warning log `MSG_WARN_RULE_BASED_NO_FIELDS` when this occurs.
  - Preserved screenshot capture and structured data logging.

- Introduced new message constants:
  - `MSG_WARN_RULE_BASED_NO_FIELDS = "Rule-based extraction produced no informative fields for {url}"`

- Adjusted tests:
  - Updated validation-failure test to expect early `None` instead of relying solely on schema exceptions.
  - Ensured log capture verifies warning emission.

Run Details

57 of 67 new or added lines in 2 files covered. (85.07%)

16 existing lines in 1 file now uncovered.

2879 of 4618 relevant lines covered (62.34%)

4.99 hits per line