ukeeper / ukeeper-readability / 24316097933

78%

Build Type

push

github

Committed by web-flow

Commit Message

Modularise URL retrieval with Cloudflare Browser Rendering support (#73)

* feat: create Retriever interface and HTTPRetriever implementation

Extract URL fetching abstraction from the inline HTTP logic in extractWithRules.
Defines Retriever interface, RetrieveResult struct, and HTTPRetriever with
Safari user-agent, redirect following, and timeout support. Includes moq
generate directive and comprehensive tests.

* feat: implement CloudflareRetriever for Browser Rendering API

* feat: wire Retriever interface into UReadability extraction pipeline

* feat: add CLI flags and wire Cloudflare retriever in main.go

* feat: generate Retriever mock, run gofmt and linter

Generate moq mock for Retriever interface as a test-only file
(retriever_mock_test.go) instead of mocks/ subpackage to avoid
import cycle (mocks/retriever.go would import extractor, cycling
with readability_test.go). Run gofmt on all modified files, zero
lint issues.

* feat: verify acceptance criteria for Retriever interface

* feat: update documentation for Retriever interface and CLI flags

* fix: address code review findings

- fix err shadowing in deferred Body.Close() in both retrievers (use closeErr)
- handle Cloudflare API success=false response explicitly instead of treating JSON error as HTML
- truncate CF API error body to 512 bytes in error messages
- add comment documenting CF retriever URL limitation (no final URL after JS redirects)
- fix pre-existing %b format verb in text.go logging (should be %v)
- replace network-dependent TestCloudflareRetriever_DefaultBaseURL with local httptest
- add TestCloudflareRetriever_SuccessFalse for the new success=false handling
- add TestExtractWithCustomRetriever integration test using RetrieverMock
- remove duplicate plan file from docs/plans/ (already in completed/)
- update README.md with new CF CLI flags and feature description
- update CLAUDE.md CI bullet to reflect split docker.yml workflow

* fix: address code review findings

* fix: a... (continued)

Coverage Stats

207 of 250 new or added lines in 5 files covered. (82.8%)

1 existing line in 1 file now uncovered.

767 of 984 relevant lines covered (77.95%)

9.39 hits per line