umputun / rlb-stats / 38504604500

84%

push

github

Add nginx tail mode: ingest access log directly (#67)

* Add nginx tail mode: ingest access log directly

Allow rlb-stats to tail an nginx access log instead of only accepting
POST /api/insert from rlb, which lets it stay useful after rlb is retired.

Required nginx log_format and the three new CF fields documented in README.
Default filters are podcast-focused (GET, 200/206, *.mp3). Storage adds Bytes
and Countries on store.Info — backward-compatible with existing boltdb files
since JSON unmarshal of old candles yields zero / nil for the new fields.

Dashboard gets Total bandwidth on summary cards, Bandwidth column in top
files, per-vhost Bandwidth, and a new Top countries panel.

Drops the rlb service block from docker-compose and the now-unused
rlb.config.yml. /api/insert and /api/candle are untouched.

Behaviour change worth stating in the PR body: nginx-tail counts
(file, ip) once per minute, whereas rlb counted one redirect once;
historical numbers are not comparable.

Plan: docs/plans/20261011-nginx-tail-mode.md

* Address revmux review findings

Critical:
- tail.go: maxLineBytes check used cap() which stays elevated after
  st.partial = st.partial[:0], permanently poisoning the buffer after
  one oversized line. Switch to len() plus nil-slice reset and a
  skipUntilNewline state that drops the remainder of the overflowing
  line until its newline, then resumes. Regression test with a 2 MiB
  line sandwiched between normal lines; mutation-verified.

Major:
- README and docker-compose: mount the nginx log directory, not the
  file. A single-file bind mount pins to the host inode at container
  start; after logrotate renames access.log, rlb-stats keeps reading
  the old (rotated) inode and silently stops counting.
- Top Files: drop the Bandwidth column. It was computed as a per-file
  proportional share of per-candle total bytes, which gives wrong
  numbers when a candle mixes small ranges with full downloads. Needs
  a real per-file bytes map on... (continued)

345 of 447 new or added lines in 10 files covered. (77.18%)

983 of 1169 relevant lines covered (84.09%)

79.5 hits per line