umputun / rlb-stats
84%

DEFAULT BRANCH: master
Repo Added
Files 12
Badge
Embed ▾
README BADGES
x

If you need to use a raster PNG badge, change the '.svg' to '.png' in the link

Markdown

Textile

RDoc

HTML

Rst

LAST BUILD ON BRANCH master
branch: SELECT
CHANGE BRANCH
x

coverage: 84.089% (-4.4%) from 88.518%
38504604500

push

github

web-flow 
Add nginx tail mode: ingest access log directly (#67)

* Add nginx tail mode: ingest access log directly

Allow rlb-stats to tail an nginx access log instead of only accepting
POST /api/insert from rlb, which lets it stay useful after rlb is retired.

Required nginx log_format and the three new CF fields documented in README.
Default filters are podcast-focused (GET, 200/206, *.mp3). Storage adds Bytes
and Countries on store.Info — backward-compatible with existing boltdb files
since JSON unmarshal of old candles yields zero / nil for the new fields.

Dashboard gets Total bandwidth on summary cards, Bandwidth column in top
files, per-vhost Bandwidth, and a new Top countries panel.

Drops the rlb service block from docker-compose and the now-unused
rlb.config.yml. /api/insert and /api/candle are untouched.

Behaviour change worth stating in the PR body: nginx-tail counts
(file, ip) once per minute, whereas rlb counted one redirect once;
historical numbers are not comparable.

Plan: docs/plans/20261011-nginx-tail-mode.md

* Address revmux review findings

Critical:
- tail.go: maxLineBytes check used cap() which stays elevated after
  st.partial = st.partial[:0], permanently poisoning the buffer after
  one oversized line. Switch to len() plus nil-slice reset and a
  skipUntilNewline state that drops the remainder of the overflowing
  line until its newline, then resumes. Regression test with a 2 MiB
  line sandwiched between normal lines; mutation-verified.

Major:
- README and docker-compose: mount the nginx log directory, not the
  file. A single-file bind mount pins to the host inode at container
  start; after logrotate renames access.log, rlb-stats keeps reading
  the old (rotated) inode and silently stops counting.
- Top Files: drop the Bandwidth column. It was computed as a per-file
  proportional share of per-candle total bytes, which gives wrong
  numbers when a candle mixes small ranges with full downloads. Needs
  a real per-file bytes map on... (continued)

345 of 447 new or added lines in 10 files covered. (77.18%)

983 of 1169 relevant lines covered (84.09%)

79.5 hits per line

Relevant lines Covered
1169 RELEVANT LINES 983 COVERED LINES
79.5 HITS PER LINE
Source Files on master

Recent builds

Builds Branch Commit Type Ran Committer Via Coverage
38504604500 master Add nginx tail mode: ingest access log directly (#67) * Add nginx tail mode: ingest access log directly Allow rlb-stats to tail an nginx access log instead of only accepting POST /api/insert from rlb, which lets it stay useful after rlb is retir... push web-flow github
84.09
38504581999 master Keep one open window in aggregateCandles instead of a bucket map Candles arrive time-sorted, so the window index never decreases and every candle for a window comes in one run. One window is therefore open at a time, closing when the index moves ... push umputun github
88.52
38139198471 nginx-tail-mode Address revmux r2 findings - Test for the drain-before-reopen fix now actually covers the race window. The r1 test passed with the drain removed because the top-of-loop drain always runs before stat under Run's ticker loop. The new test calls... Pull #67 paskal github
84.09
38137752856 fix/aggregate-running-bucket Keep one open window in aggregateCandles instead of a bucket map Candles arrive time-sorted, so the window index never decreases and every candle for a window comes in one run. One window is therefore open at a time, closing when the index moves ... Pull #66 paskal github
88.52
38137756216 fix/aggregate-running-bucket Keep one open window in aggregateCandles instead of a bucket map Candles arrive time-sorted, so the window index never decreases and every candle for a window comes in one run. One window is therefore open at a time, closing when the index moves ... Pull #66 paskal github
88.52
34170407953 master Reject non-positive aggregate on /api/candle (#64) Previously, aggregate=-5m or aggregate=0s parsed successfully and was silently raised to one minute inside aggregateCandles, so the request answered 200 with an interval the caller never asked for. push web-flow github
88.52
34156048009 master Aggregate candles in a single pass (#65) Previously, aggregateCandles rebuilt each output window by rescanning the whole candle slice, O(windows x candles): a 30-day dashboard with about 43k minute candles and 100 windows walked the slice once pe... push web-flow github
88.44
33739402016 fix/aggregate-single-pass Aggregate candles in a single pass Previously, aggregateCandles rebuilt each output window by rescanning the whole candle slice, O(windows x candles): a 30-day dashboard with about 43k minute candles and 100 windows walked the slice once per wind... Pull #65 paskal github
88.44
33739405608 fix/aggregate-single-pass Aggregate candles in a single pass Previously, aggregateCandles rebuilt each output window by rescanning the whole candle slice, O(windows x candles): a 30-day dashboard with about 43k minute candles and 100 windows walked the slice once per wind... Pull #65 paskal github
88.44
33737335096 fix/aggregate-single-pass Aggregate candles in a single pass Previously, aggregateCandles rebuilt each output window by rescanning the whole candle slice, O(windows x candles): a 30-day dashboard with about 43k minute candles and 100 windows walked the slice once per wind... Pull #65 paskal github
88.43
See All Builds (215)