decentraland / profile-images / 25497297206

88%

main: 89%

Build Type

Pull #151

github

Committed by kuruk-mm

Commit Message

feat: add disk-usage telemetry around godot subprocess runs

Adds a disk-monitor component that snapshots disk usage immediately
before and after every godot subprocess execution to diagnose the
prod ENOSPC crashes (20 GiB ephemeral storage filled in ~2h with
only 97 godot invocations) that are not reproducible locally.

The hook runs inside godot.ts around runGodot, so each pre/post line
aligns with the existing godot-snapshot "about to exec ... temp-avatars-N.json"
log. Logs are emitted in the same format as SERVER_LOGS.log.

Per run, always emits:
- pre  run=N batch=K disk=X% used=A/B /tmp=.. /app=.. /root=.. /var/log=..
        temp-avatars=count(bytes) sentry-runs=K output-files=K
- post run=N disk=X% Δ=±M (per-dir delta) batch_size=K duration=Tms
        cumulative_since_start=±M

When a single run delta exceeds DISK_SPIKE_MB_THRESHOLD (default 50 MB)
or disk usage crosses DISK_USAGE_CRITICAL_PCT, also dumps:
- du -shx top entries for /tmp /app /root
- find -newer of files >1M created since previous run
- count of "deleted but still open" file descriptors via /proc/*/fd

Every DISK_MONITOR_FULL_AUDIT_EVERY runs (default 10) emits a full
audit: du -shx /*, df -h, top, deleted-fd count.

Once usage crosses DISK_USAGE_WARN_PCT/CRITICAL_PCT all per-run lines
escalate to WARN/ERROR so they survive default log filters.

All toggleable via env, fully no-op when DISK_MONITOR_ENABLED=false.

Refactored shared probes (df, top, du, find) into src/utils/system-probes.ts
so godot.ts and disk-monitor.ts share one implementation.

Defaults in .env.default:
  DISK_MONITOR_ENABLED=true
  DISK_SPIKE_MB_THRESHOLD=50
  DISK_USAGE_WARN_PCT=50
  DISK_USAGE_CRITICAL_PCT=75
  DISK_MONITOR_FULL_AUDIT_EVERY=10

Pull Request Pull Request #151: feat: disk-usage telemetry around godot subprocess runs

Coverage Stats

179 of 280 branches covered (63.93%)

Branch coverage included in aggregate %.

55 of 151 new or added lines in 4 files covered. (36.42%)

9 existing lines in 1 file now uncovered.

529 of 661 relevant lines covered (80.03%)

16.75 hits per line