decentraland / social-service-ea / 24508747248 / 1

90%

main: 90%

push

github

fix: prevent OOM crashes from subscriber leak and duplicate subscriptions (#407)

* fix: prevent OOM crashes from subscriber leak and duplicate subscriptions

Three critical fixes for the heap exhaustion crash (510 MB after ~62h):

1. Fix cleanup bug in ws-handler: the three cleanup steps (transport.close,
   detachUser, emitter.clear) were in a single try/catch. If transport.close()
   threw (uWS throws on closed sockets), detachUser and emitter.clear were
   skipped, permanently leaking the subscriber's emitter and generators.
   Each step is now an independent function with its own error handling.

2. Guard against duplicate subscriptions: clients can call the same stream
   RPC multiple times on one connection. Each call created another async
   generator with a 1000-item value queue, doubling memory per connection.
   Crash logs confirmed this for communityMemberConnectivityUpdate.
   Added per-(address, event) deduplication in handleSubscriptionUpdates.

3. Set explicit V8 heap limit (768 MB for 1024 MB prod containers) via
   NODE_OPTIONS env var in Dockerfile, overridable at deploy time.

* fix: add cork wrapping, subscriber metrics, reconciliation, and sScan

Hardening changes to complement the critical OOM fixes:

1. Wrap all HTTP responses in res.cork() to prevent uWS warnings about
   writes outside corked callbacks. Under GC pressure these interleaved
   writes can hold object references longer than necessary.

2. Add subscriber monitoring metrics (gauges): subscribers_local_count,
   subscribers_generators_count, subscribers_redis_set_size, and a
   subscribers_stale_cleaned counter. Reported every 30s via setInterval.

3. Add periodic stale subscriber reconciliation (every 5 min, configurable
   via SUBSCRIBER_RECONCILIATION_INTERVAL_MS). Cross-references local
   subscribers against active WS connections from wsPool and removes any
   orphaned subscribers that the cleanup path missed.

4. Replace redis.sMembers() with redis.client.sSca... (continued)

2371 of 2772 branches covered (85.53%)

Branch coverage included in aggregate %.

10944 of 11942 relevant lines covered (91.64%)

35.71 hits per line