decentraland / deployments-to-sqs / 31910803467

91%

main: 91%

Build Type

Pull #536

github

Committed by LautaroPetaccio

Commit Message

perf: split the deployer and synchronizer queues and widen concurrency

Measured on two identical 40s runs against peer.decentraland.zone:

  concurrency  5 (old):  333 entities stored,  7.9/s
  concurrency 15 (new): 1175 entities stored, 27.6/s   ~3.5x

Zero download failures, queue rejections or non-SNS errors at either level.

`createJobQueue` was a single instance shared by the deployer and the
synchronizer, which coupled three unrelated things:

1. `deployer.onIdle()` is awaited by snapshots-fetcher at the end of EVERY
   poll of EVERY server. On a shared queue, server A's poll could not close
   until entities scheduled by servers B..F — and the synchronizer's own
   /snapshots fetches — had all drained. Six independent streams serialized
   behind the slowest download.
2. /snapshots fetches were scheduled on that same queue, behind a backlog of
   up to 1000 entities. p-queue only applies `timeout` once a job STARTS, so
   they stalled silently rather than timing out.
3. Entity throughput was capped by a concurrency sized for the control plane.

Split, `onIdle()` means "this deployer's work drained" — the contract
snapshots-fetcher actually needs — and each side is sized for its own job.

The service could ingest ~1000 entities/s and drained ~8/s, so the queue sat
pinned at its backpressure limit and the streams ran at the drain rate. The
scheduling histogram shows it directly: at concurrency 5 the rate collapses
to 4-12/s after the first second; at 15 it sustains 16-96/s.

15 is deliberately conservative rather than the largest value that helped
locally. Local numbers use folder storage and profile-heavy traffic, so they
do not predict production, where S3 latency and scene content fan-out both
differ. Tune from the gauges instead.

DOWNLOAD_QUEUE_CONCURRENCY (15), DOWNLOAD_QUEUE_TIMEOUT_MS (100000),
DOWNLOAD_QUEUE_MAX_PENDING (1000), SYNCHRONIZER_QUEUE_CONCURRENCY (10),
SYNCHRONIZER_QUEUE_TIMEOUT_MS (100000), CONTENT_FILES_CONCURRENCY (4).

U... (continued)

Pull Request Pull Request #536: perf: split the deployer and synchronizer queues and widen concurrency

Coverage Stats

58 of 73 branches covered (79.45%)

Branch coverage included in aggregate %.

10 of 11 new or added lines in 4 files covered. (90.91%)

10 existing lines in 2 files now uncovered.

278 of 296 relevant lines covered (93.92%)

8.49 hits per line