NVIDIA / nodewright / 38661793541

84%

Build Type

push

github

Committed by web-flow

Commit Message

perf(operator): cut reconcile-path allocations on large clusters (#797)

On a large cluster the leader runs a GC every few seconds and holds about
twice its live heap, while the follower, which only holds caches, is flat.
Profiles show the allocation is on the reconcile path, not cache upkeep:

- globalDelayHandler.relevant listed every NodeWright from the cache on
  every Node event, and a cached List deep-copies each object. NodeWright
  status carries per-node state, so each heartbeat copied megabytes to run
  a label selector, and initial sync replayed that for every Node. The
  list is now read without the copy inside skyhookSelectors, which lets
  only the parsed selectors out, so no cache pointer outlives that call.
  An unparseable selector now matches every node instead of dereferencing
  a nil selector.
- BuildState deep-copied a Node once per NodeWright that selected it,
  although ObjectTracker.Track keeps the first copy and drops the rest. It
  also parsed the selector and built a wrapper for every node before
  checking the match. Parse once per NodeWright, check first, copy once.
- Cached objects kept metadata.managedFields, which nothing reads and
  which is a large share of Pods and Nodes. Strip it on the way into the
  cache; a nil managedFields on update leaves the server's intact.

Two behaviour changes in BuildState, both covered by new specs:
- A NodeWright whose package graph cannot be built (the only error
  NewSkyhookNode can return) and which selects no node no longer fails the
  pass for every NodeWright. It still fails when it selects a node.
- An invalid nodeSelector now fails the pass even with no nodes listed,
  where before the parse only ran inside the node loop. Admission already
  rejects both cases.

Signed-off-by: Taylor Daugherty <tdaugherty@nvidia.com>
Co-authored-by: Brian Lockwood <lockwobr@gmail.com>
Co-authored-by: ayuskauskas <ayuskauskas@nvidia.com>

Coverage Stats

38 of 38 new or added lines in 3 files covered. (100.0%)

3 existing lines in 1 file now uncovered.

10079 of 12047 relevant lines covered (83.66%)

7.63 hits per line