aws / karpenter-core / 5600975159

80%

push

github

fix: prevent consolidation from pausing for 5m (#415)

There was a race with deprovisioning with steps 2 & 3 where we could:
1) See the cluster as unconsolidated
2) Run the deprovisioners unsuccessfully
3) Something happens that makes consolidation possible (e.g. pod
   deletion)
4) We mark the cluster as unconsolidatable
5) On the next round, do nothing since cluster is unconsolidatable

The end result is that we mark the cluster as unconsolidatable which
prevents us from trying again to save CPU time. Thankfully this was
built with a 5 minute timeout to handle recovering from ICE situations
automatically which limited the severity of the problem.

This change modifies the code to use a monotonically increasing
timestamp.  If the cluster state changes (or nothing changes for five
minutes), it increases.

The race above is now prevented:

1) See the cluster state as X
2) Run the deprovisioners unsuccessfully
3) Something happens that makes consolidation possible (e.g. pod
   deletion), this increases cluster state to X+1
4) We mark the cluster as unconsolidatable as of X
5) On the next round, see the cluster state as X+1 and retry since
   we last tried at state X

44 of 44 new or added lines in 6 files covered. (100.0%)

7059 of 8668 relevant lines covered (81.44%)

212704.98 hits per line