cilium / cilium / 41654

39%

push

travis-ci-com

Add concurrency limiting for DNS message processing

This commit does the following:

* Adds a configuration option for controlling the concurrency of the DNS
  proxy
* Adds a configuration option for the semaphore (from above) timeout
* Exposes an additional metric for the time taken to perform the policy
  check on a DNS request within the DNS proxy

The concurrency limitation is done by introducing a semaphore to DNS
proxy. By default, no such limit is imposed.

Users are advised to take into account the number of DNS requests[1] and
how many CPUs on each node in their cluster in order to come up with an
appropriate concurrency limit.

In addition, we expose the semaphore grace period as a configurable
option. Assuming the "right" for this timeout is a tradeoff that we
shouldn't really assume for the user.

The semaphore grace period is to prevent the situation where Cilium
deadlocks or consistently high rate of DNS traffic causing Cilium to be
unable to keep up.

See https://github.com/cilium/cilium/pull/19543#discussion_r862136748 by
<joe@cilium.io>.

The user can take into account the rate that they expect DNS requests to
be following into Cilium and how many of those requests should be
processed without retrying. If retrying isn't an issue then keeping the
grace period at 0 (default) will immediately free the goroutine handling
the DNS request if the semaphore acquire fails. Conversely, if a backlog
of "unproductive" goroutines is acceptable (and DNS request retries are
not), then setting the grace period is advisable. This gives the
goroutines some time to acquire the semaphore. Goroutines could pile up
if the grace period is too high and there's a consistently high rate of
DNS requests.

It's worth noting that blindly increasing the concurrency limit will not
linearly improve performance. It might actually degrade instead due to
internal downstream lock contention (as seen by the recent commits to
move Endpoint-related functions to use read-l... (continued)

84 of 84 new or added lines in 4 files covered. (100.0%)

47462 of 140020 relevant lines covered (33.9%)

1952.54 hits per line