k8snetworkplumbingwg / multus-cni

64%

master: 50%

LAST BUILD ON BRANCH thickplugin_graceful_term2

branch: thickplugin_graceful_term2

CHANGE BRANCH

x

Reset

Sync Branches

• thickplugin_graceful_term2
• AllowTryDeleteOnErrOptions
• CORENET-6353
• Sagit-chu-patch-1
• add-netdef-informer
• add-networks-annotation-controller
• add-ready-check
• add-sleep-thin
• add-test-to-capture-flaky-behavior
• add-timeout
• add-trivy
• all_capabilities
• allow_undersocre_in_ifname
• auto_confDir
• backport-releasev3-defroute-noifplugin
• baseimage
• bugfix/ensure-resource-is-passed
• bump-all/adil
• bump-net-attach-def-client-lib-1.7.6
• bump-netattachdef-client
• bump-ver
• bumps
• cherry-pick/762
• chore/update-dra-e2e
• chroot
• chroot_test_findpath
• clean-ioutil
• cleanup
• cleanup-multus-conf
• client-lib-multiple-if-cni-result
• cni-check-log-message
• cni-daemon-config-does-not-need-shim-socket-path
• cni110
• code-clean
• codeclean-if
• codeql
• configuration-options-refactor
• crio-mount
• cross-compilation
• custom_log
• cve-2021-38561
• deads_upstreampick_fallback
• debounce-dial-socket
• default-route-fix-cache-name
• default-route-pointer
• delegateAdd-bug
• dependabot/go_modules/golang.org/x/net-0.17.0
• dependabot/go_modules/golang.org/x/net-0.23.0
• dependabot/go_modules/golang.org/x/net-0.33.0
• dependabot/go_modules/golang.org/x/net-0.36.0
• dependabot/go_modules/golang.org/x/net-0.7.0
• dependabot/go_modules/google.golang.org/grpc-1.53.0
• dependabot/go_modules/google.golang.org/grpc-1.56.3
• dependabot/go_modules/google.golang.org/protobuf-1.33.0
• detect-python3
• dev/action-update
• dev/add-arm32
• dev/add-cluster-net-name
• dev/add-ut
• dev/chroot
• dev/cleanup-dev-info
• dev/cni100
• dev/delegate-api
• dev/distroless-thin
• dev/doc-modified
• dev/fedora-image
• dev/fix-thick-e2e
• dev/go118
• dev/metrics
• dev/no-cniversion
• dev/nosetenv
• dev/rc-api
• dev/refine-build
• dev/refine-config
• dev/refine-e2e
• dev/refine-k8s-lib
• dev/refine-multus-ut
• dev/refine_multus_ut
• dev/remove-old-conf
• dev/remove-restart-crio
• dev/simple-kubeletpkg
• dev/simplify-dockerfile
• dev/update-golang
• dev/update-golang119
• dev/update-layout
• dev/update-lib
• dev/use-ghcr
• dev/use-non-default-libcni-cache
• disable-dra-e2e-temporarily
• doc-clusternetwork
• docker-updates
• docs-thick-plugin-config-update
• dont-wait-too-long-for-apiserver
• downgrade-k8s-vendor
• downwardapi
• dra
• e2e-kind-logs
• e2e-tests-always-pull-new-imgs
• e2e-update
• enable-events-on-e2e-tests
• enable_bandwidth
• entrypoint-refresh-kubeconfig
• entrypoint.sh
• example-pod
• exec_chroot_test
• failure-logs
• feature/multus-4.0
• fix-1186
• fix-archived-package
• fix-config-error-msg
• fix-cve-moshiur
• fix-e2e
• fix-e2e-dra
• fix-empty-cni-result
• fix-flaky-config-regen-test
• fix-legacy-entrypoint-e2e-tests
• fix-letter
• fix-logging
• fix-mount-prop-var-lib-kubelet
• fix-panic-arr
• fix-params
• fix-race-condition
• fix-readiness
• fix-readiness2
• fix-retry
• fix-strings-split
• fix-type-scope-comment
• fix-typo
• fix-typos
• fix-xnet/adil
• fix/1126
• fix/1130
• fix/773
• fix/948
• fix/add-disclaimers
• fix/change-image-url
• fix/change-org-name
• fix/chroot
• fix/cleanup-code
• fix/cmddel
• fix/cmddel-nostatus-update
• fix/cni-bin-dir
• fix/default-route
• fix/del-noerr
• fix/delegate-api
• fix/delete_old_main
• fix/filepath
• fix/fix-sliceindex
• fix/gateway-nil
• fix/go-mod-tidy
• fix/goreleaser
• fix/image
• fix/image-tag
• fix/license-boilerplate
• fix/liner-warning
• fix/m40-revive-err
• fix/mod-ver
• fix/no-network-error
• fix/okd-build
• fix/opt-out-mount-sa
• fix/path-injection
• fix/plugin-without-interface
• fix/reload-kubeconfig-if-failed
• fix/remove-global
• fix/remove-origin
• fix/remove-origin-build
• fix/revive
• fix/revive-err
• fix/server
• fix/so-long-gorilla
• fix/sriov
• fix/thick-log-options
• fix/update-nad-lib
• fix/use-multus-install
• fix/version
• fix/warning
• fix/wg-config
• fix_CmdDel
• fix_entrypoint
• fix_flag_usage
• fix_multiple_default_gw
• fix_url
• format-code
• getlivepod
• ginkgov2
• golang-e2e-tests
• health-endpoint-thick-plugin
• hostroot-mount-propagation
• idiomatic-config-unit-tests
• improve-v4-config
• jk_readme_thick_plugin
• jmh/limit_concurrency
• jmh/limit_listener
• kubeconfig-atomic-move
• kubelet_socket
• livequery-context
• manifest-toleration
• master
• master_cni_file
• mount-multus-conf-dir
• multus-4-docs-thickplugin-default
• multus40-prefer-configmap
• multus40-startup-tweaks
• nanafix
• net-attach-def-client-v172
• net-attach-def-client-v173
• net-attach-def-client-v174
• net-attach-def-lib-175
• netns-cni-del
• no-dproxy
• no-vendor
• noop-cni110
• notice
• opt_error
• optionally-process-specific-network-interface
• origin_build
• param-config-watcher-interval
• patch-1
• patch-2
• patch-3
• patch-4
• per-node-cert
• pod-deleted-on-add
• pod-not-found-on-add
• pod-uid
• podman-kind-cluster
• poduid-cni-del-path
• pollimmediate-errormsg
• pr-1302-update
• print-kubeconfig
• quick-start
• re_run_e2e_tests
• readme-updates
• rebase5-cni-subdirectory-chain
• recreate-configs-on-file-change
• refine-build
• refine-options
• refs/tags/v3.7.1
• refs/tags/v3.7.2
• refs/tags/v3.8
• refs/tags/v3.8.1
• refs/tags/v3.9
• refs/tags/v3.9.2
• refs/tags/v3.9.3
• refs/tags/v4.0-alpha
• refs/tags/v4.0.0
• refs/tags/v4.0.0-alpha
• refs/tags/v4.0.1
• refs/tags/v4.0.2
• refs/tags/v4.1.0
• refs/tags/v4.1.1
• refs/tags/v4.1.2
• refs/tags/v4.1.3
• refs/tags/v4.1.4
• refs/tags/v4.2.0
• refs/tags/v4.2.1
• refs/tags/v4.2.2
• release-policy
• release-v3
• remove-extra-mnt-point-thick
• remove-readiness-check-on-del
• remove-status-set-del
• remove-unused-cni-args
• replace-entrypoint-script-with-initcontainers
• resource_api
• resource_api_grpc
• retry-on-multiple-error-types-with-backoff
• return_error
• returnlogging
• save-informer-memory
• set_timeout
• shared-informer
• skip-api-ready-on-cni-del
• skip-dra
• slack-invite
• stale-issues
• stale-kubeconfig-hash
• static-pod-e2e
• supplment_log_for_stdindata
• support_mtu
• test
• test-gh-ci
• test-go
• test-macvlan
• test-multi-if-cni-result-gateway-marking
• test_go_path
• testsh
• thick-arch
• thick-arm
• thick-atomic-config-write
• thick-readiness
• thin
• thin-master-cni-file-flag
• try-older-debian-for-ci-jobs
• tryDelegate
• update-e2e
• update-gh-action
• update-go-version
• update-golang
• update-kind
• update-libcni
• update-net-attach-client
• update-net-attach-client-v3
• update-to-latest-kind-and-stable-k8s-release
• update_e2e_tools
• update_golang_version
• upgrade-fsnotify
• upgrade-reboot-tests
• upgradefix
• upstream-sync-redux-4.8
• use-same-base-image
• v1beta1
• yulng-e2e-readme

Build # 10947803402

Build Type

Pull #1338

github

Committed by dougbtv

Commit Message

This PR introduces graceful shutdown functionality to the Multus daemon by adding a /readyz endpoint

That is added alongside the existing /healthz. The /readyz endpoint starts returning 500 once a SIGTERM is received, indicating the daemon is in shutdown mode. During this time, CNI requests can still be processed for a short window. The daemonset configs have been updated to increase terminationGracePeriodSeconds from 10 to 30 seconds, ensuring we have a bit more time for these clean shutdowns.

This addresses a race condition during pod transitions where the readiness check might return true, but a subsequent CNI request could fail if the daemon shuts down too quickly. By introducing the /readyz endpoint and delaying the shutdown, we can handle ongoing CNI requests more gracefully, reducing the risk of disruptions during critical transitions.

Major thanks to @deads2k for the find, identification, fix, and of course, the explanations. Appreciate it.

Pull Request Pull Request #1338: Thick plugin graceful termination

Run Details

8 of 16 new or added lines in 1 file covered. (50.0%)

2348 of 3679 relevant lines covered (63.82%)

16.46 hits per line