noironetworks / opflex

80%

master: 68%

LAST BUILD ON BRANCH svc-stale

branch: svc-stale

CHANGE BRANCH

x

Reset

• svc-stale
• 5.1.1.0
• 5.1.1.1
• 5.1.3.0
• 5.1.3.1
• 5.1.3.2
• 5.1.3.3
• 5.1.3.4
• 5.1.3.5
• 5.2.1.0
• 5.2.1.1
• 5.2.1.2
• 5.2.3.0
• 5.2.3.1
• 5.2.3.2
• 5.2.3.3
• 5.2.3.4
• 5.2.3.5
• 5.2.3.6
• 5.2.3.7
• Revert-Revert-openshift-hack-jefferson
• access-allow-untagged
• add-cwrapper-uts
• add-dir-order-to-sg
• add-extepfile-test
• add-lgtm-rules
• add-missing-stop
• add_pipeline_script
• addcodeqlfilter
• adjust_drop_counters
• anm-opflex-rpm
• anm-patch
• apic
• arm64
• asan-gbp-issues
• asyncjson
• asyncjson-kmr2
• asyncjson_kmr2
• atomic-1
• backport-fixes
• backport-kmr2
• backport-krm2
• backport_drop_prune_bug
• backport_droplog_part1
• backport_droplog_prune
• backport_droplog_pruning
• backport_egress_dns
• backport_prune_fix
• backport_tsan_fix
• bbva
• bhav-opflex-rpm
• bpf
• build-opt
• bump-release
• bump-vers-k+
• bump-version
• centos
• centos-master
• certs
• cfd-ut
• change-retry-timeout
• cicd-lmr-update
• cleanup-dotfiles
• cleanup-includes
• cleanup-lgtm-complaints
• cleanup-qos
• cleanup_droplogs
• clsr
• clsr2
• clsr3
• clsr4
• codacity
• codacity-8-14
• codacity-8-17
• codacity-cleanup
• codacity-usleep
• codacy-warnings
• collapse-debug-levels
• common-getuuid
• complaint-cleanup
• con_deny
• config_filter_droplog
• configurable-timeouts
• configured-peer-issue
• const-operator
• cov-1
• cov-2
• cov-8-21
• cov-exc
• cov-fixes
• cov-issue
• cov_0513
• coveralls
• coveralls-logs
• coveralls-reporting-fixes
• coverity
• coverity-8-14
• coverity-fixes-uninit
• coverity_0529
• coverity_0531
• coverity_0601
• coverity_0602
• coverity_fix_01
• cpp-min-version-change
• cpp14-bump
• csr-flows
• csr-integration
• csr-service
• ct-rel
• deb-build-problems
• deb-rules
• debian_warnings-2
• debian_warnings-3
• deny_rule_2
• deny_rules
• devtoolset-8
• disable-scheduled-codeql-scan
• disable-untagged-dup
• dns_bkup_restore
• dns_fixes
• dns_records
• dns_uts
• drop-log-redirect-args
• drop_deny
• dropflow-ut
• droplog-uts
• droplog_part1
• droplog_test
• droplogcfg
• droplogger_thread
• dual-stack
• dup-os-on-os
• egress_dns
• enable-prometheus
• enable-sanitizers
• encap
• enhance-epfile-tests
• epcounter
• epcounter-tests
• epdeclare
• epfaults_UT
• erspan-id
• erspan-update-port
• erspan-updates
• event_array_fix
• exc_err_log
• except-1
• fault_manager
• faultcode
• faults-rpm-deb
• faults_changes
• file_dump_unresolved
• fix-60min-outage
• fix-certs
• fix-cov-issue
• fix-deb-rpm
• fix-liopflex-uts-ubuntu2304
• fix-lmr
• fix-pre-handshake-timeouts
• fix-ssl
• fix-ssl-version
• fix-travis-badge
• fix-unavail-resolves
• fix_acc_br_pktin
• fix_commit_message
• fix_commit_message_kmr2
• fix_droplog_access_flow
• fix_dropprunecore
• fix_lgtm
• fix_tsan
• fix_uninit_tests
• fixes
• flag_sync
• flaky-tests-6-16
• focal-disable-deb-test
• focal-travis
• fsdroplog-ut
• gautvenk-patch-1
• gbp_inspect-crash
• gbpinspect
• gbpserver-coverity
• genie-cleanup
• genie-codacity
• genie-gen-c++11
• genie-relation-uris
• genie-remove-reverse-relationship
• githash-incorrect
• gprof
• grafana
• grafana-2
• grafana-6-23
• grafana-images
• grafana-uc
• grpc-cache
• grpc-coverity
• grpc-travis
• grpc-v1.31.0
• high_freq_timers
• high_freq_timers-lmr-6.0.4
• high_freq_timers-mmr-6.1.1
• incorrect_arg_droplog
• increase-ovsdb-buffer-size
• indus-cfd
• inspect-timeout
• ipfix-domid
• ipfix-fields
• ipfix-output-only
• jammy-disabledgprof
• jammy-travis-wait
• jefferson
• jefferson-invalid-read
• jefferson-lb-loopback
• jefferson-logs
• jefferson-ovsdb
• jefferson-peer-list-reset
• jefferson-svc-stats-thread
• jsonrcp-errormsg
• jsonrpc-monitor
• jsonrpc-monitor-support
• judith
• judith-buffer-issue
• judith-buffer-size-bump
• judith-nat-prio
• judith-peer-reset
• judith-remote-ip
• kmr2
• kmr2-5.2.7
• kmr2-bbva
• kmr2-change-retry-timeout
• kmr2-restart
• kmr2-revert
• kwai
• kwai-keepalive
• kwai-nat-prio
• kwai-prom-0.12.1
• kwai-remote-ip
• kwai-session-aff
• l2c
• lgtm-1
• lgtm-2
• lgtm-cleanup
• lgtm-coverty-1
• lgtm-fixes
• lgtm-grpc
• lgtm-grpc-build
• lgtm-lo
• lgtm-os
• lgtm-prom
• lgtm-revert
• lgtm_2
• lgtm_build
• lgtm_fixes
• lmr
• lmr-6.0.4
• lmr-inspect
• lmr-stoul
• lmr-ubi8
• local-hpp-counters
• local-hpp-tests
• lock-cleanup-1
• logaction
• loopback-lb
• mac-vector-leak
• mac_mask
• master
• mcast-flaky
• minor_br_access_fix
• mmr-6.1.1
• mock-server-1
• modb-count-agent
• more-cov-issues
• more-uts
• more_default_rules
• move-to-ovs-renderer
• mtufault
• multi-travis
• multiple-switches
• nat
• nat-fixes
• nat-prio
• of-timeout
• ofpeer
• ofstats-del
• openssl
• opflex-fixes
• opflex-hpp-optimization
• opflex-rpm
• opflex-svc-stats-thread
• opflex_denyrules
• opflex_epfaults
• os-deadlock-8-10
• os-gbp-1
• os-grafana
• os-refactor
• os-refactor-2
• os-stats-8-20
• os-tsan-1
• os-tsan-7-23
• os-tsan-8-6
• ospm-7-2
• overflow-on-read
• overlay_tunnel_adv
• ovs-lib-includes
• ovsdb-async
• ovsdb-connect
• ovsdb-id
• ovsdb-rework
• ovsdb-test-update
• ovsdb-update
• ovsdb-update-mock-fix
• ovsdb-ut
• parse_fix
• parse_srv
• peer-tsan
• permit_logging
• policydrop
• port-commits
• port-tunneladv-change
• prefer-range-for
• prom-conditionals
• prom-define-remove
• prom-deps
• prom-v1.0.1
• prom-v1.0.1-kmr2
• prom_unit_test
• prometheus-cpp
• prometheus-doc
• prr-fixes
• prune_bug
• pruning_core_fix
• pthread-6-8
• qos-coverage
• recursive_resolve
• reduce-logs
• reduce-noisy-logs
• reduce-travis-usage
• remote-ep
• remoteep-tests
• remove-asssert
• remove-clone
• remove-cpp-version-checks
• remove-debug5
• remove-epg-erspan
• remove-oftypes
• remove-simstats
• remove-snat-unset
• remove-unused-method
• remove-vlog
• remove_travis_tags
• removelog
• renderer-stop-fix
• reset-wait
• restart
• revert-258-dscp_2
• revert-297-jefferson-svc-stats-thread
• revert-Revert-oshift-dup
• revert-agent-c++14
• rework-boost-placeholder-usage
• rework-jsonrpc-code
• rhel8-rpm
• rhel8rpm
• rpm-deb-8-23
• rpm_build_fix
• rpm_fix
• rpmbuild
• runtime-6-10
• serialize-unresolved
• server-prom
• server-readme
• service-UT
• service-secgrp-bypass
• session-aff
• set-erspan-dir
• sm-7-13
• snat
• snat-bounce
• snat-fixes
• snat-uts
• snat_test_coverage
• span-bridge-fix
• span-coverage
• span-create-delete
• span-ut
• span-uts
• srv_flow_changes
• ssu-wipe
• startup
• startup-enable
• std-to-string
• sumit-kmr2-travis-builds
• svc-arp
• svc-bypass-secgrp-jefferson
• svc-dup
• svc-loopback
• svc-stats-disable
• svc-stats-flow-disable
• svc_flood
• sys-stats-1
• test-dscp
• test-en
• test-old-master
• test-updatemos
• testing
• time_gprof
• timer-mutex
• timer-protect
• totals-1
• travis-focal
• travis-log
• travis-logs
• travis-rc
• travis_build_fix
• triangular-deadlock
• tsan-afm
• tsan-check
• tsan-clang-cleanup
• tsan-cleanup
• tsan-epcounter
• tsan-erspan
• tsan-fixes
• tsan-issues
• tsan-ubuntu2010
• tsan_cov_fix
• tsanissues-jsonrpc
• tunnel-mtx
• tupledata-rename
• ubi
• ubi9rpmbuild
• ubsan
• ubsan-disable
• ubsan-enable
• ubuntu-21.04-issue
• ubuntu2010-failure
• ubuntu23.10
• uc-6-29
• unicast_garp
• unrel_count
• unrel_pol2
• unrel_pol3
• unresolved_relations
• unresolved_relations_2
• unresolved_rels
• update-codeql-config
• update-docs
• upgrade-prom-cpp
• vmotion-reset-all-peers

Fix stale entries seen after deletion of service during make check tests

Issue was happening once every few runs. But then it became unreproducible due to the way threads were scheduled. Tried 25+ times with the fix and things look good.

Description:
- SNAT/DNAT flows will be programmed initially from higher prio Agent thread.
- Low prio Svc Stats thread today programs the flows with stats cookies
later on so that snat/dnat flows become eventually consistent to correlate stats.
- In case svc gets deleted, then agent io thread will delete svc flows immediately.
Since svc stats thread is low priority, it could be in the process of updating
SNAT/DNAT flows after agent thread deleted SNAT/DNAT flows. This can lead to
stale snat/dnat flow entries for that svc.
- To avoid these stale entries, call programServiceSnatDnatFlows() for the deleted
svc so that the stale entries (if any) can be cleared for this service. If there
are no stale entries (already deleted), then the below call is a NO-OP.
- Also, when a svc is deleted, prior to the fix we were looping over list of svcs
to update snat/dnat flows. This loop is needed only for EP updates. For svc updates,
we just need to call programServiceSnatDnatFlows() only once for that svc.

Signed-off-by: Gautam Venkataramanan <gautam.chennai@gmail.com>