noironetworks / opflex

79%

master: 68%

LAST BUILD ON BRANCH triangular-deadlock

branch: triangular-deadlock

CHANGE BRANCH

x

Reset

• triangular-deadlock
• 5.1.1.0
• 5.1.1.1
• 5.1.3.0
• 5.1.3.1
• 5.1.3.2
• 5.1.3.3
• 5.1.3.4
• 5.1.3.5
• 5.2.1.0
• 5.2.1.1
• 5.2.1.2
• 5.2.3.0
• 5.2.3.1
• 5.2.3.2
• 5.2.3.3
• 5.2.3.4
• 5.2.3.5
• 5.2.3.6
• 5.2.3.7
• Revert-Revert-openshift-hack-jefferson
• access-allow-untagged
• add-cwrapper-uts
• add-dir-order-to-sg
• add-extepfile-test
• add-lgtm-rules
• add-missing-stop
• add_pipeline_script
• addcodeqlfilter
• adjust_drop_counters
• anm-opflex-rpm
• anm-patch
• apic
• arm64
• asan-gbp-issues
• asyncjson
• asyncjson-kmr2
• asyncjson_kmr2
• atomic-1
• backport-fixes
• backport-kmr2
• backport-krm2
• backport_drop_prune_bug
• backport_droplog_part1
• backport_droplog_prune
• backport_droplog_pruning
• backport_egress_dns
• backport_prune_fix
• backport_tsan_fix
• bbva
• bhav-opflex-rpm
• bpf
• build-opt
• bump-release
• bump-vers-k+
• bump-version
• centos
• centos-master
• certs
• cfd-ut
• change-retry-timeout
• cicd-lmr-update
• cleanup-dotfiles
• cleanup-includes
• cleanup-lgtm-complaints
• cleanup-qos
• cleanup_droplogs
• clsr
• clsr2
• clsr3
• clsr4
• codacity
• codacity-8-14
• codacity-8-17
• codacity-cleanup
• codacity-usleep
• codacy-warnings
• collapse-debug-levels
• common-getuuid
• complaint-cleanup
• con_deny
• config_filter_droplog
• configurable-timeouts
• configured-peer-issue
• const-operator
• cov-1
• cov-2
• cov-8-21
• cov-exc
• cov-fixes
• cov-issue
• cov_0513
• coveralls
• coveralls-logs
• coveralls-reporting-fixes
• coverity
• coverity-8-14
• coverity-fixes-uninit
• coverity_0529
• coverity_0531
• coverity_0601
• coverity_0602
• coverity_fix_01
• cpp-min-version-change
• cpp14-bump
• csr-flows
• csr-integration
• csr-service
• ct-rel
• deb-build-problems
• deb-rules
• debian_warnings-2
• debian_warnings-3
• deny_rule_2
• deny_rules
• devtoolset-8
• disable-scheduled-codeql-scan
• disable-untagged-dup
• dns_bkup_restore
• dns_fixes
• dns_records
• dns_uts
• drop-log-redirect-args
• drop_deny
• dropflow-ut
• droplog-uts
• droplog_part1
• droplog_test
• droplogcfg
• droplogger_thread
• dual-stack
• dup-os-on-os
• egress_dns
• enable-prometheus
• enable-sanitizers
• encap
• enhance-epfile-tests
• epcounter
• epcounter-tests
• epdeclare
• epfaults_UT
• erspan-id
• erspan-update-port
• erspan-updates
• event_array_fix
• exc_err_log
• except-1
• fault_manager
• faultcode
• faults-rpm-deb
• faults_changes
• file_dump_unresolved
• fix-60min-outage
• fix-certs
• fix-cov-issue
• fix-deb-rpm
• fix-liopflex-uts-ubuntu2304
• fix-lmr
• fix-pre-handshake-timeouts
• fix-ssl
• fix-ssl-version
• fix-travis-badge
• fix-unavail-resolves
• fix_acc_br_pktin
• fix_commit_message
• fix_commit_message_kmr2
• fix_droplog_access_flow
• fix_dropprunecore
• fix_lgtm
• fix_tsan
• fix_uninit_tests
• fixes
• flag_sync
• flaky-tests-6-16
• focal-disable-deb-test
• focal-travis
• fsdroplog-ut
• gautvenk-patch-1
• gbp_inspect-crash
• gbpinspect
• gbpserver-coverity
• genie-cleanup
• genie-codacity
• genie-gen-c++11
• genie-relation-uris
• genie-remove-reverse-relationship
• githash-incorrect
• gprof
• grafana
• grafana-2
• grafana-6-23
• grafana-images
• grafana-uc
• grpc-cache
• grpc-coverity
• grpc-travis
• grpc-v1.31.0
• high_freq_timers
• high_freq_timers-lmr-6.0.4
• high_freq_timers-mmr-6.1.1
• incorrect_arg_droplog
• increase-ovsdb-buffer-size
• indus-cfd
• inspect-timeout
• ipfix-domid
• ipfix-fields
• ipfix-output-only
• jammy-disabledgprof
• jammy-travis-wait
• jefferson
• jefferson-invalid-read
• jefferson-lb-loopback
• jefferson-logs
• jefferson-ovsdb
• jefferson-peer-list-reset
• jefferson-svc-stats-thread
• jsonrcp-errormsg
• jsonrpc-monitor
• jsonrpc-monitor-support
• judith
• judith-buffer-issue
• judith-buffer-size-bump
• judith-nat-prio
• judith-peer-reset
• judith-remote-ip
• kmr2
• kmr2-5.2.7
• kmr2-bbva
• kmr2-change-retry-timeout
• kmr2-restart
• kmr2-revert
• kwai
• kwai-keepalive
• kwai-nat-prio
• kwai-prom-0.12.1
• kwai-remote-ip
• kwai-session-aff
• l2c
• lgtm-1
• lgtm-2
• lgtm-cleanup
• lgtm-coverty-1
• lgtm-fixes
• lgtm-grpc
• lgtm-grpc-build
• lgtm-lo
• lgtm-os
• lgtm-prom
• lgtm-revert
• lgtm_2
• lgtm_build
• lgtm_fixes
• lmr
• lmr-6.0.4
• lmr-inspect
• lmr-stoul
• lmr-ubi8
• local-hpp-counters
• local-hpp-tests
• lock-cleanup-1
• logaction
• loopback-lb
• mac-vector-leak
• mac_mask
• master
• mcast-flaky
• minor_br_access_fix
• mmr-6.1.1
• mock-server-1
• modb-count-agent
• more-cov-issues
• more-uts
• more_default_rules
• move-to-ovs-renderer
• mtufault
• multi-travis
• multiple-switches
• nat
• nat-fixes
• nat-prio
• of-timeout
• ofpeer
• ofstats-del
• openssl
• opflex-fixes
• opflex-hpp-optimization
• opflex-rpm
• opflex-svc-stats-thread
• opflex_denyrules
• opflex_epfaults
• os-deadlock-8-10
• os-gbp-1
• os-grafana
• os-refactor
• os-refactor-2
• os-stats-8-20
• os-tsan-1
• os-tsan-7-23
• os-tsan-8-6
• ospm-7-2
• overflow-on-read
• overlay_tunnel_adv
• ovs-lib-includes
• ovsdb-async
• ovsdb-connect
• ovsdb-id
• ovsdb-rework
• ovsdb-test-update
• ovsdb-update
• ovsdb-update-mock-fix
• ovsdb-ut
• parse_fix
• parse_srv
• peer-tsan
• permit_logging
• policydrop
• port-commits
• port-tunneladv-change
• prefer-range-for
• prom-conditionals
• prom-define-remove
• prom-deps
• prom-v1.0.1
• prom-v1.0.1-kmr2
• prom_unit_test
• prometheus-cpp
• prometheus-doc
• prr-fixes
• prune_bug
• pruning_core_fix
• pthread-6-8
• qos-coverage
• recursive_resolve
• reduce-logs
• reduce-noisy-logs
• reduce-travis-usage
• remote-ep
• remoteep-tests
• remove-asssert
• remove-clone
• remove-cpp-version-checks
• remove-debug5
• remove-epg-erspan
• remove-oftypes
• remove-simstats
• remove-snat-unset
• remove-unused-method
• remove-vlog
• remove_travis_tags
• removelog
• renderer-stop-fix
• reset-wait
• restart
• revert-258-dscp_2
• revert-297-jefferson-svc-stats-thread
• revert-Revert-oshift-dup
• revert-agent-c++14
• rework-boost-placeholder-usage
• rework-jsonrpc-code
• rhel8-rpm
• rhel8rpm
• rpm-deb-8-23
• rpm_build_fix
• rpm_fix
• rpmbuild
• runtime-6-10
• serialize-unresolved
• server-prom
• server-readme
• service-UT
• service-secgrp-bypass
• session-aff
• set-erspan-dir
• sm-7-13
• snat
• snat-bounce
• snat-fixes
• snat-uts
• snat_test_coverage
• span-bridge-fix
• span-coverage
• span-create-delete
• span-ut
• span-uts
• srv_flow_changes
• ssu-wipe
• startup
• startup-enable
• std-to-string
• sumit-kmr2-travis-builds
• svc-arp
• svc-bypass-secgrp-jefferson
• svc-dup
• svc-loopback
• svc-stale
• svc-stats-disable
• svc-stats-flow-disable
• svc_flood
• sys-stats-1
• test-dscp
• test-en
• test-old-master
• test-updatemos
• testing
• time_gprof
• timer-mutex
• timer-protect
• totals-1
• travis-focal
• travis-log
• travis-logs
• travis-rc
• travis_build_fix
• tsan-afm
• tsan-check
• tsan-clang-cleanup
• tsan-cleanup
• tsan-epcounter
• tsan-erspan
• tsan-fixes
• tsan-issues
• tsan-ubuntu2010
• tsan_cov_fix
• tsanissues-jsonrpc
• tunnel-mtx
• tupledata-rename
• ubi
• ubi9rpmbuild
• ubsan
• ubsan-disable
• ubsan-enable
• ubuntu-21.04-issue
• ubuntu2010-failure
• ubuntu23.10
• uc-6-29
• unicast_garp
• unrel_count
• unrel_pol2
• unrel_pol3
• unresolved_relations
• unresolved_relations_2
• unresolved_rels
• update-codeql-config
• update-docs
• upgrade-prom-cpp
• vmotion-reset-all-peers

Fix more data race issues

1) Triangular deadlock
Scenario:
Svc Stats thread:
Holds switch manager mutex (sm_mutex) and sends barrier message to OVS and is stuck to get a response.
CondVar needs to be notified by sw connection thread.
Due to this all flow CRUD stalls.

Agent thread:
During contract/table-drop timer kickoff, it holds pstatMtx (to protect while going through all tables and collects stats from flows).
It is blocked while trying to acquire switch manager mutex.... (which is needed to protect flow writes)

Switch Connection thread:
Tries to acquire pstatMtx during stats response, and is stuck
It wont be able to send condVar notify to unblock svc stats thread since it doesnt get a chance to service barrier response.....

Fix:
Unfortunately not detected by tsan. The fix is to break the cycle by making the lock order static.
i.e. "sm_mutex --> pstatMtx"

2) Optimization: Reduce critical section of svcStatsMutex. Both stats CRUD and collection handling are done by Svc Stats thread. Reducing the critical section just to protect cookie podsvc map.

3) Protecting idgen maps during cleanup timer that does garbage collection (every 360 seconds)

Signed-off-by: Gautam Venkataramanan <gautam.chennai@gmail.com>