tarantool / test-run

63%

master: 63%

LAST BUILD ON BRANCH Totktonada/gh-287-lift-pipe-buffer-size-restriction

branch: Totktonada/gh-287-lift-pipe-buffer-size-restriction

CHANGE BRANCH

x

Reset

• Totktonada/gh-287-lift-pipe-buffer-size-restriction
• 0x501D/fix-get_iproto_port
• 0x501D/fix-luarocks-warnings
• Totktonada/add-github-actions-step-summary
• Totktonada/bump-luatest-to-1.0.1-15-gfcabd2a
• Totktonada/calc-jobs-using-available-cpus
• Totktonada/descrease-unix-socket-path-length
• Totktonada/dynamic-test-report-width
• Totktonada/fix-worker-failure-on-failed-tap13-parsing
• Totktonada/gh-22-tags
• Totktonada/gh-325-add-env-option
• Totktonada/gh-361-status-line
• Totktonada/gh-382-add-test-timeout-for-luatest
• Totktonada/gh-437-fix-ability-to-run-test-several-times
• Totktonada/luatest-experimenting
• Totktonada/luatest-run-test-cases-in-parallel
• Totktonada/prettify-help-output
• Totktonada/qa-128-strengthen-wait-cond
• Totktonada/test-python-versions-in-ci
• Totktonada/update-flake8
• VitaliyaIoffe/gh-276-hangs-timeout-doesnt-work
• VitaliyaIoffe/gh-304-include-luatest
• add-extra-log-msg
• avtikhon/gh-qa-97-disk-bound-collect
• avtikhon/gh-qa-97-disk-bound-collect-TMP
• avtikhon/restart-tests
• avtikhon/restart-tests_TMP
• bump-luatest
• bump-luatest-to-1.0.0-3-gf37b353
• dependabot/pip/gevent-23.9.0
• drewdzzz/wait_cluster_vclock_fix
• fix-flake8-E721-do-not-compare-types
• gh-119-buffering-problems
• gh-330-fix-consistent-mode
• gh-tarantool-6646-workflow-catching-flaky-tests
• gorcunov/wait-log-drop-redundant-assert
• grafin/add_ignore_zero_to_get_vclock
• grafin/delete-zero-vclock-in-get_cluster_vclock
• i-cod/fix-bug-when-lua-script-name-truncated-by-dot
• i-cod/fix-incorrect-char-in-log-file-name
• i-cod/timeout-adjustment
• iverbin/fix-non-utf8-chars-parsing
• kyukhin/enable-parallel-fragile
• ligurio/add-unittest
• ligurio/fix-typo
• ligurio/support-skip-tap13
• locker/rm-luatest-submodule-v2
• lua-workflow
• master
• mechanik20051988/implement-lua-libs-for-luatests
• sergos/qa-272
• sergos/readme-update
• skaplun/pretest-clean-dis-arm64
• slavakirichenko/refactoring-ci-workflow
• utf8-fixup
• ylobanklov/improve-failed-test-report
• ylobankov/add-retries-option
• ylobankov/bump-gevent
• ylobankov/bump-luatest
• ylobankov/bump-luatest-to-new-version
• ylobankov/change-vardir-default
• ylobankov/consider-flaky-tests
• ylobankov/drop-checksums
• ylobankov/fix-ci
• ylobankov/fix-crash-detect
• ylobankov/fix-ports-issue
• ylobankov/fix-server-start-check
• ylobankov/fix-server-start-check-backup
• ylobankov/fix-test-run
• ylobankov/handle-server-start-errors-in-py-tests
• ylobankov/ignore-sockets-on-artifact-copying
• ylobankov/improve-ports-lua-script
• ylobankov/kill-procs-of-not-started-servers
• ylobankov/read-logfile-safely
• ylobankov/refactor-luatest-helpers
• ylobankov/retry-all-failed-tests
• ylobankov/run-luatest-tests-by-pattern
• ylobankov/switch-to-python2
• ylobankov/update-luatest
• ylobankov/update-requirements

Build # 9285128968

Build Type

Pull #434

github

Committed by Totktonada

Commit Message

dispatcher: lift pipe buffer size restriction

A task queue dispatcher puts all the tasks to the task queue at startup.
Then workers are started and are taking the tasks from it.

If there are many tasks in a task group (which roughly corresponds to a
test suite), we can reach the pipe buffer size on putting into the
queue, because `multiprocessing.SimpleQueue` uses a pipe under the hood.

The solution is to use `multiprocessing.Queue`, which has an
intermediate buffer before the underlying pipe and writes to the pipe in
a background thread, without blocking a thread that calls
`<queue>.put()`.

The `Queue` API is a superset of the `SimpleQueue` API, so we can just
replace the implementation.

Let's also use `Queue` for the worker's output queue to be on the safe
side and for consistency.

Fixes #287

Pull Request Pull Request #434: dispatcher: lift pipe buffer size restriction

Run Details

760 of 1564 branches covered (48.59%)

Branch coverage included in aggregate %.

3 of 4 new or added lines in 1 file covered. (75.0%)

1 existing line in 1 file now uncovered.

2938 of 4349 relevant lines covered (67.56%)

0.68 hits per line