skeema / skeema / 13447353665

93%

main: 92%

push

github

workspaces: for table cleanup, replace concurrency with chunking

MariaDB's foreign key locking behavior during DDL changed in their May 2024
server releases for MariaDB 10.6 and up. As a result of that change, if tables
with circular foreign key relationships are dropped concurrently, the DROP
TABLE statements can enter an infinite loop due to server bug MDEV-36129. To
prevent Skeema from ever triggering this situation, this commit removes
concurrent table dropping during workspace cleanup, for all flavors. Instead,
we now drop tables using a single thread, but potentially dropping several
tables per statement, depending on the server flavor and temp-schema-threads
option.

Background on Skeema workspace cleanup behavior:

To interpret the CREATE statements in *.sql files, Skeema runs the statements
in a temporary location, called a workspace. When cleaning up a workspace, we
drop tables separately *before* dropping the entire workspace database,
because DROP DATABASE with many tables present (even if all empty) can hold
server locks/mutexes for a disruptively long time period.

Skeema used concurrency in these DROP TABLEs, controlled by the
temp-schema-threads option when using the default workspace=temp-schema. Even
though these DROP TABLEs likely get serialized behind global locks behind the
scenes, queueing up concurrent DROPs still helped cleanup performance on slow
networks, as this avoids full network round-trip between each DROP. However,
this concurrent logic has already needed numerous tweaks over the years:

* In older MySQL releases and in MariaDB, dropping a table causes a buffer
  pool scan, even if the table is empty. With larger buffer pools, this harms
  DROP TABLE performance, so Skeema disabled concurrent cleanup if the buffer
  pool was 32 GB or larger.

* In MySQL 8, foreign key metadata locking was changed such that it extends
  MDLs from child tables to parent tables. When dropping tables concurrently,
  this could result in so... (continued)

62 of 68 new or added lines in 5 files covered. (91.18%)

10266 of 11005 relevant lines covered (93.28%)

1.11 hits per line