skeema / skeema / 16789023696

92%

push

github

workspace options: add temp-schema-mode, deprecate temp-schema-threads

This commit introduces a new workspace option for configuring temp-schema
performance and concurrency behaviors. The new option, temp-schema-mode,
accepts a descriptive enum value, rather than an int like the older
temp-schema-threads option.

This commit also deprecates temp-schema-threads, with an intention of likely
removing it entirely in Skeema v2. There were two main problems with the
design of temp-schema-threads: many users tend to set it far too high, causing
contention which actually harms performance; and its name is misleading in its
effects for workspace creation (multi-threaded) vs workspace cleanup (chunked
but single-threaded in recent Skeema versions, to avoid problems with
frequently-changing FK lock behaviors in MariaDB).

In the process of implementing temp-schema-mode, an additional major
performance enhancement has been added: ability to use chunking for workspace
table creation, mitigating network latency by creating multiple tables per
round-trip. This functionality makes use of the server's multi-statement
support. It is only enabled for tables, to avoid potential problems from typos
inside compound statements used by stored program bodies.

At this time, there are 5 possible temp-schema-mode values:

* serial: single-threaded creation, no chunking for creation or cleanup. This
  is safest when operating on a busy production server, especially an older
  server version which has problematic data dictionary mutex behavior;
  however, Skeema's performance can suffer substantially with this mode, which
  is effectively the same as temp-schema-threads=1.
* light: use some concurrency for creation, but no chunking; use a small
  amount of chunking for cleanup. Similar motivations as serial mode, but not
  nearly as restrictive/slow.
* regular (default): Behaves similarly to the old default for
  temp-schema-threads, with the additional enhancement of enabling w... (continued)

257 of 286 new or added lines in 6 files covered. (89.86%)

10789 of 11603 relevant lines covered (92.98%)

1.11 hits per line