pantsbuild / pants / 31327056242

93%

Build Type

push

github

Committed by web-flow

Commit Message

Run pants under free threaded python (#23477)

Switches Pants' own runtime to free-threaded CPython and makes the
engine and
Pants' Python code thread-safe without the GIL.

This has been validated as running without error in my companies CI, and
nets great speedups.

Most important callouts:
- Keep a persistent per-thread PyThreadState in the executor. Without
this, every `Python::attach` cycle tears down the thread state, and on
3.14t
that abandons the thread's mimalloc heaps. Created per tokio thread,
torn down in `on_thread_stop` (blocking
  threads are recycled, so the state must not leak per thread creation).
- Makes run-scoped internals thread-safe: rule_visitor, exception_sink,
logging, build_root, run_tracker.

I used https://github.com/pantsbuild/pants/pull/23437 as a launching
point.

I have on speculation set this as shipping in 2.34.x, as I believe 2.33
is too far along the dev-cycles. But I also do not see any reason to
delay further.

Blocks https://github.com/pantsbuild/scie-pants/pull/541

Edit:
This branch now serves as proof of green CI of the accumulation of
changes, and will be rebased as its parts are accepted incrementally:
- https://github.com/pantsbuild/pants/pull/23487
- https://github.com/pantsbuild/pants/pull/23488
- https://github.com/pantsbuild/pants/pull/23489
- https://github.com/pantsbuild/pants/pull/23490
- https://github.com/pantsbuild/pants/pull/23491
- https://github.com/pantsbuild/pants/pull/23492
- https://github.com/pantsbuild/pants/pull/23497
- https://github.com/pantsbuild/pants/pull/23514

Edit2:

Now that all pre-requisites are in, this PR contains CI/CD changes and
the new PANTS_FREE_THREADED option.
Current proposal for consumer UX is

| Consumer | `pants.toml#[GLOBAL].pants_free_threaded` | Resolves to |
Bootstrap warning |
|---|---|---|---|
| **scie-pants - not updated** | absent / `false` | `cp314` (GIL 3.14) |
- |
| **scie-pants - not updated** | `true` | `cp314` (GIL 3.14) | **"update
scie-pants"**. O... (continued)

Coverage Stats

42 of 49 new or added lines in 7 files covered. (85.71%)

18 existing lines in 1 file now uncovered.

95546 of 102872 relevant lines covered (92.88%)

3.66 hits per line