pantsbuild / pants / 29627956700

Build # 29627956700

Build Type

push

github

Committed by web-flow

Commit Message

perf(golang): materialize package subset for third party compile (#23515)

When compiling third party Go packages, Pants materializes the entire
module digest into the sandbox. This can be expensive, especially for
large modules. And it quickly gets more expensive as you compile
multiple packages from that large module.

This change reduces that overhead significantly by slicing the digests
down to the package that is being compiled. There's 3 tiers to ensure
unaffected builds:
- When there's native code, we must keep the entire module since its
include closure can reference files not reported by the analysis.
- When it includes `//go:embed`, they can escape downwards in the
subtree, but not to the parent, so we must slice the dir recursively.
- In all other cases we can take a shallow slice of the package.

### Verification

Packaged 13 first party binaries with a mix of third party dependencies
including native libraries, with byte identical output.

### Benchmark

Measured with a cold `pants check` on ~160 first party packages
importing from a mix of ~650 third party modules, across ~3300 process
executions.

| metric | before | after | change
| :-- | --: | --: | --: |
| wall time | 465.8s | 242.9s | 1.92x faster |
| CPU sys | 3498.8s | 1491.1s | 2.35x less |
| CPU user | 461.5s | 407.3s | -12% |
| RSS max | 2.73 GB | 2.76 GB | ~flat |

The same check measured with a cache pre-seeded with module download and
analysis, covering only the compile phase.

| metric | before | after | change
| :-- | --: | --: | --: |
| wall time | 300.3s | 75.1s | 4.00x faster |
| CPU sys | 2232.7s | 241.2s | 9.26x less |
| CPU user | 385.1s | 333.9s | -13% |
| RSS max | 2.23 GB | 1.73 GB | -22% |

Sandbox materialization (system CPU) is the main driver for wall time
improvement.

Warm and cached runs measured unaffected.

Notice: Claude Code assisted with generating tests and
verification/benchmark tooling

Coverage Stats

94728 of 101871 relevant lines covered (92.99%)

3.68 hits per line

Source File

Press 'n' to go to next uncovered line, 'b' for previous

Source Not Available