pantsbuild / pants / 29627956700
93%

DEFAULT BRANCH: main
Ran
Jobs 11
Files 1771
Run time 3min
Badge
Embed ▾
README BADGES
x

If you need to use a raster PNG badge, change the '.svg' to '.png' in the link

Markdown

Textile

RDoc

HTML

Rst
coverage: 92.988% (+0.002%) from 92.986%
29627956700

push

github

web-flow 
perf(golang): materialize package subset for third party compile (#23515)

When compiling third party Go packages, Pants materializes the entire
module digest into the sandbox. This can be expensive, especially for
large modules. And it quickly gets more expensive as you compile
multiple packages from that large module.

This change reduces that overhead significantly by slicing the digests
down to the package that is being compiled. There's 3 tiers to ensure
unaffected builds:
- When there's native code, we must keep the entire module since its
include closure can reference files not reported by the analysis.
- When it includes `//go:embed`, they can escape downwards in the
subtree, but not to the parent, so we must slice the dir recursively.
- In all other cases we can take a shallow slice of the package.

### Verification

Packaged 13 first party binaries with a mix of third party dependencies
including native libraries, with byte identical output.

### Benchmark

Measured with a cold `pants check` on ~160 first party packages
importing from a mix of ~650 third party modules, across ~3300 process
executions.

| metric | before | after | change
| :-- | --: | --: | --: |
| wall time | 465.8s | 242.9s | 1.92x faster |
| CPU sys | 3498.8s | 1491.1s | 2.35x less |
| CPU user | 461.5s | 407.3s | -12% |
| RSS max | 2.73 GB | 2.76 GB | ~flat |

The same check measured with a cache pre-seeded with module download and
analysis, covering only the compile phase.

| metric | before | after | change
| :-- | --: | --: | --: |
| wall time | 300.3s | 75.1s | 4.00x faster |
| CPU sys | 2232.7s | 241.2s | 9.26x less |
| CPU user | 385.1s | 333.9s | -13% |
| RSS max | 2.23 GB | 1.73 GB | -22% |

Sandbox materialization (system CPU) is the main driver for wall time
improvement.

Warm and cached runs measured unaffected.

Notice: Claude Code assisted with generating tests and
verification/benchmark tooling

94728 of 101871 relevant lines covered (92.99%)

3.68 hits per line

Coverage Regressions

Lines Coverage ∆ File
27
90.36
 0.28% src/python/pants/backend/go/util_rules/third_party_pkg.py
22
90.07
 0.0% src/python/pants/backend/go/util_rules/build_pkg_target.py
9
94.64
 0.77% src/python/pants/backend/go/util_rules/build_pkg_target_test.py
Jobs
ID Job ID Ran Files Coverage
1 test_python_linux_x86_64_6/10 - 29627956700.1 1206
49.71
 GitHub Action Run
2 test_python_linux_x86_64_7/10 - 29627956700.2 1212
50.83
 GitHub Action Run
3 test_python_linux_x86_64_3/10 - 29627956700.3 1209
52.37
 GitHub Action Run
4 test_python_linux_x86_64_1/10 - 29627956700.4 1211
49.01
 GitHub Action Run
5 test_python_linux_x86_64_5/10 - 29627956700.5 1224
59.93
 GitHub Action Run
6 test_python_linux_x86_64_2/10 - 29627956700.6 1211
58.65
 GitHub Action Run
7 test_python_linux_x86_64_9/10 - 29627956700.7 1221
53.99
 GitHub Action Run
8 test_python_linux_x86_64_8/10 - 29627956700.8 1210
52.87
 GitHub Action Run
9 test_python_linux_x86_64_4/10 - 29627956700.9 1241
59.0
 GitHub Action Run
10 test_python_linux_x86_64_0/10 - 29627956700.10 1231
57.38
 GitHub Action Run
11 test_python_linux_arm64 - 29627956700.11 1190
51.76
 GitHub Action Run
Source Files on build 29627956700