pantsbuild / pants / 28296853179

93%

push

github

perf(jvm): Add option to enable remote cache for intermediate JARs (#23408)

Remote caching of all jars was [disabled some time
ago](https://github.com/pantsbuild/pants/pull/21290) due to it being
slower to fetch from cache than re-executing the process to produce the
jar. However, this is not the case if you have a fast remote cache on an
internal network in a CI environment, especially when executing a high
volume of these tasks. It turns into a bit of a fork bomb on a fully
remotely cached run as it has to spawn a ton of tiny processes to
construct these jars.

These changes allow enabling remote cache for the intermediate jars,
which are both generally small and can be numerous. In my case we have
over 1000 compile targets that are going through these processes, and
the cost of churning them greatly outweighs going to the remote cache in
CI.

This PR:
- Adds the `[jvm].intermediate_jars_remote_cache` option to enable
remote cache of intermediate and intermediate-stripped jars.
- Adds the `[jvm].intermediate_jars_remote_cache_speculation_delay` to
configure a delay before spawning these micro-processes.

Here's some benchmarks on 1000+ targets simulating a CI environment
where these would be cached remotely but not locally.

| Core Metric | Fully Cached Run | Shallow Incremental Change | Large
Cascading Change |
| :--- | :--- | :--- | :--- |
| **Wall-Clock Time** | 83s → **11s** `(-87%)` | 81s → **17s** `(-79%)`
| 208s → **61s** `(-71%)` |
| **User CPU Time** | 201s → **10s** `(-95%)` | 205s → **11s** `(-95%)`
| 1,442s → **14s** `(-99%)` |
| **Kernel CPU Time**| 164s → **7s** `(-95%)` | 160s → **9s** `(-94%)` |
267s → **51s** `(-81%)` |
| **Local OS Forks** | 2,449 → **4** `(-100%)` | 2,451 → **11**
`(-100%)` | 2,981 → **540** `(-82%)` |
| **Context Switches** | 2,520,167 → **671,455** `(-73%)` | 2,522,822 →
**715,700** `(-72%)` | 6,038,084 → **944,366** `(-84%)` |
| **Page Reclaims** | 15,488,957 → **576,400** `(-96%)` | 15,511,196 →
**593,848** `... (continued)

14 of 15 new or added lines in 4 files covered. (93.33%)

93197 of 100433 relevant lines covered (92.8%)

3.69 hits per line