pantsbuild / pants / 40845801144

93%

push

github

perf(jvm): stop merging a jvm_artifact's transitive jars into its entry (#23737)

`jvm_artifact` classpath entries held their transitive jars in a single
digest. So a widely depended-on jar, like scala-library, was included in
several classpath entry digests. These duplicates were then materialized
to sandboxes, such as for compile. The classpath also included the
duplicates. Their paths included the fingerprint of the merged digest,
so caches in for example scalac held the duplicate in memory. That also
grew by how many artifacts pulled in the same jars.

This separates them. So a `jvm_artifact` only holds its own jar in its
classpath entry digest, and a new field `exports` is added which refers
to the transitive jars by their own classpath entries. Anything that
previously merged into the classpath entries now gets exported instead.

`export_closure`, `closure`, `args` and `immutable_inputs(_args)`
include each export right after its exporter, to retain the same jar
order. `args` and `immutable_inputs(_args)` take `transitive=True` to
cover the closure in one walk. Equality uses the same trick as
`CoarsenedTarget` and `BuildGoPackageRequest` to only visit each pair
once, as it now represents this transitive graph instead of pre-merging
it.

On a relatively large scala codebase, 57% of the compile classpath was
duplicates. This removes all of them. The scalac nailgun JVMs held about
60% less heap after compiling, and across repeated compiles their jar
index grew about 40% less. Immutable inputs dropped from 1.9GB to 0.6GB
of disk. Cold compiles were slightly faster, and cached compiles are
unchanged.

A side-effect fix is that `deploy_jar` no longer gets duplicates of the
same jars via this mechanism, so its `duplicate_policy` outcomes behave
more as expected. Before, a jar could conflict with itself just because
two artifacts depended on it. That is covered in release notes.

Plugins reading a classpath entry digest or filenames directly no longer
s... (continued)

299 of 302 new or added lines in 17 files covered. (99.01%)

97894 of 105184 relevant lines covered (93.07%)

3.65 hits per line