pantsbuild / pants / 29631338772

push

github

feat(golang): add module level granularity for third party dependencies (#23499)

This PR adds a new opt-in model for go third party module dependencies,
which is one target per module instead of one target per package.

### Background

The problem today is that some dependencies, for example cloud provider
libraries, contain a lot of packages. Currently the go backend maps each
third party package to a generated target, and infers the entire
dependency graph between all packages across all third party modules. In
a repo with 800+ third party dependencies, this leads to **over 12000
third party package targets** that all go through a dependency inference
walk. This leads to a lot of engine overhead. This is especially painful
for CI when running `--changed-since` with `--changed-dependents`, since
every single CI run is hit by this at startup regardless if golang is
affected by the change.

In my case with a multi-language repo where Go is under-represented,
this inference is the majority cost of pants startup on all runs.

This improvement goes beyond just dependency inference, as shown in
benchmarks, but it was the main driver for the change.

### This PR

Adds a new option `[golang].third_party_target_granularity` with the
options `package` or `module`. `package` retains the current behavior of
generating targets per third party package. `module` only creates one
target per third party module.

I've added the two modes as moving to `module` has some theoretical
breaking cases since third party packages are not addressable with this
mode. So any explicit dependency references or a main-reference into a
third party package would no longer work.

I have instead added another field `main_import_path` to `go_binary`
which covers targeting a third party package as the main package even
when using `module` granularity. This field is also supported for
`package` mode.

### Implementation notes

Module targets are represented by the existing `go_third_party... (continued)

492 of 536 new or added lines in 19 files covered. (91.79%)

95217 of 102293 relevant lines covered (93.08%)

3.68 hits per line