• Home
  • Features
  • Pricing
  • Docs
  • Announcements
  • Sign In

pantsbuild / pants / 20147226056

11 Dec 2025 08:58PM UTC coverage: 78.827% (-1.5%) from 80.293%
20147226056

push

github

web-flow
Forwarded the `style` and `complete-platform` args from pants.toml to PEX (#22910)

## Context

After Apple switched to the `arm64` architecture, some package
publishers stopped releasing `x86_64` variants of their packages for
`darwin`. As a result, generating a universal lockfile now fails because
no single package version is compatible with both `x86_64` and `arm64`
on `darwin`.

The solution is to use the `--style` and `--complete-platform` flags
with PEX. For example:
```
pex3 lock create \
    --style strict \
    --complete-platform 3rdparty/platforms/manylinux_2_28_aarch64.json \
    --complete-platform 3rdparty/platforms/macosx_26_0_arm64.json \
    -r 3rdparty/python/requirements_pyarrow.txt \
    -o python-pyarrow.lock
```

See the Slack discussion here:
https://pantsbuild.slack.com/archives/C046T6T9U/p1760098582461759

## Reproduction

* `BUILD`
```
python_requirement(
    name="awswrangler",
    requirements=["awswrangler==3.12.1"],
    resolve="awswrangler",
)
```
* Run `pants generate-lockfiles --resolve=awswrangler` on macOS with an
`arm64` CPU
```
pip: ERROR: Cannot install awswrangler==3.12.1 because these package versions have conflicting dependencies.
pip: ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts
pip:  
pip:  The conflict is caused by:
pip:      awswrangler 3.12.1 depends on pyarrow<18.0.0 and >=8.0.0; sys_platform == "darwin" and platform_machine == "x86_64"
pip:      awswrangler 3.12.1 depends on pyarrow<21.0.0 and >=18.0.0; sys_platform != "darwin" or platform_machine != "x86_64"
pip:  
pip:  Additionally, some packages in these conflicts have no matching distributions available for your environment:
pip:      pyarrow
pip:  
pip:  To fix this you could try to:
pip:  1. loosen the range of package versions you've specified
pip:  2. remove package versions to allow pip to attempt to solve the dependency conflict
```

## Implementation
... (continued)

77 of 100 new or added lines in 6 files covered. (77.0%)

868 existing lines in 42 files now uncovered.

74471 of 94474 relevant lines covered (78.83%)

3.18 hits per line

Source File
Press 'n' to go to next uncovered line, 'b' for previous

37.8
/src/python/pants/backend/nfpm/util_rules/sandbox.py
1
# Copyright 2023 Pants project contributors (see CONTRIBUTORS.md).
2
# Licensed under the Apache License, Version 2.0 (see LICENSE).
3

4
from __future__ import annotations
4✔
5

6
import dataclasses
4✔
7
from dataclasses import dataclass
4✔
8
from enum import Enum, auto
4✔
9
from typing import cast
4✔
10

11
from pants.backend.nfpm.field_sets import NFPM_PACKAGE_FIELD_SET_TYPES, NfpmPackageFieldSet
4✔
12
from pants.backend.nfpm.fields.contents import (
4✔
13
    NfpmContentDirDstField,
14
    NfpmContentDstField,
15
    NfpmContentFileSourceField,
16
    NfpmContentSrcField,
17
    NfpmContentSymlinkDstField,
18
)
19
from pants.backend.nfpm.target_types import NfpmContentFile, NfpmPackageTarget
4✔
20
from pants.core.goals.package import (
4✔
21
    EnvironmentAwarePackageRequest,
22
    PackageFieldSet,
23
    TraverseIfNotPackageTarget,
24
    environment_aware_package,
25
)
26
from pants.core.util_rules.source_files import SourceFilesRequest, determine_source_files
4✔
27
from pants.core.util_rules.source_files import rules as source_files_rules
4✔
28
from pants.engine.fs import CreateDigest, DigestEntries, Directory, FileEntry, SymlinkEntry
4✔
29
from pants.engine.internals.graph import find_valid_field_sets, hydrate_sources
4✔
30
from pants.engine.internals.graph import transitive_targets as transitive_targets_get
4✔
31
from pants.engine.internals.native_engine import Digest, MergeDigests
4✔
32
from pants.engine.internals.selectors import concurrently
4✔
33
from pants.engine.intrinsics import create_digest, get_digest_entries, merge_digests
4✔
34
from pants.engine.rules import collect_rules, implicitly, rule
4✔
35
from pants.engine.target import (
4✔
36
    FieldSetsPerTargetRequest,
37
    GenerateSourcesRequest,
38
    HydrateSourcesRequest,
39
    SourcesField,
40
    Target,
41
    TransitiveTargets,
42
    TransitiveTargetsRequest,
43
)
44
from pants.engine.unions import UnionMembership
4✔
45
from pants.util.ordered_set import FrozenOrderedSet
4✔
46

47

48
class _DepCategory(Enum):
4✔
49
    ignore = auto()
4✔
50
    nfpm_content_from_dependency = auto()
4✔
51
    nfpm_content_from_source = auto()
4✔
52
    nfpm_package = auto()
4✔
53
    remaining = auto()
4✔
54

55
    @classmethod
4✔
56
    def for_target(
4✔
57
        cls,
58
        tgt: Target,
59
        field_set_type: type[NfpmPackageFieldSet],
60
    ) -> _DepCategory:
61
        # Assumption: this gets called with atomic targets not target generators. For example,
62
        # TransitiveTargets gets calculated AFTER target generation/expansion.
UNCOV
63
        if tgt.has_field(NfpmContentDirDstField) or tgt.has_field(NfpmContentSymlinkDstField):
×
64
            # NfpmContentDir and NfpmContentSymlink targets don't go in the sandbox.
65
            # They're only registered in the nfpm config.
UNCOV
66
            return _DepCategory.ignore
×
UNCOV
67
        if tgt.has_field(NfpmContentDstField):
×
68
            # an NfpmContentFile DOES need something in the sandbox
69

70
            # 'source' must be either None or a non-empty string.
71
            # If 'source' is None, the file comes from dependencies.
UNCOV
72
            if tgt[NfpmContentFileSourceField].value is None:
×
73
                # The file must be hydrated from one of the dependencies.
UNCOV
74
                return _DepCategory.nfpm_content_from_dependency
×
75
            # The file must be hydrated from the 'source' field
UNCOV
76
            return _DepCategory.nfpm_content_from_source
×
77

UNCOV
78
        for pkg_field_set_type in NFPM_PACKAGE_FIELD_SET_TYPES:
×
UNCOV
79
            if pkg_field_set_type.is_applicable(tgt):
×
80
                # we only respect nfpm package deps for the same packager
81
                # (For example, deb targets will ignore any deps on rpm targets)
UNCOV
82
                if pkg_field_set_type == field_set_type:
×
UNCOV
83
                    return _DepCategory.nfpm_package
×
UNCOV
84
                return _DepCategory.ignore
×
85

UNCOV
86
        return _DepCategory.remaining
×
87

88

89
@dataclass(frozen=True)
4✔
90
class _NfpmSortedDeps:
4✔
91
    nfpm_content_from_dependency_targets: tuple[NfpmContentFile, ...]
4✔
92
    nfpm_content_from_source_targets: tuple[NfpmContentFile, ...]
4✔
93
    nfpm_package_targets: tuple[NfpmPackageTarget, ...]
4✔
94
    package_targets: tuple[Target, ...]
4✔
95
    remaining_targets: tuple[Target, ...]
4✔
96

97
    @classmethod
4✔
98
    def sort(
4✔
99
        cls,
100
        field_set: NfpmPackageFieldSet,
101
        transitive_targets: TransitiveTargets,
102
        union_membership: UnionMembership,
103
    ) -> _NfpmSortedDeps:
104
        package_field_set_types = (
×
105
            FrozenOrderedSet(union_membership.get(PackageFieldSet)) - NFPM_PACKAGE_FIELD_SET_TYPES
106
        )
107

108
        nfpm_content_from_dependency_targets: list[NfpmContentFile] = []
×
109
        nfpm_content_from_source_targets: list[NfpmContentFile] = []
×
110
        nfpm_package_targets: list[NfpmPackageTarget] = []
×
111
        package_targets: list[Target] = []
×
112
        remaining_targets: list[Target] = []
×
113

114
        for tgt in transitive_targets.dependencies:
×
115
            category = _DepCategory.for_target(tgt, type(field_set))
×
116
            if category == _DepCategory.ignore:
×
117
                continue
×
118
            elif category == _DepCategory.nfpm_content_from_dependency:
×
119
                nfpm_content_from_dependency_targets.append(cast(NfpmContentFile, tgt))
×
120
            elif category == _DepCategory.nfpm_content_from_source:
×
121
                nfpm_content_from_source_targets.append(cast(NfpmContentFile, tgt))
×
122
            elif category == _DepCategory.nfpm_package:
×
123
                nfpm_package_targets.append(cast(NfpmPackageTarget, tgt))
×
124
            elif category == _DepCategory.remaining:
×
125
                is_package = False
×
126
                for field_set_type in package_field_set_types:
×
127
                    if field_set_type.is_applicable(tgt):
×
128
                        is_package = True
×
129
                        package_targets.append(tgt)
×
130
                        break
×
131
                if not is_package:
×
132
                    remaining_targets.append(tgt)
×
133
            else:
134
                raise ValueError(f"Please file a bug report--got unknown _DepCategory: {category}")
×
135

136
        return cls(
×
137
            nfpm_content_from_dependency_targets=tuple(nfpm_content_from_dependency_targets),
138
            nfpm_content_from_source_targets=tuple(nfpm_content_from_source_targets),
139
            nfpm_package_targets=tuple(nfpm_package_targets),
140
            package_targets=tuple(package_targets),
141
            remaining_targets=tuple(remaining_targets),
142
        )
143

144

145
@dataclass(frozen=True)
4✔
146
class NfpmContentSandboxRequest:
4✔
147
    field_set: NfpmPackageFieldSet
4✔
148

149

150
@dataclass(frozen=True)
4✔
151
class NfpmContentSandbox:
4✔
152
    digest: Digest
4✔
153

154

155
@rule
4✔
156
async def populate_nfpm_content_sandbox(
4✔
157
    request: NfpmContentSandboxRequest, union_membership: UnionMembership
158
) -> NfpmContentSandbox:
159
    transitive_targets = await transitive_targets_get(
×
160
        TransitiveTargetsRequest(
161
            [request.field_set.address],
162
            should_traverse_deps_predicate=TraverseIfNotPackageTarget(
163
                roots=[request.field_set.address],
164
                union_membership=union_membership,
165
            ),
166
        ),
167
        **implicitly(),
168
    )
169

170
    deps = _NfpmSortedDeps.sort(request.field_set, transitive_targets, union_membership)
×
171

172
    # 1. Build packages for deps that are (non-nfpm) Packages
173

174
    package_field_sets_per_target = await find_valid_field_sets(
×
175
        FieldSetsPerTargetRequest(PackageFieldSet, deps.package_targets), **implicitly()
176
    )
177
    packages = await concurrently(
×
178
        environment_aware_package(EnvironmentAwarePackageRequest(field_set))
179
        for field_set in package_field_sets_per_target.field_sets
180
    )
181

182
    # 2. Hydrate 'source' fields for nfpm_content_file targets.
183

184
    nfpm_content_source_fields_to_relocate: list[
×
185
        tuple[NfpmContentFileSourceField, NfpmContentSrcField]
186
    ] = []
187
    nfpm_content_source_fields: list[SourcesField] = []
×
188

189
    for nfpm_content_tgt in deps.nfpm_content_from_source_targets:
×
190
        source = nfpm_content_tgt[NfpmContentFileSourceField]
×
191
        src = nfpm_content_tgt[NfpmContentSrcField]
×
192
        # If 'src' is empty, it defaults to the content target's 'source'.
193
        if src.value and source.value != src.value:
×
194
            nfpm_content_source_fields_to_relocate.append((source, src))
×
195
        else:
196
            nfpm_content_source_fields.append(source)
×
197

198
    hydrated_sources_to_relocate = await concurrently(
×
199
        hydrate_sources(HydrateSourcesRequest(field), **implicitly())
200
        for field, _ in nfpm_content_source_fields_to_relocate
201
    )
202
    relocated_source_entries = await concurrently(
×
203
        get_digest_entries(hydrated.snapshot.digest) for hydrated in hydrated_sources_to_relocate
204
    )
205
    moved_entries: list[FileEntry | SymlinkEntry | Directory] = []
×
206
    digest_entries: DigestEntries
207
    for digest_entries, (source, src) in zip(
×
208
        relocated_source_entries, nfpm_content_source_fields_to_relocate
209
    ):
210
        for entry in digest_entries:
×
211
            if isinstance(entry, FileEntry) and entry.path == source.value:
×
212
                new_path = src.value
×
213
                if new_path is None:
×
214
                    raise ValueError("unexpected None")
×
215

216
                moved_entries.append(dataclasses.replace(entry, path=new_path))
×
217
            else:
218
                moved_entries.append(entry)
×
219

220
    nfpm_content_relocated_sources_digest, nfpm_content_sources = await concurrently(
×
221
        create_digest(CreateDigest(moved_entries)),
222
        # nfpm_content_file sources are simply files -- no codegen required.
223
        # anything more involved (like downloading http_source()) should use 'dependencies' instead
224
        # (for example, depend on a 'file(source=http_source(...))' target to download something).
225
        determine_source_files(SourceFilesRequest(nfpm_content_source_fields)),
226
    )
227

228
    # 3. Hydrate sources from 'dependencies' fields for nfpm_content_file targets,
229
    # which should all be accounted for in the transitive targets in deps.remaining_targets.
230

231
    # This involves doing as much codegen as possible (based on export_codegen goal).
232
    codegen_inputs_to_outputs = [
×
233
        (req.input, req.output) for req in union_membership.get(GenerateSourcesRequest)
234
    ]
235
    codegen_sources_fields_with_output = []
×
236
    for tgt in deps.remaining_targets:
×
237
        if not tgt.has_field(SourcesField):
×
238
            continue
×
239
        sources_field = tgt[SourcesField]
×
240
        found = False
×
241
        for input_type, output_type in codegen_inputs_to_outputs:
×
242
            if isinstance(sources_field, input_type):
×
243
                codegen_sources_fields_with_output.append((sources_field, output_type))
×
244
                found = True
×
245
        # make sure to include anything where codegen doesn't apply
246
        if not found:
×
247
            codegen_sources_fields_with_output.append((sources_field, type(sources_field)))
×
248
    hydrated_dep_sources = await concurrently(
×
249
        hydrate_sources(
250
            HydrateSourcesRequest(
251
                sources,
252
                for_sources_types=(output_type,),
253
                enable_codegen=True,
254
            ),
255
            **implicitly(),
256
        )
257
        for sources, output_type in codegen_sources_fields_with_output
258
    )
259

260
    # I would love to cleanly support relocations to 'src' from 'dependencies' files.
261
    # But, I don't see any clean approaches to identify which package, generated file,
262
    # or workspace file needs to be relocated to 'src'.
263
    # TODO: handle relocations from 'dependencies' to 'src'
264
    #       deps.nfpm_content_from_dependency_targets
265

266
    # This should include at least all files in 'src' fields of nfpm_content_file targets.
267
    # Other dependency files aren't required since nFPM will ignore anything not configured.
268
    sandbox_digest = await merge_digests(
×
269
        MergeDigests(
270
            [
271
                *(package.digest for package in packages),
272
                # nfpm_content_file 'src' from 'source' field
273
                nfpm_content_relocated_sources_digest,
274
                nfpm_content_sources.snapshot.digest,
275
                # nfpm_content_file 'src' from 'dependencies' field
276
                *(hydrated.snapshot.digest for hydrated in hydrated_dep_sources),
277
            ]
278
        )
279
    )
280

281
    return NfpmContentSandbox(sandbox_digest)
×
282

283

284
def rules():
4✔
285
    return [
4✔
286
        *source_files_rules(),
287
        *collect_rules(),
288
    ]
STATUS · Troubleshooting · Open an Issue · Sales · Support · CAREERS · ENTERPRISE · START FREE · SCHEDULE DEMO
ANNOUNCEMENTS · TWITTER · TOS & SLA · Supported CI Services · What's a CI service? · Automated Testing

© 2026 Coveralls, Inc