• Home
  • Features
  • Pricing
  • Docs
  • Announcements
  • Sign In

pantsbuild / pants / 19529437518

20 Nov 2025 07:44AM UTC coverage: 78.884% (-1.4%) from 80.302%
19529437518

push

github

web-flow
nfpm.native_libs: Add RPM package depends from packaged pex_binaries (#22899)

## PR Series Overview

This is the second in a series of PRs that introduces a new backend:
`pants.backend.npm.native_libs`
Initially, the backend will be available as:
`pants.backend.experimental.nfpm.native_libs`

I proposed this new backend (originally named `bindeps`) in discussion
#22396.

This backend will inspect ELF bin/lib files (like `lib*.so`) in packaged
contents (for this PR series, only in `pex_binary` targets) to identify
package dependency metadata and inject that metadata on the relevant
`nfpm_deb_package` or `nfpm_rpm_package` targets. Effectively, it will
provide an approximation of these native packager features:
- `rpm`: `rpmdeps` + `elfdeps`
- `deb`: `dh_shlibdeps` + `dpkg-shlibdeps` (These substitute
`${shlibs:Depends}` in debian control files have)

### Goal: Host-agnostic package builds

This pants backend is designed to be host-agnostic, like
[nFPM](https://nfpm.goreleaser.com/).

Native packaging tools are often restricted to a single release of a
single distro. Unlike native package builders, this new pants backend
does not use any of those distro-specific or distro-release-specific
utilities or local package databases. This new backend should be able to
run (help with building deb and rpm packages) anywhere that pants can
run (MacOS, rpm linux distros, deb linux distros, other linux distros,
docker, ...).

### Previous PRs in series

- #22873

## PR Overview

This PR adds rules in `nfpm.native_libs` to add package dependency
metadata to `nfpm_rpm_package`. The 2 new rules are:

- `inject_native_libs_dependencies_in_package_fields`:

    - An implementation of the polymorphic rule `inject_nfpm_package_fields`.
      This rule is low priority (`priority = 2`) so that in-repo plugins can
      override/augment what it injects. (See #22864)

    - Rule logic overview:
        - find any pex_binaries that will be packaged in an `nfpm_rpm_package`
   ... (continued)

96 of 118 new or added lines in 3 files covered. (81.36%)

910 existing lines in 53 files now uncovered.

73897 of 93678 relevant lines covered (78.88%)

3.21 hits per line

Source File
Press 'n' to go to next uncovered line, 'b' for previous

52.99
/src/python/pants/core/util_rules/system_binaries.py
1
# Copyright 2022 Pants project contributors (see CONTRIBUTORS.md).
2
# Licensed under the Apache License, Version 2.0 (see LICENSE).
3

4
from __future__ import annotations
11✔
5

6
import dataclasses
11✔
7
import hashlib
11✔
8
import logging
11✔
9
import os
11✔
10
import shlex
11✔
11
import subprocess
11✔
12
from collections.abc import Iterable, Mapping, Sequence
11✔
13
from dataclasses import dataclass
11✔
14
from enum import Enum
11✔
15
from itertools import groupby
11✔
16
from textwrap import dedent  # noqa: PNT20
11✔
17
from typing import Self
11✔
18

19
from pants.core.environments.target_types import EnvironmentTarget
11✔
20
from pants.core.subsystems import python_bootstrap
11✔
21
from pants.engine.collection import DeduplicatedCollection
11✔
22
from pants.engine.engine_aware import EngineAwareReturnType
11✔
23
from pants.engine.fs import CreateDigest, FileContent, PathMetadataRequest
11✔
24
from pants.engine.internals.native_engine import Digest, PathMetadataKind, PathNamespace
11✔
25
from pants.engine.internals.selectors import concurrently
11✔
26
from pants.engine.intrinsics import create_digest, execute_process, path_metadata_request
11✔
27
from pants.engine.platform import Platform
11✔
28
from pants.engine.process import Process, execute_process_or_raise
11✔
29
from pants.engine.rules import collect_rules, implicitly, rule
11✔
30
from pants.option.option_types import StrListOption
11✔
31
from pants.option.subsystem import Subsystem
11✔
32
from pants.util.frozendict import FrozenDict
11✔
33
from pants.util.logging import LogLevel
11✔
34
from pants.util.memo import memoized_property
11✔
35
from pants.util.ordered_set import OrderedSet
11✔
36
from pants.util.strutil import pluralize, softwrap
11✔
37

38
logger = logging.getLogger(__name__)
11✔
39

40

41
class SystemBinariesSubsystem(Subsystem):
11✔
42
    options_scope = "system-binaries"
11✔
43
    help = "System binaries related settings."
11✔
44

45
    class EnvironmentAware(Subsystem.EnvironmentAware):
11✔
46
        env_vars_used_by_options = ("PATH",)
11✔
47

48
        _DEFAULT_SEARCH_PATHS = ("/usr/bin", "/bin", "/usr/local/bin", "/opt/homebrew/bin")
11✔
49

50
        _system_binary_paths = StrListOption(
11✔
51
            default=[*_DEFAULT_SEARCH_PATHS],
52
            help=softwrap(
53
                """
54
                The PATH value that will searched for executables.
55

56
                The special string `"<PATH>"` will expand to the contents of the PATH env var.
57
                """
58
            ),
59
        )
60

61
        @memoized_property
11✔
62
        def system_binary_paths(self) -> SearchPath:
11✔
63
            def iter_path_entries():
×
64
                for entry in self._system_binary_paths:
×
65
                    if entry == "<PATH>":
×
66
                        path = self._options_env.get("PATH")
×
67
                        if path:
×
68
                            for prefix in path.split(os.pathsep):
×
69
                                if prefix.startswith("$") or prefix.startswith("~"):
×
70
                                    logger.warning(
×
71
                                        f"Ignored unexpanded path prefix `{prefix}` in the `PATH` environment variable while processing the `<PATH>` marker from the `[system-binaries].system_binary_paths` option. Please check the value of the `PATH` environment variable in your shell."
72
                                    )
73
                                else:
74
                                    yield prefix
×
75
                    else:
76
                        yield entry
×
77

78
            return SearchPath(iter_path_entries())
×
79

80

81
# -------------------------------------------------------------------------------------------
82
# `BinaryPath` types
83
# -------------------------------------------------------------------------------------------
84

85

86
@dataclass(frozen=True)
11✔
87
class BinaryPath:
11✔
88
    path: str
11✔
89
    fingerprint: str
11✔
90

91
    def __init__(self, path: str, fingerprint: str | None = None) -> None:
11✔
92
        object.__setattr__(self, "path", path)
2✔
93
        object.__setattr__(
2✔
94
            self, "fingerprint", self._fingerprint() if fingerprint is None else fingerprint
95
        )
96

97
    @staticmethod
11✔
98
    def _fingerprint(content: bytes | bytearray | memoryview | None = None) -> str:
11✔
99
        hasher = hashlib.sha256() if content is None else hashlib.sha256(content)
2✔
100
        return hasher.hexdigest()
2✔
101

102
    @classmethod
11✔
103
    def fingerprinted(
11✔
104
        cls, path: str, representative_content: bytes | bytearray | memoryview
105
    ) -> Self:
106
        return cls(path, fingerprint=cls._fingerprint(representative_content))
×
107

108

109
@dataclass(unsafe_hash=True)
11✔
110
class BinaryPathTest:
11✔
111
    args: tuple[str, ...]
11✔
112
    fingerprint_stdout: bool
11✔
113

114
    def __init__(self, args: Iterable[str], fingerprint_stdout: bool = True) -> None:
11✔
115
        object.__setattr__(self, "args", tuple(args))
1✔
116
        object.__setattr__(self, "fingerprint_stdout", fingerprint_stdout)
1✔
117

118

119
class SearchPath(DeduplicatedCollection[str]):
11✔
120
    """The search path for binaries; i.e.: the $PATH."""
121

122

123
@dataclass(unsafe_hash=True)
11✔
124
class BinaryPathRequest:
11✔
125
    """Request to find a binary of a given name.
126

127
    If `check_file_entries` is `True` a BinaryPathRequest will consider any entries in the
128
    `search_path` that are file paths in addition to traditional directory paths.
129

130
    If a `test` is specified all binaries that are found will be executed with the test args and
131
    only those binaries whose test executions exit with return code 0 will be retained.
132
    Additionally, if test execution includes stdout content, that will be used to fingerprint the
133
    binary path so that upgrades and downgrades can be detected. A reasonable test for many programs
134
    might be `BinaryPathTest(args=["--version"])` since it will both ensure the program runs and
135
    also produce stdout text that changes upon upgrade or downgrade of the binary at the discovered
136
    path.
137
    """
138

139
    search_path: SearchPath
11✔
140
    binary_name: str
11✔
141
    check_file_entries: bool
11✔
142
    test: BinaryPathTest | None
11✔
143

144
    def __init__(
11✔
145
        self,
146
        *,
147
        search_path: Iterable[str],
148
        binary_name: str,
149
        check_file_entries: bool = False,
150
        test: BinaryPathTest | None = None,
151
    ) -> None:
152
        object.__setattr__(self, "search_path", SearchPath(search_path))
2✔
153
        object.__setattr__(self, "binary_name", binary_name)
2✔
154
        object.__setattr__(self, "check_file_entries", check_file_entries)
2✔
155
        object.__setattr__(self, "test", test)
2✔
156

157

158
@dataclass(frozen=True)
11✔
159
class BinaryPaths(EngineAwareReturnType):
11✔
160
    binary_name: str
11✔
161
    paths: tuple[BinaryPath, ...]
11✔
162

163
    def __init__(self, binary_name: str, paths: Iterable[BinaryPath] | None = None):
11✔
164
        object.__setattr__(self, "binary_name", binary_name)
2✔
165
        object.__setattr__(self, "paths", tuple(OrderedSet(paths) if paths else ()))
2✔
166

167
    def message(self) -> str:
11✔
168
        if not self.paths:
×
169
            return f"failed to find {self.binary_name}"
×
170
        found_msg = f"found {self.binary_name} at {self.paths[0]}"
×
171
        if len(self.paths) > 1:
×
172
            found_msg = f"{found_msg} and {pluralize(len(self.paths) - 1, 'other location')}"
×
173
        return found_msg
×
174

175
    @property
11✔
176
    def first_path(self) -> BinaryPath | None:
11✔
177
        """Return the first path to the binary that was discovered, if any."""
178
        return next(iter(self.paths), None)
2✔
179

180
    def first_path_or_raise(self, request: BinaryPathRequest, *, rationale: str) -> BinaryPath:
11✔
181
        """Return the first path to the binary that was discovered, if any."""
182
        first_path = self.first_path
2✔
183
        if not first_path:
2✔
184
            raise BinaryNotFoundError.from_request(request, rationale=rationale)
2✔
185
        return first_path
2✔
186

187

188
class BinaryNotFoundError(EnvironmentError):
11✔
189
    @classmethod
11✔
190
    def from_request(
11✔
191
        cls,
192
        request: BinaryPathRequest,
193
        *,
194
        rationale: str | None = None,
195
        alternative_solution: str | None = None,
196
    ) -> BinaryNotFoundError:
197
        """When no binary is found via `BinaryPaths`, and it is not recoverable.
198

199
        :param rationale: A short description of why this binary is needed, e.g.
200
            "download the tools Pants needs" or "run Python programs".
201
        :param alternative_solution: A description of what else users can do to fix the issue,
202
            beyond installing the program. For example, "Alternatively, you can set the option
203
            `--python-bootstrap-search-path` to change the paths searched."
204
        """
205
        msg = softwrap(
2✔
206
            f"""
207
            Cannot find `{request.binary_name}` on `{sorted(request.search_path)}`.
208
            Please ensure that it is installed
209
            """
210
        )
211
        msg += f" so that Pants can {rationale}." if rationale else "."
2✔
212
        if alternative_solution:
2✔
213
            msg += f"\n\n{alternative_solution}"
×
214
        return BinaryNotFoundError(msg)
2✔
215

216

217
# -------------------------------------------------------------------------------------------
218
# Binary shims
219
# Creates a Digest with a shim for each requested binary in a directory suitable for PATH.
220
# -------------------------------------------------------------------------------------------
221

222

223
@dataclass(frozen=True)
11✔
224
class BinaryShimsRequest:
11✔
225
    """Request to create shims for one or more system binaries."""
226

227
    rationale: str = dataclasses.field(compare=False)
11✔
228

229
    # Create shims for provided binary paths
230
    paths: tuple[BinaryPath, ...] = tuple()
11✔
231

232
    # Create shims for the provided binary names after looking up the paths.
233
    requests: tuple[BinaryPathRequest, ...] = tuple()
11✔
234

235
    @classmethod
11✔
236
    def for_binaries(
11✔
237
        cls, *names: str, rationale: str, search_path: Sequence[str]
238
    ) -> BinaryShimsRequest:
UNCOV
239
        return cls(
×
240
            requests=tuple(
241
                BinaryPathRequest(binary_name=binary_name, search_path=search_path)
242
                for binary_name in names
243
            ),
244
            rationale=rationale,
245
        )
246

247
    @classmethod
11✔
248
    def for_paths(
11✔
249
        cls,
250
        *paths: BinaryPath,
251
        rationale: str,
252
    ) -> BinaryShimsRequest:
253
        # Remove any duplicates (which may result if the caller merges `BinaryPath` instances from multiple sources)
254
        # and also sort to ensure a stable order for better caching.
255
        paths = tuple(sorted(set(paths), key=lambda bp: bp.path))
2✔
256

257
        # Then ensure that there are no duplicate paths with mismatched content.
258
        duplicate_paths = set()
2✔
259
        for path, group in groupby(paths, key=lambda x: x.path):
2✔
260
            if len(list(group)) > 1:
2✔
UNCOV
261
                duplicate_paths.add(path)
×
262
        if duplicate_paths:
2✔
UNCOV
263
            raise ValueError(
×
264
                "Detected duplicate paths with mismatched content at paths: "
265
                f"{', '.join(sorted(duplicate_paths))}"
266
            )
267

268
        return cls(
2✔
269
            paths=paths,
270
            rationale=rationale,
271
        )
272

273

274
@dataclass(frozen=True)
11✔
275
class BinaryShims:
11✔
276
    """The shims created for a BinaryShimsRequest are placed in `bin_directory` of the `digest`.
277

278
    The purpose of these shims is so that a Process may be executed with `immutable_input_digests`
279
    provided to the `Process`, and `path_component` included in its `PATH` environment variable.
280

281
    The alternative is to add the directories hosting the binaries directly, but that opens up for
282
    many more unrelated binaries to also be executable from PATH, leaking into the sandbox
283
    unnecessarily.
284
    """
285

286
    digest: Digest
11✔
287
    cache_name: str
11✔
288

289
    @property
11✔
290
    def immutable_input_digests(self) -> Mapping[str, Digest]:
11✔
291
        return FrozenDict({self.cache_name: self.digest})
2✔
292

293
    @property
11✔
294
    def path_component(self) -> str:
11✔
295
        return os.path.join("{chroot}", self.cache_name)
2✔
296

297

298
# -------------------------------------------------------------------------------------------
299
# Binaries
300
# -------------------------------------------------------------------------------------------
301

302

303
class BashBinary(BinaryPath):
11✔
304
    """The `bash` binary."""
305

306
    DEFAULT_SEARCH_PATH = SearchPath(("/usr/bin", "/bin", "/usr/local/bin"))
11✔
307

308

309
# Note that updating this will impact the `archive` target defined in `core/target_types.py`.
310
class ArchiveFormat(Enum):
11✔
311
    TAR = "tar"
11✔
312
    TGZ = "tar.gz"
11✔
313
    TBZ2 = "tar.bz2"
11✔
314
    TXZ = "tar.xz"
11✔
315
    ZIP = "zip"
11✔
316

317

318
class ZipBinary(BinaryPath):
11✔
319
    pass
11✔
320

321

322
class UnzipBinary(BinaryPath):
11✔
323
    def extract_archive_argv(self, archive_path: str, extract_path: str) -> tuple[str, ...]:
11✔
324
        # Note that the `output_dir` does not need to already exist.
325
        # The caller should validate that it's a valid `.zip` file.
326
        return (self.path, archive_path, "-d", extract_path)
×
327

328

329
@dataclass(frozen=True)
11✔
330
class TarBinary(BinaryPath):
11✔
331
    platform: Platform
11✔
332

333
    def create_archive_argv(
11✔
334
        self,
335
        output_filename: str,
336
        tar_format: ArchiveFormat,
337
        *,
338
        input_files: Sequence[str] = (),
339
        input_file_list_filename: str | None = None,
340
    ) -> tuple[str, ...]:
341
        # Note that the parent directory for the output_filename must already exist.
342
        #
343
        # We do not use `-a` (auto-set compression) because it does not work with older tar
344
        # versions. Not all tar implementations will support these compression formats - in that
345
        # case, the user will need to choose a different format.
346
        compression = {ArchiveFormat.TGZ: "z", ArchiveFormat.TBZ2: "j", ArchiveFormat.TXZ: "J"}.get(
×
347
            tar_format, ""
348
        )
349

350
        files_from = ("--files-from", input_file_list_filename) if input_file_list_filename else ()
×
351
        return (self.path, f"c{compression}f", output_filename, *input_files) + files_from
×
352

353
    def extract_archive_argv(
11✔
354
        self, archive_path: str, extract_path: str, *, archive_suffix: str
355
    ) -> tuple[str, ...]:
356
        # Note that the `output_dir` must already exist.
357
        # The caller should validate that it's a valid `.tar` file.
358
        prog_args = (
×
359
            ("-Ilz4",) if archive_suffix == ".tar.lz4" and not self.platform.is_macos else ()
360
        )
361
        return (self.path, *prog_args, "-xf", archive_path, "-C", extract_path)
×
362

363

364
class AwkBinary(BinaryPath):
11✔
365
    pass
11✔
366

367

368
class Base64Binary(BinaryPath):
11✔
369
    pass
11✔
370

371

372
class BasenameBinary(BinaryPath):
11✔
373
    pass
11✔
374

375

376
class Bzip2Binary(BinaryPath):
11✔
377
    pass
11✔
378

379

380
class Bzip3Binary(BinaryPath):
11✔
381
    pass
11✔
382

383

384
class CatBinary(BinaryPath):
11✔
385
    pass
11✔
386

387

388
class ChmodBinary(BinaryPath):
11✔
389
    pass
11✔
390

391

392
class CksumBinary(BinaryPath):
11✔
393
    pass
11✔
394

395

396
class CpBinary(BinaryPath):
11✔
397
    pass
11✔
398

399

400
class CutBinary(BinaryPath):
11✔
401
    pass
11✔
402

403

404
class DateBinary(BinaryPath):
11✔
405
    pass
11✔
406

407

408
class DdBinary(BinaryPath):
11✔
409
    pass
11✔
410

411

412
class DfBinary(BinaryPath):
11✔
413
    pass
11✔
414

415

416
class DiffBinary(BinaryPath):
11✔
417
    pass
11✔
418

419

420
class DirnameBinary(BinaryPath):
11✔
421
    pass
11✔
422

423

424
class DuBinary(BinaryPath):
11✔
425
    pass
11✔
426

427

428
class ExprBinary(BinaryPath):
11✔
429
    pass
11✔
430

431

432
class FindBinary(BinaryPath):
11✔
433
    pass
11✔
434

435

436
class GetentBinary(BinaryPath):
11✔
437
    pass
11✔
438

439

440
class GpgBinary(BinaryPath):
11✔
441
    pass
11✔
442

443

444
class GzipBinary(BinaryPath):
11✔
445
    pass
11✔
446

447

448
class HeadBinary(BinaryPath):
11✔
449
    pass
11✔
450

451

452
class IdBinary(BinaryPath):
11✔
453
    pass
11✔
454

455

456
class LnBinary(BinaryPath):
11✔
457
    pass
11✔
458

459

460
class Lz4Binary(BinaryPath):
11✔
461
    pass
11✔
462

463

464
class LzopBinary(BinaryPath):
11✔
465
    pass
11✔
466

467

468
class Md5sumBinary(BinaryPath):
11✔
469
    pass
11✔
470

471

472
class MkdirBinary(BinaryPath):
11✔
473
    pass
11✔
474

475

476
class MktempBinary(BinaryPath):
11✔
477
    pass
11✔
478

479

480
class MvBinary(BinaryPath):
11✔
481
    pass
11✔
482

483

484
class OpenBinary(BinaryPath):
11✔
485
    pass
11✔
486

487

488
class PwdBinary(BinaryPath):
11✔
489
    pass
11✔
490

491

492
class ReadlinkBinary(BinaryPath):
11✔
493
    pass
11✔
494

495

496
class RmBinary(BinaryPath):
11✔
497
    pass
11✔
498

499

500
class SedBinary(BinaryPath):
11✔
501
    pass
11✔
502

503

504
class ShBinary(BinaryPath):
11✔
505
    pass
11✔
506

507

508
class ShasumBinary(BinaryPath):
11✔
509
    pass
11✔
510

511

512
class SortBinary(BinaryPath):
11✔
513
    pass
11✔
514

515

516
class TailBinary(BinaryPath):
11✔
517
    pass
11✔
518

519

520
class TestBinary(BinaryPath):
11✔
521
    pass
11✔
522

523

524
class TouchBinary(BinaryPath):
11✔
525
    pass
11✔
526

527

528
class TrBinary(BinaryPath):
11✔
529
    pass
11✔
530

531

532
class WcBinary(BinaryPath):
11✔
533
    pass
11✔
534

535

536
class XargsBinary(BinaryPath):
11✔
537
    pass
11✔
538

539

540
class XzBinary(BinaryPath):
11✔
541
    pass
11✔
542

543

544
class ZstdBinary(BinaryPath):
11✔
545
    pass
11✔
546

547

548
class GitBinaryException(Exception):
11✔
549
    pass
11✔
550

551

552
class GitBinary(BinaryPath):
11✔
553
    def _invoke_unsandboxed(self, cmd: list[str]) -> bytes:
11✔
554
        """Invoke the given git command, _without_ the sandboxing provided by the `Process` API.
555

556
        This API is for internal use only: users should prefer to consume methods of the
557
        `GitWorktree` class.
558
        """
UNCOV
559
        cmd = [self.path, *cmd]
×
560

UNCOV
561
        self._log_call(cmd)
×
562

UNCOV
563
        try:
×
UNCOV
564
            process = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
×
UNCOV
565
        except OSError as e:
×
566
            # Binary DNE or is not executable
UNCOV
567
            cmd_str = shlex.join(cmd)
×
UNCOV
568
            raise GitBinaryException(f"Failed to execute command {cmd_str}: {e!r}")
×
UNCOV
569
        out, err = process.communicate()
×
570

UNCOV
571
        self._check_result(cmd, process.returncode, err.decode())
×
572

UNCOV
573
        return out.strip()
×
574

575
    def _check_result(self, cmd: list[str], result: int, failure_msg: str | None = None) -> None:
11✔
576
        # git diff --exit-code exits with 1 if there were differences.
UNCOV
577
        if result != 0 and (result != 1 or "diff" not in cmd):
×
UNCOV
578
            cmd_str = shlex.join(cmd)
×
UNCOV
579
            raise GitBinaryException(failure_msg or f"{cmd_str} failed with exit code {result}")
×
580

581
    def _log_call(self, cmd: Iterable[str]) -> None:
11✔
UNCOV
582
        logger.debug("Executing: " + " ".join(cmd))
×
583

584

585
# -------------------------------------------------------------------------------------------
586
# Binaries Rules
587
# -------------------------------------------------------------------------------------------
588

589

590
@rule(desc="Finding the `bash` binary", level=LogLevel.DEBUG)
11✔
591
async def get_bash(system_binaries: SystemBinariesSubsystem.EnvironmentAware) -> BashBinary:
11✔
592
    search_path = system_binaries.system_binary_paths
×
593

594
    request = BinaryPathRequest(
×
595
        binary_name="bash",
596
        search_path=search_path,
597
        test=BinaryPathTest(args=["--version"]),
598
    )
599
    paths = await find_binary(request, **implicitly())
×
600
    first_path = paths.first_path
×
601
    if not first_path:
×
602
        raise BinaryNotFoundError.from_request(request)
×
603
    return BashBinary(first_path.path, first_path.fingerprint)
×
604

605

606
async def _find_candidate_paths_via_path_metadata_lookups(
11✔
607
    request: BinaryPathRequest,
608
) -> tuple[str, ...]:
609
    search_path = [os.path.abspath(path) for path in request.search_path]
×
610

611
    metadata_results = await concurrently(
×
612
        path_metadata_request(PathMetadataRequest(path=path, namespace=PathNamespace.SYSTEM))
613
        for path in search_path
614
    )
615

616
    found_paths_and_requests: list[str | PathMetadataRequest] = []
×
617
    file_metadata_requests: list[PathMetadataRequest] = []
×
618

619
    for metadata_result in metadata_results:
×
620
        metadata = metadata_result.metadata
×
621
        if not metadata:
×
622
            continue
×
623

624
        if metadata.kind in (PathMetadataKind.DIRECTORY, PathMetadataKind.SYMLINK):
×
625
            file_metadata_request = PathMetadataRequest(
×
626
                path=os.path.join(metadata.path, request.binary_name),
627
                namespace=PathNamespace.SYSTEM,
628
            )
629
            found_paths_and_requests.append(file_metadata_request)
×
630
            file_metadata_requests.append(file_metadata_request)
×
631

632
        elif metadata.kind == PathMetadataKind.FILE and request.check_file_entries:
×
633
            found_paths_and_requests.append(metadata.path)
×
634

635
    file_metadata_results = await concurrently(
×
636
        path_metadata_request(file_metadata_request)
637
        for file_metadata_request in file_metadata_requests
638
    )
639
    file_metadata_results_by_request = dict(zip(file_metadata_requests, file_metadata_results))
×
640

641
    found_paths: list[str] = []
×
642
    for found_path_or_request in found_paths_and_requests:
×
643
        if isinstance(found_path_or_request, str):
×
644
            found_paths.append(found_path_or_request)
×
645
        else:
646
            file_metadata_result = file_metadata_results_by_request[found_path_or_request]
×
647
            file_metadata = file_metadata_result.metadata
×
648
            if not file_metadata:
×
649
                continue
×
650
            found_paths.append(file_metadata.path)
×
651

652
    return tuple(found_paths)
×
653

654

655
async def _find_candidate_paths_via_subprocess_helper(
11✔
656
    request: BinaryPathRequest, env_target: EnvironmentTarget
657
) -> tuple[str, ...]:
658
    # If we are not already locating bash, recurse to locate bash to use it as an absolute path in
659
    # our shebang. This avoids mixing locations that we would search for bash into the search paths
660
    # of the request we are servicing.
661
    # TODO(#10769): Replace this script with a statically linked native binary so we don't
662
    #  depend on either /bin/bash being available on the Process host.
663
    if request.binary_name == "bash":
×
664
        shebang = "#!/usr/bin/env bash"
×
665
    else:
666
        bash = await get_bash(**implicitly())
×
667
        shebang = f"#!{bash.path}"
×
668

669
    # HACK: For workspace environments, the `find_binary.sh` will be mounted in the "chroot" directory
670
    # which is not the current directory (since the process will execute in the workspace). Thus,
671
    # adjust the path used as argv[0] to find the script.
672
    script_name = "find_binary.sh"
×
673
    sandbox_base_path = env_target.sandbox_base_path()
×
674
    script_exec_path = (
×
675
        f"./{script_name}" if not sandbox_base_path else f"{sandbox_base_path}/{script_name}"
676
    )
677

678
    script_header = dedent(
×
679
        f"""\
680
        {shebang}
681

682
        set -euox pipefail
683

684
        CHECK_FILE_ENTRIES={"1" if request.check_file_entries else ""}
685
        """
686
    )
687
    script_body = dedent(
×
688
        """\
689
        for path in ${PATH//:/ }; do
690
            if [[ -d "${path}" ]]; then
691
                # Handle traditional directory PATH element.
692
                maybe_exe="${path}/$1"
693
            elif [[ -n "${CHECK_FILE_ENTRIES}" ]]; then
694
                # Handle PATH elements that are filenames to allow for precise selection.
695
                maybe_exe="${path}"
696
            else
697
                maybe_exe=
698
            fi
699
            if [[ "$1" == "${maybe_exe##*/}" && -f "${maybe_exe}" && -x "${maybe_exe}" ]]
700
            then
701
                echo "${maybe_exe}"
702
            fi
703
        done
704
        """
705
    )
706
    script_content = script_header + script_body
×
707
    script_digest = await create_digest(
×
708
        CreateDigest([FileContent(script_name, script_content.encode(), is_executable=True)]),
709
    )
710

711
    # Some subtle notes about executing this script:
712
    #
713
    #  - We run the script with `ProcessResult` instead of `FallibleProcessResult` so that we
714
    #      can catch bugs in the script itself, given an earlier silent failure.
715
    search_path = os.pathsep.join(request.search_path)
×
716
    result = await execute_process_or_raise(
×
717
        **implicitly(
718
            Process(
719
                description=f"Searching for `{request.binary_name}` on PATH={search_path}",
720
                level=LogLevel.DEBUG,
721
                input_digest=script_digest,
722
                argv=[script_exec_path, request.binary_name],
723
                env={"PATH": search_path},
724
                cache_scope=env_target.executable_search_path_cache_scope(),
725
            ),
726
        )
727
    )
728
    return tuple(result.stdout.decode().splitlines())
×
729

730

731
@rule
11✔
732
async def find_binary(
11✔
733
    request: BinaryPathRequest,
734
    env_target: EnvironmentTarget,
735
) -> BinaryPaths:
736
    found_paths: tuple[str, ...]
737
    if env_target.can_access_local_system_paths:
×
738
        found_paths = await _find_candidate_paths_via_path_metadata_lookups(request)
×
739
    else:
740
        found_paths = await _find_candidate_paths_via_subprocess_helper(request, env_target)
×
741

742
    if not request.test:
×
743
        return BinaryPaths(
×
744
            binary_name=request.binary_name,
745
            paths=(BinaryPath(path) for path in found_paths),
746
        )
747

748
    results = await concurrently(
×
749
        execute_process(
750
            Process(
751
                description=f"Test binary {path}.",
752
                level=LogLevel.DEBUG,
753
                argv=[path, *request.test.args],
754
                # NB: Since a failure is a valid result for this script, we always cache it,
755
                # regardless of success or failure.
756
                cache_scope=env_target.executable_search_path_cache_scope(cache_failures=True),
757
            ),
758
            **implicitly(),
759
        )
760
        for path in found_paths
761
    )
762
    return BinaryPaths(
×
763
        binary_name=request.binary_name,
764
        paths=[
765
            (
766
                BinaryPath.fingerprinted(path, result.stdout)
767
                if request.test.fingerprint_stdout
768
                else BinaryPath(path, result.stdout.decode())
769
            )
770
            for path, result in zip(found_paths, results)
771
            if result.exit_code == 0
772
        ],
773
    )
774

775

776
@rule
11✔
777
async def create_binary_shims(
11✔
778
    binary_shims_request: BinaryShimsRequest,
779
    bash: BashBinary,
780
) -> BinaryShims:
781
    """Creates a bin directory with shims for all requested binaries.
782

783
    This can be provided to a `Process` as an `immutable_input_digest`, or can be merged into the
784
    input digest.
785
    """
786

787
    paths = binary_shims_request.paths
×
788
    requests = binary_shims_request.requests
×
789
    if requests:
×
790
        all_binary_paths = await concurrently(
×
791
            find_binary(request, **implicitly()) for request in requests
792
        )
793
        first_paths = tuple(
×
794
            binary_paths.first_path_or_raise(request, rationale=binary_shims_request.rationale)
795
            for binary_paths, request in zip(all_binary_paths, requests)
796
        )
797
        paths += first_paths
×
798

799
    def _create_shim(bash: str, binary: str) -> bytes:
×
800
        """The binary shim script to be placed in the output directory for the digest."""
801
        return dedent(
×
802
            f"""\
803
            #!{bash}
804
            exec "{binary}" "$@"
805
            """
806
        ).encode()
807

808
    scripts = [
×
809
        FileContent(
810
            os.path.basename(path.path), _create_shim(bash.path, path.path), is_executable=True
811
        )
812
        for path in paths
813
    ]
814

815
    digest = await create_digest(CreateDigest(scripts))
×
816
    cache_name = f"_binary_shims_{digest.fingerprint}"
×
817

818
    return BinaryShims(digest, cache_name)
×
819

820

821
@rule(desc="Finding the `awk` binary", level=LogLevel.DEBUG)
11✔
822
async def find_awk(system_binaries: SystemBinariesSubsystem.EnvironmentAware) -> AwkBinary:
11✔
823
    request = BinaryPathRequest(binary_name="awk", search_path=system_binaries.system_binary_paths)
×
824
    paths = await find_binary(request, **implicitly())
×
825
    first_path = paths.first_path_or_raise(request, rationale="awk file")
×
826
    return AwkBinary(first_path.path, first_path.fingerprint)
×
827

828

829
@rule(desc="Finding the `base64` binary", level=LogLevel.DEBUG)
11✔
830
async def find_base64(system_binaries: SystemBinariesSubsystem.EnvironmentAware) -> Base64Binary:
11✔
831
    request = BinaryPathRequest(
×
832
        binary_name="base64", search_path=system_binaries.system_binary_paths
833
    )
834
    paths = await find_binary(request, **implicitly())
×
835
    first_path = paths.first_path_or_raise(request, rationale="base64 file")
×
836
    return Base64Binary(first_path.path, first_path.fingerprint)
×
837

838

839
@rule(desc="Finding the `basename` binary", level=LogLevel.DEBUG)
11✔
840
async def find_basename(
11✔
841
    system_binaries: SystemBinariesSubsystem.EnvironmentAware,
842
) -> BasenameBinary:
843
    request = BinaryPathRequest(
×
844
        binary_name="basename", search_path=system_binaries.system_binary_paths
845
    )
846
    paths = await find_binary(request, **implicitly())
×
847
    first_path = paths.first_path_or_raise(request, rationale="basename file")
×
848
    return BasenameBinary(first_path.path, first_path.fingerprint)
×
849

850

851
@rule(desc="Finding the `bzip2` binary", level=LogLevel.DEBUG)
11✔
852
async def find_bzip2(system_binaries: SystemBinariesSubsystem.EnvironmentAware) -> Bzip2Binary:
11✔
853
    request = BinaryPathRequest(
×
854
        binary_name="bzip2", search_path=system_binaries.system_binary_paths
855
    )
856
    paths = await find_binary(request, **implicitly())
×
857
    first_path = paths.first_path_or_raise(request, rationale="bzip2 file")
×
858
    return Bzip2Binary(first_path.path, first_path.fingerprint)
×
859

860

861
@rule(desc="Finding the `bzip3` binary", level=LogLevel.DEBUG)
11✔
862
async def find_bzip3(system_binaries: SystemBinariesSubsystem.EnvironmentAware) -> Bzip3Binary:
11✔
863
    request = BinaryPathRequest(
×
864
        binary_name="bzip3", search_path=system_binaries.system_binary_paths
865
    )
866
    paths = await find_binary(request, **implicitly())
×
867
    first_path = paths.first_path_or_raise(request, rationale="bzip3 file")
×
868
    return Bzip3Binary(first_path.path, first_path.fingerprint)
×
869

870

871
@rule(desc="Finding the `cat` binary", level=LogLevel.DEBUG)
11✔
872
async def find_cat(system_binaries: SystemBinariesSubsystem.EnvironmentAware) -> CatBinary:
11✔
873
    request = BinaryPathRequest(binary_name="cat", search_path=system_binaries.system_binary_paths)
×
874
    paths = await find_binary(request, **implicitly())
×
875
    first_path = paths.first_path_or_raise(request, rationale="outputting content from files")
×
876
    return CatBinary(first_path.path, first_path.fingerprint)
×
877

878

879
@rule(desc="Finding the `chmod` binary", level=LogLevel.DEBUG)
11✔
880
async def find_chmod(system_binaries: SystemBinariesSubsystem.EnvironmentAware) -> ChmodBinary:
11✔
881
    request = BinaryPathRequest(
×
882
        binary_name="chmod", search_path=system_binaries.system_binary_paths
883
    )
884
    paths = await find_binary(request, **implicitly())
×
885
    first_path = paths.first_path_or_raise(
×
886
        request, rationale="change file modes or Access Control Lists"
887
    )
888
    return ChmodBinary(first_path.path, first_path.fingerprint)
×
889

890

891
@rule(desc="Finding the `cksum` binary", level=LogLevel.DEBUG)
11✔
892
async def find_cksum(system_binaries: SystemBinariesSubsystem.EnvironmentAware) -> CksumBinary:
11✔
893
    request = BinaryPathRequest(
×
894
        binary_name="cksum", search_path=system_binaries.system_binary_paths
895
    )
896
    paths = await find_binary(request, **implicitly())
×
897
    first_path = paths.first_path_or_raise(request, rationale="cksum file")
×
898
    return CksumBinary(first_path.path, first_path.fingerprint)
×
899

900

901
@rule(desc="Finding the `cp` binary", level=LogLevel.DEBUG)
11✔
902
async def find_cp(system_binaries: SystemBinariesSubsystem.EnvironmentAware) -> CpBinary:
11✔
903
    request = BinaryPathRequest(binary_name="cp", search_path=system_binaries.system_binary_paths)
×
904
    paths = await find_binary(request, **implicitly())
×
905
    first_path = paths.first_path_or_raise(request, rationale="copy files")
×
906
    return CpBinary(first_path.path, first_path.fingerprint)
×
907

908

909
@rule(desc="Finding the `cut` binary", level=LogLevel.DEBUG)
11✔
910
async def find_cut(system_binaries: SystemBinariesSubsystem.EnvironmentAware) -> CutBinary:
11✔
911
    request = BinaryPathRequest(binary_name="cut", search_path=system_binaries.system_binary_paths)
×
912
    paths = await find_binary(request, **implicitly())
×
913
    first_path = paths.first_path_or_raise(request, rationale="cut file")
×
914
    return CutBinary(first_path.path, first_path.fingerprint)
×
915

916

917
@rule(desc="Finding the `date` binary", level=LogLevel.DEBUG)
11✔
918
async def find_date(system_binaries: SystemBinariesSubsystem.EnvironmentAware) -> DateBinary:
11✔
919
    request = BinaryPathRequest(binary_name="date", search_path=system_binaries.system_binary_paths)
×
920
    paths = await find_binary(request, **implicitly())
×
921
    first_path = paths.first_path_or_raise(request, rationale="date file")
×
922
    return DateBinary(first_path.path, first_path.fingerprint)
×
923

924

925
@rule(desc="Finding the `dd` binary", level=LogLevel.DEBUG)
11✔
926
async def find_dd(system_binaries: SystemBinariesSubsystem.EnvironmentAware) -> DdBinary:
11✔
927
    request = BinaryPathRequest(binary_name="dd", search_path=system_binaries.system_binary_paths)
×
928
    paths = await find_binary(request, **implicitly())
×
929
    first_path = paths.first_path_or_raise(request, rationale="dd file")
×
930
    return DdBinary(first_path.path, first_path.fingerprint)
×
931

932

933
@rule(desc="Finding the `df` binary", level=LogLevel.DEBUG)
11✔
934
async def find_df(system_binaries: SystemBinariesSubsystem.EnvironmentAware) -> DfBinary:
11✔
935
    request = BinaryPathRequest(binary_name="df", search_path=system_binaries.system_binary_paths)
×
936
    paths = await find_binary(request, **implicitly())
×
937
    first_path = paths.first_path_or_raise(request, rationale="df file")
×
938
    return DfBinary(first_path.path, first_path.fingerprint)
×
939

940

941
@rule(desc="Finding the `diff` binary", level=LogLevel.DEBUG)
11✔
942
async def find_diff(system_binaries: SystemBinariesSubsystem.EnvironmentAware) -> DiffBinary:
11✔
943
    request = BinaryPathRequest(binary_name="diff", search_path=system_binaries.system_binary_paths)
×
944
    paths = await find_binary(request, **implicitly())
×
945
    first_path = paths.first_path_or_raise(request, rationale="compare files line by line")
×
946
    return DiffBinary(first_path.path, first_path.fingerprint)
×
947

948

949
@rule(desc="Finding the `dirname` binary", level=LogLevel.DEBUG)
11✔
950
async def find_dirname(system_binaries: SystemBinariesSubsystem.EnvironmentAware) -> DirnameBinary:
11✔
951
    request = BinaryPathRequest(
×
952
        binary_name="dirname", search_path=system_binaries.system_binary_paths
953
    )
954
    paths = await find_binary(request, **implicitly())
×
955
    first_path = paths.first_path_or_raise(request, rationale="dirname file")
×
956
    return DirnameBinary(first_path.path, first_path.fingerprint)
×
957

958

959
@rule(desc="Finding the `du` binary", level=LogLevel.DEBUG)
11✔
960
async def find_du(system_binaries: SystemBinariesSubsystem.EnvironmentAware) -> DuBinary:
11✔
961
    request = BinaryPathRequest(binary_name="du", search_path=system_binaries.system_binary_paths)
×
962
    paths = await find_binary(request, **implicitly())
×
963
    first_path = paths.first_path_or_raise(request, rationale="du file")
×
964
    return DuBinary(first_path.path, first_path.fingerprint)
×
965

966

967
@rule(desc="Finding the `expr` binary", level=LogLevel.DEBUG)
11✔
968
async def find_expr(system_binaries: SystemBinariesSubsystem.EnvironmentAware) -> ExprBinary:
11✔
969
    request = BinaryPathRequest(binary_name="expr", search_path=system_binaries.system_binary_paths)
×
970
    paths = await find_binary(request, **implicitly())
×
971
    first_path = paths.first_path_or_raise(request, rationale="expr file")
×
972
    return ExprBinary(first_path.path, first_path.fingerprint)
×
973

974

975
@rule(desc="Finding the `find` binary", level=LogLevel.DEBUG)
11✔
976
async def find_find(system_binaries: SystemBinariesSubsystem.EnvironmentAware) -> FindBinary:
11✔
977
    request = BinaryPathRequest(binary_name="find", search_path=system_binaries.system_binary_paths)
×
978
    paths = await find_binary(request, **implicitly())
×
979
    first_path = paths.first_path_or_raise(request, rationale="find file")
×
980
    return FindBinary(first_path.path, first_path.fingerprint)
×
981

982

983
@rule(desc="Finding the `git` binary", level=LogLevel.DEBUG)
11✔
984
async def find_git(system_binaries: SystemBinariesSubsystem.EnvironmentAware) -> GitBinary:
11✔
985
    request = BinaryPathRequest(binary_name="git", search_path=system_binaries.system_binary_paths)
×
986
    paths = await find_binary(request, **implicitly())
×
987
    first_path = paths.first_path_or_raise(
×
988
        request, rationale="track changes to files in your build environment"
989
    )
990
    return GitBinary(first_path.path, first_path.fingerprint)
×
991

992

993
@rule(desc="Finding the `getent` binary", level=LogLevel.DEBUG)
11✔
994
async def find_getent(system_binaries: SystemBinariesSubsystem.EnvironmentAware) -> GetentBinary:
11✔
995
    request = BinaryPathRequest(
×
996
        binary_name="getent", search_path=system_binaries.system_binary_paths
997
    )
998
    paths = await find_binary(request, **implicitly())
×
999
    first_path = paths.first_path_or_raise(request, rationale="getent file")
×
1000
    return GetentBinary(first_path.path, first_path.fingerprint)
×
1001

1002

1003
@rule(desc="Finding the `gpg` binary", level=LogLevel.DEBUG)
11✔
1004
async def find_gpg(system_binaries: SystemBinariesSubsystem.EnvironmentAware) -> GpgBinary:
11✔
1005
    request = BinaryPathRequest(binary_name="gpg", search_path=system_binaries.system_binary_paths)
×
1006
    paths = await find_binary(request, **implicitly())
×
1007
    first_path = paths.first_path_or_raise(request, rationale="gpg file")
×
1008
    return GpgBinary(first_path.path, first_path.fingerprint)
×
1009

1010

1011
@rule(desc="Finding the `gzip` binary", level=LogLevel.DEBUG)
11✔
1012
async def find_gzip(system_binaries: SystemBinariesSubsystem.EnvironmentAware) -> GzipBinary:
11✔
1013
    request = BinaryPathRequest(binary_name="gzip", search_path=system_binaries.system_binary_paths)
×
1014
    paths = await find_binary(request, **implicitly())
×
1015
    first_path = paths.first_path_or_raise(request, rationale="gzip file")
×
1016
    return GzipBinary(first_path.path, first_path.fingerprint)
×
1017

1018

1019
@rule(desc="Finding the `head` binary", level=LogLevel.DEBUG)
11✔
1020
async def find_head(system_binaries: SystemBinariesSubsystem.EnvironmentAware) -> HeadBinary:
11✔
1021
    request = BinaryPathRequest(binary_name="head", search_path=system_binaries.system_binary_paths)
×
1022
    paths = await find_binary(request, **implicitly())
×
1023
    first_path = paths.first_path_or_raise(request, rationale="head file")
×
1024
    return HeadBinary(first_path.path, first_path.fingerprint)
×
1025

1026

1027
@rule(desc="Finding the `id` binary", level=LogLevel.DEBUG)
11✔
1028
async def find_id(system_binaries: SystemBinariesSubsystem.EnvironmentAware) -> IdBinary:
11✔
1029
    request = BinaryPathRequest(binary_name="id", search_path=system_binaries.system_binary_paths)
×
1030
    paths = await find_binary(request, **implicitly())
×
1031
    first_path = paths.first_path_or_raise(request, rationale="id file")
×
1032
    return IdBinary(first_path.path, first_path.fingerprint)
×
1033

1034

1035
@rule(desc="Finding the `ln` binary", level=LogLevel.DEBUG)
11✔
1036
async def find_ln(system_binaries: SystemBinariesSubsystem.EnvironmentAware) -> LnBinary:
11✔
1037
    request = BinaryPathRequest(binary_name="ln", search_path=system_binaries.system_binary_paths)
×
1038
    paths = await find_binary(request, **implicitly())
×
1039
    first_path = paths.first_path_or_raise(request, rationale="link files")
×
1040
    return LnBinary(first_path.path, first_path.fingerprint)
×
1041

1042

1043
@rule(desc="Finding the `lz4` binary", level=LogLevel.DEBUG)
11✔
1044
async def find_lz4(system_binaries: SystemBinariesSubsystem.EnvironmentAware) -> Lz4Binary:
11✔
1045
    request = BinaryPathRequest(binary_name="lz4", search_path=system_binaries.system_binary_paths)
×
1046
    paths = await find_binary(request, **implicitly())
×
1047
    first_path = paths.first_path_or_raise(request, rationale="lz4 file")
×
1048
    return Lz4Binary(first_path.path, first_path.fingerprint)
×
1049

1050

1051
@rule(desc="Finding the `lzop` binary", level=LogLevel.DEBUG)
11✔
1052
async def find_lzop(system_binaries: SystemBinariesSubsystem.EnvironmentAware) -> LzopBinary:
11✔
1053
    request = BinaryPathRequest(binary_name="lzop", search_path=system_binaries.system_binary_paths)
×
1054
    paths = await find_binary(request, **implicitly())
×
1055
    first_path = paths.first_path_or_raise(request, rationale="lzop file")
×
1056
    return LzopBinary(first_path.path, first_path.fingerprint)
×
1057

1058

1059
@rule(desc="Finding the `md5sum` binary", level=LogLevel.DEBUG)
11✔
1060
async def find_md5sum(system_binaries: SystemBinariesSubsystem.EnvironmentAware) -> Md5sumBinary:
11✔
1061
    request = BinaryPathRequest(
×
1062
        binary_name="md5sum", search_path=system_binaries.system_binary_paths
1063
    )
1064
    paths = await find_binary(request, **implicitly())
×
1065
    first_path = paths.first_path_or_raise(request, rationale="md5sum file")
×
1066
    return Md5sumBinary(first_path.path, first_path.fingerprint)
×
1067

1068

1069
@rule(desc="Finding the `mkdir` binary", level=LogLevel.DEBUG)
11✔
1070
async def find_mkdir(system_binaries: SystemBinariesSubsystem.EnvironmentAware) -> MkdirBinary:
11✔
1071
    request = BinaryPathRequest(
×
1072
        binary_name="mkdir", search_path=system_binaries.system_binary_paths
1073
    )
1074
    paths = await find_binary(request, **implicitly())
×
1075
    first_path = paths.first_path_or_raise(request, rationale="create directories")
×
1076
    return MkdirBinary(first_path.path, first_path.fingerprint)
×
1077

1078

1079
@rule(desc="Finding the `mktempt` binary", level=LogLevel.DEBUG)
11✔
1080
async def find_mktemp(system_binaries: SystemBinariesSubsystem.EnvironmentAware) -> MktempBinary:
11✔
1081
    request = BinaryPathRequest(
×
1082
        binary_name="mktemp", search_path=system_binaries.system_binary_paths
1083
    )
1084
    paths = await find_binary(request, **implicitly())
×
1085
    first_path = paths.first_path_or_raise(request, rationale="create temporary files/directories")
×
1086
    return MktempBinary(first_path.path, first_path.fingerprint)
×
1087

1088

1089
@rule(desc="Finding the `mv` binary", level=LogLevel.DEBUG)
11✔
1090
async def find_mv(system_binaries: SystemBinariesSubsystem.EnvironmentAware) -> MvBinary:
11✔
1091
    request = BinaryPathRequest(binary_name="mv", search_path=system_binaries.system_binary_paths)
×
1092
    paths = await find_binary(request, **implicitly())
×
1093
    first_path = paths.first_path_or_raise(request, rationale="move files")
×
1094
    return MvBinary(first_path.path, first_path.fingerprint)
×
1095

1096

1097
@rule(desc="Finding the `open` binary", level=LogLevel.DEBUG)
11✔
1098
async def find_open(
11✔
1099
    platform: Platform, system_binaries: SystemBinariesSubsystem.EnvironmentAware
1100
) -> OpenBinary:
1101
    request = BinaryPathRequest(
×
1102
        binary_name=("open" if platform.is_macos else "xdg-open"),
1103
        search_path=system_binaries.system_binary_paths,
1104
    )
1105
    paths = await find_binary(request, **implicitly())
×
1106
    first_path = paths.first_path_or_raise(request, rationale="open URLs with default browser")
×
1107
    return OpenBinary(first_path.path, first_path.fingerprint)
×
1108

1109

1110
@rule(desc="Finding the `pwd` binary", level=LogLevel.DEBUG)
11✔
1111
async def find_pwd(system_binaries: SystemBinariesSubsystem.EnvironmentAware) -> PwdBinary:
11✔
1112
    request = BinaryPathRequest(binary_name="pwd", search_path=system_binaries.system_binary_paths)
×
1113
    paths = await find_binary(request, **implicitly())
×
1114
    first_path = paths.first_path_or_raise(request, rationale="pwd file")
×
1115
    return PwdBinary(first_path.path, first_path.fingerprint)
×
1116

1117

1118
@rule(desc="Finding the `readlink` binary", level=LogLevel.DEBUG)
11✔
1119
async def find_readlink(
11✔
1120
    system_binaries: SystemBinariesSubsystem.EnvironmentAware,
1121
) -> ReadlinkBinary:
1122
    request = BinaryPathRequest(
×
1123
        binary_name="readlink", search_path=system_binaries.system_binary_paths
1124
    )
1125
    paths = await find_binary(request, **implicitly())
×
1126
    first_path = paths.first_path_or_raise(request, rationale="dereference symlinks")
×
1127
    return ReadlinkBinary(first_path.path, first_path.fingerprint)
×
1128

1129

1130
@rule(desc="Finding the `rm` binary", level=LogLevel.DEBUG)
11✔
1131
async def find_rm(system_binaries: SystemBinariesSubsystem.EnvironmentAware) -> RmBinary:
11✔
1132
    request = BinaryPathRequest(binary_name="rm", search_path=system_binaries.system_binary_paths)
×
1133
    paths = await find_binary(request, **implicitly())
×
1134
    first_path = paths.first_path_or_raise(request, rationale="rm file")
×
1135
    return RmBinary(first_path.path, first_path.fingerprint)
×
1136

1137

1138
@rule(desc="Finding the `sed` binary", level=LogLevel.DEBUG)
11✔
1139
async def find_sed(system_binaries: SystemBinariesSubsystem.EnvironmentAware) -> SedBinary:
11✔
1140
    request = BinaryPathRequest(binary_name="sed", search_path=system_binaries.system_binary_paths)
×
1141
    paths = await find_binary(request, **implicitly())
×
1142
    first_path = paths.first_path_or_raise(request, rationale="sed file")
×
1143
    return SedBinary(first_path.path, first_path.fingerprint)
×
1144

1145

1146
@rule(desc="Finding the `sh` binary", level=LogLevel.DEBUG)
11✔
1147
async def find_sh(system_binaries: SystemBinariesSubsystem.EnvironmentAware) -> ShBinary:
11✔
1148
    request = BinaryPathRequest(binary_name="sh", search_path=system_binaries.system_binary_paths)
×
1149
    paths = await find_binary(request, **implicitly())
×
1150
    first_path = paths.first_path_or_raise(request, rationale="sh file")
×
1151
    return ShBinary(first_path.path, first_path.fingerprint)
×
1152

1153

1154
@rule(desc="Finding the `shasum` binary", level=LogLevel.DEBUG)
11✔
1155
async def find_shasum(system_binaries: SystemBinariesSubsystem.EnvironmentAware) -> ShasumBinary:
11✔
1156
    request = BinaryPathRequest(
×
1157
        binary_name="shasum", search_path=system_binaries.system_binary_paths
1158
    )
1159
    paths = await find_binary(request, **implicitly())
×
1160
    first_path = paths.first_path_or_raise(request, rationale="shasum file")
×
1161
    return ShasumBinary(first_path.path, first_path.fingerprint)
×
1162

1163

1164
@rule(desc="Finding the `sort` binary", level=LogLevel.DEBUG)
11✔
1165
async def find_sort(system_binaries: SystemBinariesSubsystem.EnvironmentAware) -> SortBinary:
11✔
1166
    request = BinaryPathRequest(binary_name="sort", search_path=system_binaries.system_binary_paths)
×
1167
    paths = await find_binary(request, **implicitly())
×
1168
    first_path = paths.first_path_or_raise(request, rationale="sort file")
×
1169
    return SortBinary(first_path.path, first_path.fingerprint)
×
1170

1171

1172
@rule(desc="Finding the `tail` binary", level=LogLevel.DEBUG)
11✔
1173
async def find_tail(system_binaries: SystemBinariesSubsystem.EnvironmentAware) -> TailBinary:
11✔
1174
    request = BinaryPathRequest(binary_name="tail", search_path=system_binaries.system_binary_paths)
×
1175
    paths = await find_binary(request, **implicitly())
×
1176
    first_path = paths.first_path_or_raise(request, rationale="tail file")
×
1177
    return TailBinary(first_path.path, first_path.fingerprint)
×
1178

1179

1180
@rule(desc="Finding the `tar` binary", level=LogLevel.DEBUG)
11✔
1181
async def find_tar(
11✔
1182
    platform: Platform, system_binaries: SystemBinariesSubsystem.EnvironmentAware
1183
) -> TarBinary:
1184
    request = BinaryPathRequest(
×
1185
        binary_name="tar",
1186
        search_path=system_binaries.system_binary_paths,
1187
        test=BinaryPathTest(args=["--version"]),
1188
    )
1189
    paths = await find_binary(request, **implicitly())
×
1190
    first_path = paths.first_path_or_raise(
×
1191
        request, rationale="download the tools Pants needs to run"
1192
    )
1193
    return TarBinary(first_path.path, first_path.fingerprint, platform)
×
1194

1195

1196
@rule(desc="Finding the `test` binary", level=LogLevel.DEBUG)
11✔
1197
async def find_test(system_binaries: SystemBinariesSubsystem.EnvironmentAware) -> TestBinary:
11✔
1198
    request = BinaryPathRequest(binary_name="test", search_path=system_binaries.system_binary_paths)
×
1199
    paths = await find_binary(request, **implicitly())
×
1200
    first_path = paths.first_path_or_raise(request, rationale="test file")
×
1201
    return TestBinary(first_path.path, first_path.fingerprint)
×
1202

1203

1204
@rule(desc="Finding the `touch` binary", level=LogLevel.DEBUG)
11✔
1205
async def find_touch(system_binaries: SystemBinariesSubsystem.EnvironmentAware) -> TouchBinary:
11✔
1206
    request = BinaryPathRequest(
×
1207
        binary_name="touch", search_path=system_binaries.system_binary_paths
1208
    )
1209
    paths = await find_binary(request, **implicitly())
×
1210
    first_path = paths.first_path_or_raise(request, rationale="touch file")
×
1211
    return TouchBinary(first_path.path, first_path.fingerprint)
×
1212

1213

1214
@rule(desc="Finding the `tr` binary", level=LogLevel.DEBUG)
11✔
1215
async def find_tr(system_binaries: SystemBinariesSubsystem.EnvironmentAware) -> TrBinary:
11✔
1216
    request = BinaryPathRequest(binary_name="tr", search_path=system_binaries.system_binary_paths)
×
1217
    paths = await find_binary(request, **implicitly())
×
1218
    first_path = paths.first_path_or_raise(request, rationale="tr file")
×
1219
    return TrBinary(first_path.path, first_path.fingerprint)
×
1220

1221

1222
@rule(desc="Finding the `unzip` binary", level=LogLevel.DEBUG)
11✔
1223
async def find_unzip(system_binaries: SystemBinariesSubsystem.EnvironmentAware) -> UnzipBinary:
11✔
1224
    request = BinaryPathRequest(
×
1225
        binary_name="unzip",
1226
        search_path=system_binaries.system_binary_paths,
1227
        test=BinaryPathTest(args=["-v"]),
1228
    )
1229
    paths = await find_binary(request, **implicitly())
×
1230
    first_path = paths.first_path_or_raise(
×
1231
        request, rationale="download the tools Pants needs to run"
1232
    )
1233
    return UnzipBinary(first_path.path, first_path.fingerprint)
×
1234

1235

1236
@rule(desc="Finding the `wc` binary", level=LogLevel.DEBUG)
11✔
1237
async def find_wc(system_binaries: SystemBinariesSubsystem.EnvironmentAware) -> WcBinary:
11✔
1238
    request = BinaryPathRequest(binary_name="wc", search_path=system_binaries.system_binary_paths)
×
1239
    paths = await find_binary(request, **implicitly())
×
1240
    first_path = paths.first_path_or_raise(request, rationale="wc file")
×
1241
    return WcBinary(first_path.path, first_path.fingerprint)
×
1242

1243

1244
@rule(desc="Finding the `xargs` binary", level=LogLevel.DEBUG)
11✔
1245
async def find_xargs(system_binaries: SystemBinariesSubsystem.EnvironmentAware) -> XargsBinary:
11✔
1246
    request = BinaryPathRequest(
×
1247
        binary_name="xargs", search_path=system_binaries.system_binary_paths
1248
    )
1249
    paths = await find_binary(request, **implicitly())
×
1250
    first_path = paths.first_path_or_raise(request, rationale="xargs file")
×
1251
    return XargsBinary(first_path.path, first_path.fingerprint)
×
1252

1253

1254
@rule(desc="Finding the `xz` binary", level=LogLevel.DEBUG)
11✔
1255
async def find_xz(system_binaries: SystemBinariesSubsystem.EnvironmentAware) -> XzBinary:
11✔
1256
    request = BinaryPathRequest(binary_name="xz", search_path=system_binaries.system_binary_paths)
×
1257
    paths = await find_binary(request, **implicitly())
×
1258
    first_path = paths.first_path_or_raise(request, rationale="xz file")
×
1259
    return XzBinary(first_path.path, first_path.fingerprint)
×
1260

1261

1262
@rule(desc="Finding the `zip` binary", level=LogLevel.DEBUG)
11✔
1263
async def find_zip(system_binaries: SystemBinariesSubsystem.EnvironmentAware) -> ZipBinary:
11✔
1264
    request = BinaryPathRequest(
×
1265
        binary_name="zip",
1266
        search_path=system_binaries.system_binary_paths,
1267
        test=BinaryPathTest(args=["-v"]),
1268
    )
1269
    paths = await find_binary(request, **implicitly())
×
1270
    first_path = paths.first_path_or_raise(request, rationale="create `.zip` archives")
×
1271
    return ZipBinary(first_path.path, first_path.fingerprint)
×
1272

1273

1274
@rule(desc="Finding the `zstd` binary", level=LogLevel.DEBUG)
11✔
1275
async def find_zstd(system_binaries: SystemBinariesSubsystem.EnvironmentAware) -> ZstdBinary:
11✔
1276
    request = BinaryPathRequest(binary_name="zstd", search_path=system_binaries.system_binary_paths)
×
1277
    paths = await find_binary(request, **implicitly())
×
1278
    first_path = paths.first_path_or_raise(request, rationale="zstd file")
×
1279
    return ZstdBinary(first_path.path, first_path.fingerprint)
×
1280

1281

1282
def rules():
11✔
1283
    return [*collect_rules(), *python_bootstrap.rules()]
11✔
1284

1285

1286
# -------------------------------------------------------------------------------------------
1287
# Rules for fallible binaries
1288
# -------------------------------------------------------------------------------------------
1289

1290

1291
@dataclass(frozen=True)
11✔
1292
class MaybeGitBinary:
11✔
1293
    git_binary: GitBinary | None = None
11✔
1294

1295

1296
@rule(desc="Finding the `git` binary", level=LogLevel.DEBUG)
11✔
1297
async def maybe_find_git(
11✔
1298
    system_binaries: SystemBinariesSubsystem.EnvironmentAware,
1299
) -> MaybeGitBinary:
1300
    request = BinaryPathRequest(binary_name="git", search_path=system_binaries.system_binary_paths)
×
1301
    paths = await find_binary(request, **implicitly())
×
1302
    first_path = paths.first_path
×
1303
    if not first_path:
×
1304
        return MaybeGitBinary()
×
1305

1306
    return MaybeGitBinary(GitBinary(first_path.path, first_path.fingerprint))
×
1307

1308

1309
class MaybeGitBinaryRequest:
11✔
1310
    pass
11✔
1311

1312

1313
@rule
11✔
1314
async def maybe_find_git_wrapper(
11✔
1315
    _: MaybeGitBinaryRequest, maybe_git_binary: MaybeGitBinary
1316
) -> MaybeGitBinary:
1317
    return maybe_git_binary
×
STATUS · Troubleshooting · Open an Issue · Sales · Support · CAREERS · ENTERPRISE · START FREE · SCHEDULE DEMO
ANNOUNCEMENTS · TWITTER · TOS & SLA · Supported CI Services · What's a CI service? · Automated Testing

© 2025 Coveralls, Inc