19529437518

Committed 20 Nov 2025 07:44AM UTC coverage: 78.884% (-1.4%) from 80.302%

Build # 19529437518

Build Type

push

github

Committed by

web-flow

Commit Message

nfpm.native_libs: Add RPM package depends from packaged pex_binaries (#22899)

## PR Series Overview

This is the second in a series of PRs that introduces a new backend:
`pants.backend.npm.native_libs`
Initially, the backend will be available as:
`pants.backend.experimental.nfpm.native_libs`

I proposed this new backend (originally named `bindeps`) in discussion
#22396.

This backend will inspect ELF bin/lib files (like `lib*.so`) in packaged
contents (for this PR series, only in `pex_binary` targets) to identify
package dependency metadata and inject that metadata on the relevant
`nfpm_deb_package` or `nfpm_rpm_package` targets. Effectively, it will
provide an approximation of these native packager features:
- `rpm`: `rpmdeps` + `elfdeps`
- `deb`: `dh_shlibdeps` + `dpkg-shlibdeps` (These substitute
`${shlibs:Depends}` in debian control files have)

### Goal: Host-agnostic package builds

This pants backend is designed to be host-agnostic, like
[nFPM](https://nfpm.goreleaser.com/).

Native packaging tools are often restricted to a single release of a
single distro. Unlike native package builders, this new pants backend
does not use any of those distro-specific or distro-release-specific
utilities or local package databases. This new backend should be able to
run (help with building deb and rpm packages) anywhere that pants can
run (MacOS, rpm linux distros, deb linux distros, other linux distros,
docker, ...).

### Previous PRs in series

- #22873

## PR Overview

This PR adds rules in `nfpm.native_libs` to add package dependency
metadata to `nfpm_rpm_package`. The 2 new rules are:

- `inject_native_libs_dependencies_in_package_fields`:

    - An implementation of the polymorphic rule `inject_nfpm_package_fields`.
      This rule is low priority (`priority = 2`) so that in-repo plugins can
      override/augment what it injects. (See #22864)

    - Rule logic overview:
        - find any pex_binaries that will be packaged in an `nfpm_rpm_package`
   ... (continued)

Run Details

96 of 118 new or added lines in 3 files covered. (81.36%)

910 existing lines in 53 files now uncovered.

73897 of 93678 relevant lines covered (78.88%)

3.21 hits per line

Source File
Press 'n' to go to next uncovered line, 'b' for previous

81.25

/src/python/pants/pantsd/service/pants_service.py

# Copyright 2015 Pants project contributors (see CONTRIBUTORS.md).
# Licensed under the Apache License, Version 2.0 (see LICENSE).

import logging
import threading
import time
from abc import ABC, abstractmethod
from collections.abc import KeysView
from dataclasses import dataclass

logger = logging.getLogger(__name__)


class PantsService(ABC):
    """Pants daemon service base class.

    The service lifecycle is made up of states described in the _ServiceState class, and controlled
    by a calling thread that is holding the Service `lifecycle_lock`. Under that lock, a caller
    can signal a service to "pause", "run", or "terminate" (see _ServiceState for more details).

    pantsd pauses all Services before forking a pantsd in order to ensure that no "relevant"
    locks are held (or awaited: see #6565) by threads that might not survive the fork. While paused,
    a Service must not have any threads running that might interact with any non-private locks.

    After forking, the pantsd (child) process should call `terminate()` to finish shutting down
    the service, and the parent process should call `resume()` to cause the service to resume running.
    """

    class ServiceError(Exception):
        pass

    def __init__(self):
        super().__init__()
        self.name = self.__class__.__name__
        self._state = _ServiceState()

    def setup(self, services: tuple["PantsService", ...]):
        """Called before `run` to allow for service->service or other side-effecting setup."""
        self.services = services

    @abstractmethod
    def run(self):
        """The main entry-point for the service called by the service runner."""

    def mark_pausing(self):
        """Triggers pausing of the service, without waiting for it to have paused.

        See the class and _ServiceState pydocs.
        """
        self._state.mark_pausing()

    def await_paused(self):
        """Once a service has been marked pausing, waits for it to have paused.

        See the class and _ServiceState pydocs.
        """
        self._state.await_paused()

    def resume(self):
        """Triggers the service to resume running, without waiting.

        See the class and _ServiceState pydocs.
        """
        self._state.mark_running()

    def terminate(self):
        """Triggers termination of the service, without waiting.

        See the class and _ServiceState pydocs.
        """
        self._state.mark_terminating()


class _ServiceState:
    """A threadsafe state machine for controlling a service running in another thread.

    The state machine represents two stable states:
      Running
      Paused
    And two transitional states:
      Pausing
      Terminating

    The methods of this class allow a caller to ask the Service to transition states, and then wait
    for those transitions to occur.

    A simplifying assumption is that there is one service thread that interacts with the state, and
    only one controlling thread. In the case of `pantsd`, the "one calling thread" condition is
    protected by the service `lifecycle_lock`.

    A complicating assumption is that while a service thread is `Paused`, it must be in a position
    where it could safely disappear and never come back. This is accounted for by having the service
    thread wait on a Condition variable while Paused: testing indicates that for multiple Pythons
    on both OSX and Linux, this does not result in poisoning of the associated Lock.
    """

    _RUNNING = "Running"
    _PAUSED = "Paused"
    _PAUSING = "Pausing"
    _TERMINATING = "Terminating"

    def __init__(self):
        """Creates a ServiceState in the Running state."""
        self._state = self._RUNNING
        self._lock = threading.Lock()
        self._condition = threading.Condition(self._lock)

    def _set_state(self, state, *valid_states):
        if valid_states and self._state not in valid_states:
            raise AssertionError(f"Cannot move {self} to `{state}` while it is `{self._state}`.")
        self._state = state
        self._condition.notify_all()

    def await_paused(self, timeout=None):
        """Blocks until the service is in the Paused state, then returns True.

        If a timeout is specified, the method may return False to indicate a timeout: with no timeout
        it will always (eventually) return True.

        Raises if the service is not currently in the Pausing state.
        """
        deadline = time.time() + timeout if timeout else None
        with self._lock:
            # Wait until the service transitions out of Pausing.
            while self._state != self._PAUSED:
                if self._state != self._PAUSING:
                    raise AssertionError(
                        f"Cannot wait for {self} to reach `{self._PAUSED}` while it is in `{self._state}`."
                    )
                timeout = deadline - time.time() if deadline else None
                if timeout and timeout <= 0:
                    return False
                self._condition.wait(timeout=timeout)
            return True

    def maybe_pause(self, timeout=None):
        """Called by the service to indicate that it is pausable.

        If the service calls this method while the state is `Pausing`, the state will transition
        to `Paused`, and the service will block here until it is marked `Running` or `Terminating`.

        If the state is not currently `Pausing`, and a timeout is not passed, this method returns
        immediately. If a timeout is passed, this method blocks up to that number of seconds to wait
        to transition to `Pausing`.
        """
        deadline = time.time() + timeout if timeout else None
        with self._lock:
            while self._state != self._PAUSING:
                # If we've been terminated, or the deadline has passed, return.
                timeout = deadline - time.time() if deadline else None
                if self._state == self._TERMINATING or not timeout or timeout <= 0:
                    return
                # Otherwise, wait for the state to change.
                self._condition.wait(timeout=timeout)

            # Set Paused, and then wait until we are no longer Paused.
            self._set_state(self._PAUSED, self._PAUSING)
            while self._state == self._PAUSED:
                self._condition.wait()

    def mark_pausing(self):
        """Requests that the service move to the Paused state, without waiting for it to do so.

        Raises if the service is not currently in the Running state.
        """
        with self._lock:
            self._set_state(self._PAUSING, self._RUNNING)

    def mark_running(self):
        """Moves the service to the Running state.

        Raises if the service is not currently in the Paused state.
        """
        with self._lock:
            self._set_state(self._RUNNING, self._PAUSED)

    def mark_terminating(self):
        """Requests that the service move to the Terminating state, without waiting for it to do
        so."""
        with self._lock:
            self._set_state(self._TERMINATING)

    @property
    def is_terminating(self):
        """Returns True if the Service should currently be terminating.

        NB: `Terminating` does not have an associated "terminated" state, because the caller uses
        liveness of the service thread to determine when a service is terminated.
        """
        with self._lock:
            return self._state == self._TERMINATING


@dataclass(frozen=True)
class PantsServices:
    """A collection of running PantsServices threads."""

    JOIN_TIMEOUT_SECONDS = 1

    _service_threads: dict[PantsService, threading.Thread]

    def __init__(self, services: tuple[PantsService, ...] = ()) -> None:
        object.__setattr__(self, "_service_threads", self._start(services))

    @classmethod
    def _make_thread(cls, service):
        name = f"{service.__class__.__name__}Thread"
        t = threading.Thread(target=service.run, name=name)
        t.daemon = True
        return t

    @classmethod
    def _start(cls, services: tuple[PantsService, ...]) -> dict[PantsService, threading.Thread]:
        """Launch a thread per service."""

        for service in services:
            logger.debug(f"setting up service {service}")
            service.setup(services)

        service_thread_map = {service: cls._make_thread(service) for service in services}

        for service, service_thread in service_thread_map.items():
            logger.debug(f"starting service {service}")
            service_thread.start()

        return service_thread_map

    @property
    def services(self) -> KeysView[PantsService]:
        return self._service_threads.keys()

    def are_all_alive(self) -> bool:
        """Return true if all services threads are still alive, and false if any have died.

        This method does not have side effects: if one service thread has died, the rest should be
        killed and joined via `self.shutdown()`.
        """
        for service, service_thread in self._service_threads.items():
            if not service_thread.is_alive():
                logger.error(f"service failure for {service}.")
                return False
        return True

    def shutdown(self) -> None:
        """Shut down and join all service threads."""
        for service, service_thread in self._service_threads.items():
            service.terminate()
        for service, service_thread in self._service_threads.items():
            logger.debug(f"terminating pantsd service: {service}")
            service_thread.join(self.JOIN_TIMEOUT_SECONDS)

1	# Copyright 2015 Pants project contributors (see CONTRIBUTORS.md).
2	# Licensed under the Apache License, Version 2.0 (see LICENSE).
3
4	import logging	11✔
5	import threading	11✔
6	import time	11✔
7	from abc import ABC, abstractmethod	11✔
8	from collections.abc import KeysView	11✔
9	from dataclasses import dataclass	11✔
10
11	logger = logging.getLogger(__name__)	11✔
12
13
14	class PantsService(ABC):	11✔
15	"""Pants daemon service base class.
16
17	The service lifecycle is made up of states described in the _ServiceState class, and controlled
18	by a calling thread that is holding the Service `lifecycle_lock`. Under that lock, a caller
19	can signal a service to "pause", "run", or "terminate" (see _ServiceState for more details).
20
21	pantsd pauses all Services before forking a pantsd in order to ensure that no "relevant"
22	locks are held (or awaited: see #6565) by threads that might not survive the fork. While paused,
23	a Service must not have any threads running that might interact with any non-private locks.
24
25	After forking, the pantsd (child) process should call `terminate()` to finish shutting down
26	the service, and the parent process should call `resume()` to cause the service to resume running.
27	"""
28
29	class ServiceError(Exception):	11✔
30	pass	11✔
31
32	def __init__(self):	11✔
33	super().__init__()	1✔
34	self.name = self.__class__.__name__	1✔
35	self._state = _ServiceState()	1✔
36
37	def setup(self, services: tuple["PantsService", ...]):	11✔
38	"""Called before `run` to allow for service->service or other side-effecting setup."""
UNCOV 39	self.services = services	×
40
41	@abstractmethod	11✔
42	def run(self):	11✔
43	"""The main entry-point for the service called by the service runner."""
44
45	def mark_pausing(self):	11✔
46	"""Triggers pausing of the service, without waiting for it to have paused.
47
48	See the class and _ServiceState pydocs.
49	"""
50	self._state.mark_pausing()	1✔
51
52	def await_paused(self):	11✔
53	"""Once a service has been marked pausing, waits for it to have paused.
54
55	See the class and _ServiceState pydocs.
56	"""
57	self._state.await_paused()	×
58
59	def resume(self):	11✔
60	"""Triggers the service to resume running, without waiting.
61
62	See the class and _ServiceState pydocs.
63	"""
64	self._state.mark_running()	1✔
65
66	def terminate(self):	11✔
67	"""Triggers termination of the service, without waiting.
68
69	See the class and _ServiceState pydocs.
70	"""
71	self._state.mark_terminating()	1✔
72
73
74	class _ServiceState:	11✔
75	"""A threadsafe state machine for controlling a service running in another thread.
76
77	The state machine represents two stable states:
78	Running
79	Paused
80	And two transitional states:
81	Pausing
82	Terminating
83
84	The methods of this class allow a caller to ask the Service to transition states, and then wait
85	for those transitions to occur.
86
87	A simplifying assumption is that there is one service thread that interacts with the state, and
88	only one controlling thread. In the case of `pantsd`, the "one calling thread" condition is
89	protected by the service `lifecycle_lock`.
90
91	A complicating assumption is that while a service thread is `Paused`, it must be in a position
92	where it could safely disappear and never come back. This is accounted for by having the service
93	thread wait on a Condition variable while Paused: testing indicates that for multiple Pythons
94	on both OSX and Linux, this does not result in poisoning of the associated Lock.
95	"""
96
97	_RUNNING = "Running"	11✔
98	_PAUSED = "Paused"	11✔
99	_PAUSING = "Pausing"	11✔
100	_TERMINATING = "Terminating"	11✔
101
102	def __init__(self):	11✔
103	"""Creates a ServiceState in the Running state."""
104	self._state = self._RUNNING	1✔
105	self._lock = threading.Lock()	1✔
106	self._condition = threading.Condition(self._lock)	1✔
107
108	def _set_state(self, state, *valid_states):	11✔
109	if valid_states and self._state not in valid_states:	1✔
110	raise AssertionError(f"Cannot move {self} to `{state}` while it is `{self._state}`.")	×
111	self._state = state	1✔
112	self._condition.notify_all()	1✔
113
114	def await_paused(self, timeout=None):	11✔
115	"""Blocks until the service is in the Paused state, then returns True.
116
117	If a timeout is specified, the method may return False to indicate a timeout: with no timeout
118	it will always (eventually) return True.
119
120	Raises if the service is not currently in the Pausing state.
121	"""
122	deadline = time.time() + timeout if timeout else None	1✔
123	with self._lock:	1✔
124	# Wait until the service transitions out of Pausing.
125	while self._state != self._PAUSED:	1✔
126	if self._state != self._PAUSING:	1✔
127	raise AssertionError(	×
128	f"Cannot wait for {self} to reach `{self._PAUSED}` while it is in `{self._state}`."
129	)
130	timeout = deadline - time.time() if deadline else None	1✔
131	if timeout and timeout <= 0:	1✔
132	return False	1✔
133	self._condition.wait(timeout=timeout)	1✔
134	return True	1✔
135
136	def maybe_pause(self, timeout=None):	11✔
137	"""Called by the service to indicate that it is pausable.
138
139	If the service calls this method while the state is `Pausing`, the state will transition
140	to `Paused`, and the service will block here until it is marked `Running` or `Terminating`.
141
142	If the state is not currently `Pausing`, and a timeout is not passed, this method returns
143	immediately. If a timeout is passed, this method blocks up to that number of seconds to wait
144	to transition to `Pausing`.
145	"""
146	deadline = time.time() + timeout if timeout else None	1✔
147	with self._lock:	1✔
148	while self._state != self._PAUSING:	1✔
149	# If we've been terminated, or the deadline has passed, return.
150	timeout = deadline - time.time() if deadline else None	1✔
151	if self._state == self._TERMINATING or not timeout or timeout <= 0:	1✔
152	return	1✔
153	# Otherwise, wait for the state to change.
154	self._condition.wait(timeout=timeout)	1✔
155
156	# Set Paused, and then wait until we are no longer Paused.
157	self._set_state(self._PAUSED, self._PAUSING)	1✔
158	while self._state == self._PAUSED:	1✔
159	self._condition.wait()	1✔
160
161	def mark_pausing(self):	11✔
162	"""Requests that the service move to the Paused state, without waiting for it to do so.
163
164	Raises if the service is not currently in the Running state.
165	"""
166	with self._lock:	1✔
167	self._set_state(self._PAUSING, self._RUNNING)	1✔
168
169	def mark_running(self):	11✔
170	"""Moves the service to the Running state.
171
172	Raises if the service is not currently in the Paused state.
173	"""
174	with self._lock:	1✔
175	self._set_state(self._RUNNING, self._PAUSED)	1✔
176
177	def mark_terminating(self):	11✔
178	"""Requests that the service move to the Terminating state, without waiting for it to do
179	so."""
180	with self._lock:	1✔
181	self._set_state(self._TERMINATING)	1✔
182
183	@property	11✔
184	def is_terminating(self):	11✔
185	"""Returns True if the Service should currently be terminating.
186
187	NB: `Terminating` does not have an associated "terminated" state, because the caller uses
188	liveness of the service thread to determine when a service is terminated.
189	"""
190	with self._lock:	1✔
191	return self._state == self._TERMINATING	1✔
192
193
194	@dataclass(frozen=True)	11✔
195	class PantsServices:	11✔
196	"""A collection of running PantsServices threads."""
197
198	JOIN_TIMEOUT_SECONDS = 1	11✔
199
200	_service_threads: dict[PantsService, threading.Thread]	11✔
201
202	def __init__(self, services: tuple[PantsService, ...] = ()) -> None:	11✔
203	object.__setattr__(self, "_service_threads", self._start(services))	1✔
204
205	@classmethod	11✔
206	def _make_thread(cls, service):	11✔
207	name = f"{service.__class__.__name__}Thread"	×
208	t = threading.Thread(target=service.run, name=name)	×
209	t.daemon = True	×
210	return t	×
211
212	@classmethod	11✔
213	def _start(cls, services: tuple[PantsService, ...]) -> dict[PantsService, threading.Thread]:	11✔
214	"""Launch a thread per service."""
215
216	for service in services:	1✔
217	logger.debug(f"setting up service {service}")	×
218	service.setup(services)	×
219
220	service_thread_map = {service: cls._make_thread(service) for service in services}	1✔
221
222	for service, service_thread in service_thread_map.items():	1✔
223	logger.debug(f"starting service {service}")	×
224	service_thread.start()	×
225
226	return service_thread_map	1✔
227
228	@property	11✔
229	def services(self) -> KeysView[PantsService]:	11✔
230	return self._service_threads.keys()	×
231
232	def are_all_alive(self) -> bool:	11✔
233	"""Return true if all services threads are still alive, and false if any have died.
234
235	This method does not have side effects: if one service thread has died, the rest should be
236	killed and joined via `self.shutdown()`.
237	"""
238	for service, service_thread in self._service_threads.items():	×
239	if not service_thread.is_alive():	×
240	logger.error(f"service failure for {service}.")	×
241	return False	×
242	return True	×
243
244	def shutdown(self) -> None:	11✔
245	"""Shut down and join all service threads."""
246	for service, service_thread in self._service_threads.items():	1✔
247	service.terminate()	×
248	for service, service_thread in self._service_threads.items():	1✔
249	logger.debug(f"terminating pantsd service: {service}")	×
250	service_thread.join(self.JOIN_TIMEOUT_SECONDS)	×

pantsbuild / pants / 19529437518

Source File Press 'n' to go to next uncovered line, 'b' for previous

Source File
Press 'n' to go to next uncovered line, 'b' for previous