19529437518

Committed 20 Nov 2025 07:44AM UTC coverage: 78.884% (-1.4%) from 80.302%

Build # 19529437518

Build Type

push

github

Committed by

web-flow

Commit Message

nfpm.native_libs: Add RPM package depends from packaged pex_binaries (#22899)

## PR Series Overview

This is the second in a series of PRs that introduces a new backend:
`pants.backend.npm.native_libs`
Initially, the backend will be available as:
`pants.backend.experimental.nfpm.native_libs`

I proposed this new backend (originally named `bindeps`) in discussion
#22396.

This backend will inspect ELF bin/lib files (like `lib*.so`) in packaged
contents (for this PR series, only in `pex_binary` targets) to identify
package dependency metadata and inject that metadata on the relevant
`nfpm_deb_package` or `nfpm_rpm_package` targets. Effectively, it will
provide an approximation of these native packager features:
- `rpm`: `rpmdeps` + `elfdeps`
- `deb`: `dh_shlibdeps` + `dpkg-shlibdeps` (These substitute
`${shlibs:Depends}` in debian control files have)

### Goal: Host-agnostic package builds

This pants backend is designed to be host-agnostic, like
[nFPM](https://nfpm.goreleaser.com/).

Native packaging tools are often restricted to a single release of a
single distro. Unlike native package builders, this new pants backend
does not use any of those distro-specific or distro-release-specific
utilities or local package databases. This new backend should be able to
run (help with building deb and rpm packages) anywhere that pants can
run (MacOS, rpm linux distros, deb linux distros, other linux distros,
docker, ...).

### Previous PRs in series

- #22873

## PR Overview

This PR adds rules in `nfpm.native_libs` to add package dependency
metadata to `nfpm_rpm_package`. The 2 new rules are:

- `inject_native_libs_dependencies_in_package_fields`:

    - An implementation of the polymorphic rule `inject_nfpm_package_fields`.
      This rule is low priority (`priority = 2`) so that in-repo plugins can
      override/augment what it injects. (See #22864)

    - Rule logic overview:
        - find any pex_binaries that will be packaged in an `nfpm_rpm_package`
   ... (continued)

Run Details

96 of 118 new or added lines in 3 files covered. (81.36%)

910 existing lines in 53 files now uncovered.

73897 of 93678 relevant lines covered (78.88%)

3.21 hits per line

Source File
Press 'n' to go to next uncovered line, 'b' for previous

42.86

/src/python/pants/bsp/protocol.py

# Copyright 2022 Pants project contributors (see CONTRIBUTORS.md).
# Licensed under the Apache License, Version 2.0 (see LICENSE).
from __future__ import annotations

import logging
from concurrent.futures import Future
from typing import Any, BinaryIO, ClassVar, Protocol

from pylsp_jsonrpc.endpoint import Endpoint  # type: ignore[import-untyped]
from pylsp_jsonrpc.exceptions import (  # type: ignore[import-untyped]
    JsonRpcException,
    JsonRpcInvalidRequest,
    JsonRpcMethodNotFound,
)
from pylsp_jsonrpc.streams import (  # type: ignore[import-untyped]
    JsonRpcStreamReader,
    JsonRpcStreamWriter,
)

from pants.bsp.context import BSPContext
from pants.bsp.spec.notification import BSPNotification
from pants.core.environments.rules import determine_bootstrap_environment
from pants.engine.environment import EnvironmentName
from pants.engine.fs import Workspace
from pants.engine.internals.scheduler import SchedulerSession
from pants.engine.internals.selectors import Params
from pants.engine.unions import UnionMembership, union

_logger = logging.getLogger(__name__)


class BSPRequestTypeProtocol(Protocol):
    @classmethod
    def from_json_dict(cls, d: dict[str, Any]) -> Any: ...


class BSPResponseTypeProtocol(Protocol):
    def to_json_dict(self) -> dict[str, Any]: ...


@union(in_scope_types=[EnvironmentName])
class BSPHandlerMapping:
    """Union type for rules to register handlers for BSP methods."""

    # Name of the JSON-RPC method to be handled.
    method_name: ClassVar[str]

    # Type requested from the engine. This will be provided as the "subject" of an engine query.
    # Must implement class method `from_json_dict`.
    request_type: type[BSPRequestTypeProtocol]

    # Type produced by the handler rule. This will be requested as the "product" of the engine query.
    # Must implement instance method `to_json_dict`.
    response_type: type[BSPResponseTypeProtocol]

    # True if this handler is for a notification.
    # TODO: Consider how to pass notifications (which do not have responses) to the engine rules.
    is_notification: bool = False


def _make_error_future(exc: Exception) -> Future:
    fut: Future = Future()
    fut.set_exception(exc)
    return fut


class BSPConnection:
    _INITIALIZE_METHOD_NAME = "build/initialize"
    _SHUTDOWN_METHOD_NAME = "build/shutdown"
    _EXIT_NOTIFICATION_NAME = "build/exit"

    def __init__(
        self,
        scheduler_session: SchedulerSession,
        union_membership: UnionMembership,
        context: BSPContext,
        inbound: BinaryIO,
        outbound: BinaryIO,
        max_workers: int = 5,
    ) -> None:
        self._scheduler_session = scheduler_session
        # TODO: We might eventually want to make this configurable.
        self._env_name = determine_bootstrap_environment(self._scheduler_session)
        self._inbound = JsonRpcStreamReader(inbound)
        self._outbound = JsonRpcStreamWriter(outbound)
        self._context: BSPContext = context
        self._endpoint = Endpoint(self, self._send_outbound_message, max_workers=max_workers)

        self._handler_mappings: dict[str, type[BSPHandlerMapping]] = {}
        impls = union_membership.get(BSPHandlerMapping)
        for impl in impls:
            self._handler_mappings[impl.method_name] = impl

    def run(self) -> None:
        """Run the listener for inbound JSON-RPC messages."""
        self._inbound.listen(self._received_inbound_message)

    def _received_inbound_message(self, msg):
        """Process each inbound JSON-RPC message."""
        _logger.info(f"_received_inbound_message: msg={msg}")
        self._endpoint.consume(msg)

    def _send_outbound_message(self, msg):
        _logger.info(f"_send_outbound_message: msg={msg}")
        self._outbound.write(msg)

    # TODO: Figure out how to run this on the `Endpoint`'s thread pool by returning a callable. For now, we
    # need to return errors as futures given that `Endpoint` only handles exceptions returned that way versus using a try ... except block.
    def _handle_inbound_message(self, *, method_name: str, params: Any):
        # If the connection is not yet initialized and this is not the initialization request, BSP requires
        # returning an error for methods (and to discard all notifications).
        #
        # Concurrency: This method can be invoked from multiple threads (for each individual request). By returning
        # an error for all other requests, only the thread running the initialization RPC should be able to proceed.
        # This ensures that we can safely call `initialize_connection` on the BSPContext with the client-supplied
        # init parameters without worrying about multiple threads. (Not entirely true though as this does not handle
        # the client making multiple concurrent initialization RPCs, but which would violate the protocol in any case.)
        if (
            not self._context.is_connection_initialized
            and method_name != self._INITIALIZE_METHOD_NAME
        ):
            return _make_error_future(
                JsonRpcException(
                    code=-32002, message=f"Client must first call `{self._INITIALIZE_METHOD_NAME}`."
                )
            )

        # Handle the `build/shutdown` method and `build/exit` notification.
        if method_name == self._SHUTDOWN_METHOD_NAME:
            # Return no-op success for the `build/shutdown` method. This doesn't actually cause the server to
            # exit. That will occur once the client sends the `build/exit` notification.
            return None
        elif method_name == self._EXIT_NOTIFICATION_NAME:
            # The `build/exit` notification directs the BSP server to immediately exit.
            # The read-dispatch loop will exit once it notices that the inbound handle is closed. So close the
            # inbound handle (and outbound handle for completeness) and then return to the dispatch loop
            # to trigger the exit.
            self._inbound.close()
            self._outbound.close()
            return None

        method_mapping = self._handler_mappings.get(method_name)
        if not method_mapping:
            return _make_error_future(JsonRpcMethodNotFound.of(method_name))

        try:
            request = method_mapping.request_type.from_json_dict(params)
        except Exception:
            return _make_error_future(JsonRpcInvalidRequest())

        # TODO: This should not be necessary: see https://github.com/pantsbuild/pants/issues/15435.
        self._scheduler_session.new_run_id()

        workspace = Workspace(self._scheduler_session)
        params = Params(request, workspace, self._env_name)
        execution_request = self._scheduler_session.execution_request(
            requests=[(method_mapping.response_type, params)],
        )
        (result,) = self._scheduler_session.execute(execution_request)
        # Initialize the BSPContext with the client-supplied init parameters. See earlier comment on why this
        # call to `BSPContext.initialize_connection` is safe.
        if method_name == self._INITIALIZE_METHOD_NAME:
            self._context.initialize_connection(request, self.notify_client)
        return result.to_json_dict()

    # Called by `Endpoint` to dispatch requests and notifications.
    # TODO: Should probably vendor `Endpoint` so we can detect notifications versus method calls, which
    # matters when ignoring unknown notifications versus erroring for unknown methods.
    def __getitem__(self, method_name):
        def handler(params):
            return self._handle_inbound_message(method_name=method_name, params=params)

        return handler

    def notify_client(self, notification: BSPNotification) -> None:
        try:
            self._endpoint.notify(notification.notification_name, notification.to_json_dict())
        except Exception as ex:
            _logger.warning(f"Received exception while notifying BSP client: {ex}")

1	# Copyright 2022 Pants project contributors (see CONTRIBUTORS.md).
2	# Licensed under the Apache License, Version 2.0 (see LICENSE).
3	from __future__ import annotations	1✔
4
5	import logging	1✔
6	from concurrent.futures import Future	1✔
7	from typing import Any, BinaryIO, ClassVar, Protocol	1✔
8
9	from pylsp_jsonrpc.endpoint import Endpoint # type: ignore[import-untyped]	1✔
10	from pylsp_jsonrpc.exceptions import ( # type: ignore[import-untyped]	1✔
11	JsonRpcException,
12	JsonRpcInvalidRequest,
13	JsonRpcMethodNotFound,
14	)
15	from pylsp_jsonrpc.streams import ( # type: ignore[import-untyped]	1✔
16	JsonRpcStreamReader,
17	JsonRpcStreamWriter,
18	)
19
20	from pants.bsp.context import BSPContext	1✔
21	from pants.bsp.spec.notification import BSPNotification	1✔
22	from pants.core.environments.rules import determine_bootstrap_environment	1✔
23	from pants.engine.environment import EnvironmentName	1✔
24	from pants.engine.fs import Workspace	1✔
25	from pants.engine.internals.scheduler import SchedulerSession	1✔
26	from pants.engine.internals.selectors import Params	1✔
27	from pants.engine.unions import UnionMembership, union	1✔
28
29	_logger = logging.getLogger(__name__)	1✔
30
31
32	class BSPRequestTypeProtocol(Protocol):	1✔
33	@classmethod
34	def from_json_dict(cls, d: dict[str, Any]) -> Any: ...
35
36
37	class BSPResponseTypeProtocol(Protocol):	1✔
38	def to_json_dict(self) -> dict[str, Any]: ...
39
40
41	@union(in_scope_types=[EnvironmentName])	1✔
42	class BSPHandlerMapping:	1✔
43	"""Union type for rules to register handlers for BSP methods."""
44
45	# Name of the JSON-RPC method to be handled.
46	method_name: ClassVar[str]	1✔
47
48	# Type requested from the engine. This will be provided as the "subject" of an engine query.
49	# Must implement class method `from_json_dict`.
50	request_type: type[BSPRequestTypeProtocol]	1✔
51
52	# Type produced by the handler rule. This will be requested as the "product" of the engine query.
53	# Must implement instance method `to_json_dict`.
54	response_type: type[BSPResponseTypeProtocol]	1✔
55
56	# True if this handler is for a notification.
57	# TODO: Consider how to pass notifications (which do not have responses) to the engine rules.
58	is_notification: bool = False	1✔
59
60
61	def _make_error_future(exc: Exception) -> Future:	1✔
UNCOV 62	fut: Future = Future()	×
UNCOV 63	fut.set_exception(exc)	×
UNCOV 64	return fut	×
65
66
67	class BSPConnection:	1✔
68	_INITIALIZE_METHOD_NAME = "build/initialize"	1✔
69	_SHUTDOWN_METHOD_NAME = "build/shutdown"	1✔
70	_EXIT_NOTIFICATION_NAME = "build/exit"	1✔
71
72	def __init__(	1✔
73	self,
74	scheduler_session: SchedulerSession,
75	union_membership: UnionMembership,
76	context: BSPContext,
77	inbound: BinaryIO,
78	outbound: BinaryIO,
79	max_workers: int = 5,
80	) -> None:
UNCOV 81	self._scheduler_session = scheduler_session	×
82	# TODO: We might eventually want to make this configurable.
UNCOV 83	self._env_name = determine_bootstrap_environment(self._scheduler_session)	×
UNCOV 84	self._inbound = JsonRpcStreamReader(inbound)	×
UNCOV 85	self._outbound = JsonRpcStreamWriter(outbound)	×
UNCOV 86	self._context: BSPContext = context	×
UNCOV 87	self._endpoint = Endpoint(self, self._send_outbound_message, max_workers=max_workers)	×
88
UNCOV 89	self._handler_mappings: dict[str, type[BSPHandlerMapping]] = {}	×
UNCOV 90	impls = union_membership.get(BSPHandlerMapping)	×
UNCOV 91	for impl in impls:	×
UNCOV 92	self._handler_mappings[impl.method_name] = impl	×
93
94	def run(self) -> None:	1✔
95	"""Run the listener for inbound JSON-RPC messages."""
UNCOV 96	self._inbound.listen(self._received_inbound_message)	×
97
98	def _received_inbound_message(self, msg):	1✔
99	"""Process each inbound JSON-RPC message."""
UNCOV 100	_logger.info(f"_received_inbound_message: msg={msg}")	×
UNCOV 101	self._endpoint.consume(msg)	×
102
103	def _send_outbound_message(self, msg):	1✔
UNCOV 104	_logger.info(f"_send_outbound_message: msg={msg}")	×
UNCOV 105	self._outbound.write(msg)	×
106
107	# TODO: Figure out how to run this on the `Endpoint`'s thread pool by returning a callable. For now, we
108	# need to return errors as futures given that `Endpoint` only handles exceptions returned that way versus using a try ... except block.
109	def _handle_inbound_message(self, *, method_name: str, params: Any):	1✔
110	# If the connection is not yet initialized and this is not the initialization request, BSP requires
111	# returning an error for methods (and to discard all notifications).
112	#
113	# Concurrency: This method can be invoked from multiple threads (for each individual request). By returning
114	# an error for all other requests, only the thread running the initialization RPC should be able to proceed.
115	# This ensures that we can safely call `initialize_connection` on the BSPContext with the client-supplied
116	# init parameters without worrying about multiple threads. (Not entirely true though as this does not handle
117	# the client making multiple concurrent initialization RPCs, but which would violate the protocol in any case.)
UNCOV 118	if (	×
119	not self._context.is_connection_initialized
120	and method_name != self._INITIALIZE_METHOD_NAME
121	):
UNCOV 122	return _make_error_future(	×
123	JsonRpcException(
124	code=-32002, message=f"Client must first call `{self._INITIALIZE_METHOD_NAME}`."
125	)
126	)
127
128	# Handle the `build/shutdown` method and `build/exit` notification.
UNCOV 129	if method_name == self._SHUTDOWN_METHOD_NAME:	×
130	# Return no-op success for the `build/shutdown` method. This doesn't actually cause the server to
131	# exit. That will occur once the client sends the `build/exit` notification.
132	return None	×
UNCOV 133	elif method_name == self._EXIT_NOTIFICATION_NAME:	×
134	# The `build/exit` notification directs the BSP server to immediately exit.
135	# The read-dispatch loop will exit once it notices that the inbound handle is closed. So close the
136	# inbound handle (and outbound handle for completeness) and then return to the dispatch loop
137	# to trigger the exit.
138	self._inbound.close()	×
139	self._outbound.close()	×
140	return None	×
141
UNCOV 142	method_mapping = self._handler_mappings.get(method_name)	×
UNCOV 143	if not method_mapping:	×
UNCOV 144	return _make_error_future(JsonRpcMethodNotFound.of(method_name))	×
145
UNCOV 146	try:	×
UNCOV 147	request = method_mapping.request_type.from_json_dict(params)	×
148	except Exception:	×
149	return _make_error_future(JsonRpcInvalidRequest())	×
150
151	# TODO: This should not be necessary: see https://github.com/pantsbuild/pants/issues/15435.
UNCOV 152	self._scheduler_session.new_run_id()	×
153
UNCOV 154	workspace = Workspace(self._scheduler_session)	×
UNCOV 155	params = Params(request, workspace, self._env_name)	×
UNCOV 156	execution_request = self._scheduler_session.execution_request(	×
157	requests=[(method_mapping.response_type, params)],
158	)
UNCOV 159	(result,) = self._scheduler_session.execute(execution_request)	×
160	# Initialize the BSPContext with the client-supplied init parameters. See earlier comment on why this
161	# call to `BSPContext.initialize_connection` is safe.
UNCOV 162	if method_name == self._INITIALIZE_METHOD_NAME:	×
UNCOV 163	self._context.initialize_connection(request, self.notify_client)	×
UNCOV 164	return result.to_json_dict()	×
165
166	# Called by `Endpoint` to dispatch requests and notifications.
167	# TODO: Should probably vendor `Endpoint` so we can detect notifications versus method calls, which
168	# matters when ignoring unknown notifications versus erroring for unknown methods.
169	def __getitem__(self, method_name):	1✔
UNCOV 170	def handler(params):	×
UNCOV 171	return self._handle_inbound_message(method_name=method_name, params=params)	×
172
UNCOV 173	return handler	×
174
175	def notify_client(self, notification: BSPNotification) -> None:	1✔
176	try:	×
177	self._endpoint.notify(notification.notification_name, notification.to_json_dict())	×
178	except Exception as ex:	×
179	_logger.warning(f"Received exception while notifying BSP client: {ex}")	×

pantsbuild / pants / 19529437518

Source File Press 'n' to go to next uncovered line, 'b' for previous

Source File
Press 'n' to go to next uncovered line, 'b' for previous