pantsbuild / pants / 8135
0%
main: 93%

LAST BUILD BRANCH: fix_uv_sources_config
DEFAULT BRANCH: main
Ran
Jobs 8
Files 0
Run time –
Badge
Embed ▾
README BADGES
x

If you need to use a raster PNG badge, change the '.svg' to '.png' in the link

Markdown

Textile

RDoc

HTML

Rst
8135

push

travis-ci

kwlzn 
[pantsd] Initial implementation of the pants nailgun service.

This is a python/pants nailgun server implementation that is intended to speed up the startup and runtime of pants by way of keeping warm state in a running daemon and using a client/server fork-on-run model. This utilizes the Nailgun protocol - a generic client/server protocol for running commands and marshalling stdio back/forth - to accomplish this; i.e. it effectively brings the same nailgun mechanics we use to execute code in a resident JVM except against a resident python interpreter with the primary intent of running pants. The current implementation is presented without any  intentional state caching optimization (clean runs) and works as-is with the native nailgun client for an already order-of-magnitude speedup of no-op and help-advanced runtime (see testing section below). Two subsequent components are expected to arrive shortly to fully wire this up: the Pants Daemon itself and a closed-loop client implementation that will utilize this service.

- Initial implementations of DaemonPantsRunner, PailgunService, PailgunServer et al - "Pailgun" is used to disambiguate between the canonical jvm-oriented "nailgun" usage in pants vs the python-oriented implementation.
- Break-out a new pants.java.nailgun_io module with NailgunStream{Reader,Writer} for general use.
- Tests!

Testing Done:
1) manual E2E testing in the pantsd dev branch.

2) CI green @ https://travis-ci.org/pantsbuild/pants/builds/92886226

3)
    [illuminati pants (kwlzn/pantsd/pailgun)]$ ptcm
    00:37:48 00:00 [main]
    00:37:48 00:00   [setup]
    00:37:48 00:00     [parse]INFO] Operating on changed 43 target(s):
      * src/python/pants/bin
      * src/python/pants/java:executor
      * src/python/pants/java:nailgun_client
      * src/python/pants/java:nailgun_executor
      * src/python/pants/java:nailgun_io
      * src/python/pants/java:nailgun_protocol
      * src/python/pants/java:util
      * src/python/pants/pantsd/service:fs_event_service
      * src/python/pants/pantsd/service:pailgun_service
      * src/python/pants/pantsd/service:pants_service
      * src/python/pants/pantsd:pailgun_server
      * src/python/pants/pantsd:process_manager
      * src/python/pants/pantsd:watchman
      * src/python/pants/pantsd:watchman_client
      * src/python/pants/util:contextutil
      * tests/python/pants_test/java
      * tests/python/pants_test/java:executor
      * tests/python/pants_test/java:nailgun_client
      * tests/python/pants_test/java:nailgun_integration
      * tests/python/pants_test/java:nailgun_io
      * tests/python/pants_test/java:nailgun_protocol
      * tests/python/pants_test/java:util
      * tests/python/pants_test/pantsd/service:fs_event_service
      * tests/python/pants_test/pantsd/service:pailgun_service
      * tests/python/pants_test/pantsd/service:pants_service
      * tests/python/pants_test/pantsd:pailgun_server
      * tests/python/pants_test/pantsd:process_manager
      * tests/python/pants_test/pantsd:test_deps
      * tests/python/pants_test/pantsd:watchman
      * tests/python/pants_test/pantsd:watchman_client
      * tests/python/pants_test/util
      * tests/python/pants_test/util:contextutil
      * tests/python/pants_test/util:dirutil
      * tests/python/pants_test/util:eval
      * tests/python/pants_test/util:fileutil
      * tests/python/pants_test/util:filtering
      * tests/python/pants_test/util:memo
      * tests/python/pants_test/util:meta
      * tests/python/pants_test/util:osutil
      * tests/python/pants_test/util:socket
      * tests/python/pants_test/util:strutil
      * tests/python/pants_test/util:timeout
      * tests/python/pants_test/util:xml_parser

                   Executing tasks in goals: bootstrap -> imports -> unpack-jars -> deferred-sources -> jvm-platform-validate -> gen -> resolve -> resources -> compile -> test -> test-changed
    00:37:48 00:00   [bootstrap]
    00:37:48 00:00     [bootstrap-jvm-tools]
    00:37:48 00:00   [imports]
    00:37:48 00:00     [ivy-imports]
    00:37:48 00:00   [unpack-jars]
    00:37:48 00:00     [unpack-jars]
    00:37:48 00:00     [unpack-libs]
    00:37:48 00:00   [deferred-sources]
    00:37:48 00:00     [deferred-sources]
    00:37:48 00:00   [jvm-platform-validate]
    00:37:48 00:00     [jvm-platform-validate]
    00:37:48 00:00   [gen]
    00:37:48 00:00     [thrift]
    00:37:48 00:00     [protoc]
    00:37:48 00:00     [antlr]
    00:37:48 00:00     [ragel]
    00:37:48 00:00     [jaxb]
    00:37:48 00:00     [wire]
    00:37:48 00:00     [aapt]
    00:37:48 00:00     [scrooge]
    00:37:48 00:00     [spindle]
    00:37:48 00:00   [resolve]
    00:37:48 00:00     [ivy]
    00:37:48 00:00     [go]
    00:37:48 00:00     [npm]
    00:37:48 00:00   [resources]
    00:37:48 00:00     [prepare]
    00:37:48 00:00     [services]
    00:37:48 00:00   [compile]
    00:37:48 00:00     [compile]
    00:37:48 00:00     [jvm]
    00:37:48 00:00       [jvm-compilers]
    00:37:48 00:00         [zinc-pre]
    00:37:48 00:00         [zinc-post]
    00:37:48 00:00     [jvm-dep-check]
    00:37:49 00:01     [checkstyle]
    00:37:49 00:01     [scalastyle]
    00:37:49 00:01     [cpp]
    00:37:49 00:01     [go]
    00:37:49 00:01     [python-eval]
                       Invalidated 1 target..
    00:37:49 00:01           [chroot]
    00:37:50 00:02           [eval]

    00:37:51 00:03     [pythonstyle]
                       Invalidated 2 targets.
    00:37:51 00:03   [test]
    00:37:51 00:03     [run_prep_command]
    00:37:51 00:03     [test]
    00:37:51 00:03     [pytest]
    00:37:51 00:03       [run]

    00:37:51 00:03         [chroot]
                         ============== test session starts ===============
                         platform darwin -- Python 2.7.9 -- py-1.4.30 -- pytest-2.6.4
                         plugins: timeout
                         collected 276 items

                         tests/python/pants_test/java/test_nailgun_integration.py .
                         tests/python/pants_test/util/test_eval.py ......
                         tests/python/pants_test/pantsd/test_watchman_client.py .
                         tests/python/pants_test/java/test_nailgun_client.py ...........
                         tests/python/pants_test/util/test_memo.py ..............
                         tests/python/pants_test/util/test_meta.py ...
                         tests/python/pants_test/pantsd/service/test_pants_service.py ...
                         tests/python/pants_test/pantsd/test_process_manager.py .................................................
                         tests/python/pants_test/java/test_nailgun_protocol.py ..................
                         tests/python/pants_test/util/test_dirutil.py .................
                         tests/python/pants_test/pantsd/service/test_pailgun_service.py .
                         tests/python/pants_test/util/test_socket.py ....
                         tests/python/pants_test/util/test_contextutil.py .................
                         tests/python/pants_test/util/test_fileutil.py ..
                         tests/python/pants_test/util/test_osutil.py ....
                         tests/python/pants_test/util/test_strutil.py ..
                         tests/python/pants_test/util/test_xml_parser.py ...........
                         tests/python/pants_test/util/test_timeout.py ....
                         tests/python/pants_test/java/test_executor.py .......
                         tests/python/pants_test/util/test_filtering.py ......
                         tests/python/pants_test/pantsd/test_pailgun_server.py .....
                         tests/python/pants_test/pantsd/service/test_fs_event_service.py .........
                         tests/python/pants_test/java/test_util.py ....
                         tests/python/pants_test/java/test_nailgun_io.py ..........
                         tests/python/pants_test/java/distribution/test_distribution.py ..........................................
                         tests/python/pants_test/java/jar/test_manifest.py ....
                         tests/python/pants_test/pantsd/test_watchman.py .....................

                         ========== 276 passed in 26.44 seconds ===========
                         Name                                                                      Stmts   Miss Branch BrMiss  Cover
                         -----------------------------------------------------------------------------------------------------------
                         src/python/pants/java/nailgun_client                                         74      1      8      1    98%
                         src/python/pants/java/nailgun_io                                             61      2     12      1    96%
                         src/python/pants/java/nailgun_protocol                                       98      1     24      1    98%
                         src/python/pants/pantsd/pailgun_server                                       63      5      6      3    88%
                         src/python/pants/pantsd/process_manager                                     214      8     42      2    96%
                         src/python/pants/pantsd/service/fs_event_service                             43      0     12      0   100%
                         src/python/pants/pantsd/service/pailgun_service                              31     10      6      4    62%
                         src/python/pants/pantsd/service/pants_service                                15      0      2      1    94%
                         src/python/pants/pantsd/watchman                                             78      0     20      0   100%
                         src/python/pants/pantsd/watchman_client                                      26     21      8      8    15%
                         src/python/pants/util/contextutil                                            98     15     22      8    81%
                         src/python/pants/util/dirutil                                               137     29     54     19    75%
                         src/python/pants/util/socket                                                 21      0      4      1    96%
                         ...
                         -----------------------------------------------------------------------------------------------------------
                         TOTAL                                                                      2570    134    270     60    93%

    00:38:22 00:34     [junit]
    00:38:22 00:34     [go]
    00:38:22 00:34     [npm]
    00:38:22 00:34   [test-changed]
    00:38:22 00:34     [test-changed]
    00:38:22 00:34   [complete]
                   SUCCESS

 4) illustrating an ~order-of-magnitude speedup on no-op runs:

    [illuminati hammerpants]$ gcc -o ng ng.c
    [illuminati hammerpants]$ ln -s ng ./pants
    [illuminati hammerpants]$ export NAILGUN_PORT=`cat ~/dev/pants/.pids/pantsd/socket_pailgun`
    [illuminati hammerpants]$ time ./pants
    [pantsd]
    No goals specified.
    Use `pants goals` to list goals.
    Use `pants help` to get help.

    real    0m0.070s
    user    0m0.001s
    sys     0m0.004s

    [illuminati hammerpants]$ time ./pants help-advanced > /dev/null
    [pantsd]

    real    0m0.077s
    user    0m0.001s
    sys     0m0.003s

vs normal startup

    [illuminati pants (kwlzn/pantsd/base)]$ time ./pants
    No goals specified.
    Use `pants goals` to list goals.
    Use `pants help` to get help.

    real  0m0.753s
    user  0m0.341s
    sys   0m0.213s

    [illuminati pants (kwlzn/pantsd/base)]$ time ./pants help-advanced > /dev/null

    real  0m1.350s
    user  0m0.387s
    sys   0m0.586s

Bugs closed: 2617

Reviewed at https://rbcommons.com/s/twitter/r/3171/
Jobs
ID Job ID Ran Files Coverage
1 8135.1 (CI_FLAGS="-cjlpn 'Various pants self checks'") 0
Travis Job 8135.1
4 8135.4 (CI_FLAGS="-fkmsrcjlp 'Python contrib tests'") 0
Travis Job 8135.4
5 8135.5 (CI_FLAGS="-fkmsrjlpn -i 0/6 'Python integration tests for pants - shard 1'") 0
Travis Job 8135.5
6 8135.6 (CI_FLAGS="-fkmsrjlpn -i 1/6 'Python integration tests for pants - shard 2'") 0
Travis Job 8135.6
7 8135.7 (CI_FLAGS="-fkmsrjlpn -i 2/6 'Python integration tests for pants - shard 3'") 0
Travis Job 8135.7
8 8135.8 (CI_FLAGS="-fkmsrjlpn -i 3/6 'Python integration tests for pants - shard 4'") 0
Travis Job 8135.8
9 8135.9 (CI_FLAGS="-fkmsrjlpn -i 4/6 'Python integration tests for pants - shard 5'") 0
Travis Job 8135.9
10 8135.10 (CI_FLAGS="-fkmsrjlpn -i 5/6 'Python integration tests for pants - shard 6'") 0
Travis Job 8135.10
Source Files on build 8135
Detailed source file information is not available for this build.