d00rman / restbase

91%

master: 84%

LAST BUILD ON BRANCH feeds/split

branch: feeds/split

CHANGE BRANCH

x

Reset

• feeds/split
• acces-check/no-cache
• action-template
• action/param-fix
• be-services/strip-headers
• blacklist/all-user-pages/commons
• blacklist/cebwiki
• blacklist/commons
• blacklist/commons/user-oxyman
• blacklist/cyberpower-report
• blacklist/de-big-bang
• blacklist/de-fr
• blacklist/dewiki
• blacklist/enwiki-bbot
• blacklist/fix-regex
• blacklist/more
• blacklist/various
• block-user-page
• blubber
• bodyonly/log-fix
• bodyonly/log-message
• bump-cass-v1.1.0
• cass-077/playground
• citoid/disable-check
• citoid/x-ample-change
• cors/expose-headers
• coverage
• deprecate/scrubWikitext
• deps/update
• docs/logo_url
• docs/typo
• eslint
• exec-server
• extract_html-xample
• feed/body-tfa-exists
• feed/increase-ttl
• feed/table-version-inc
• flagged-revs
• graphoid/monitoring-hash
• headers/lowercase-ua
• htcp/small-improvement
• key_value/rev_policy
• lang-var/add-title
• last-switch
• lists-doc-fix
• log_4xx
• logs-revert
• ma-headers
• master
• math/x-store-ct
• mathoid/beta-uri
• mathoid/open-post
• mathoid/png
• mathoid/post-routes
• mathoid/purging
• mathoid/resources
• mathoid/routes
• mathoid/spec-module
• mathoid/test/formats
• mathoid/texvcinfo
• mcs/disable-onthisday-check
• mcs/no-definition-storage
• metrics/keep-alive
• metrics/mediarequests/fix-indent
• metrics/rate-limiting
• mobileapps-updates
• mobileapps/feed-random
• mobileapps/pregeneration
• mobileapps/routes
• mobileapps/spec-mod
• mobileapps/storage
• mobileapps/test-mobile-sections
• mobileapps/unstable
• mod-loading
• mods/cleanup
• no-ng
• no-purges
• node-4.2.4
• node/4.3
• node4
• node446
• package/clean-up
• page-title-removal
• paging
• parsoid-temp/bug-fix
• parsoid-time_to_live
• parsoid/421-fallback
• parsoid/bodyonly-fix
• parsoid/ensure-headers
• parsoid/flag-fix
• parsoid/lint-wikitext
• parsoid/log-ve
• parsoid/log-ve-fix4
• parsoid/log-ve-fix5
• parsoid/no-html-tid
• parsoid/php
• parsoid/php/shim
• parsoid/remove-ve-logging
• parsoid/stash
• parsoid/stash-bucket
• parsoid/stash-fix
• parsoid/stash-ratelimit
• parsoid/stash-rev-check
• parsoid/tid-suffix
• parsoidproxy/mirror-html2html
• parsoidproxy/mirror-in-split
• pcs/metadata/no-storage
• pr-166-merge
• probe/visit-async
• project/sqlite
• ratelimit/enforce
• rb-mod-cass-table
• readme/node4
• recommendation-api-route-fix
• recommendation-api-spec
• redirect/include-anchor
• rel/0.13.0
• rel/0.15.1
• rel/v0.14.3
• rel/v0.14.4
• rel/v0.15.0
• rel/v0.15.2
• rel/v0.16.2
• rel/v0.16.3
• rel/v0.16.4
• rel/v0.16.5
• rel/v0.16.8
• rel/v0.18.0
• rel/v0.19.1
• rel/v0.19.3
• rel/v1.0.0
• rel/v1.1.0
• rel/v1.1.1
• rel/v1.1.2
• release/v0.11.2
• release/v0.12.0
• remove-debian
• remove-extend
• remove-updates
• remove-wikifeeds
• req/body-parse
• requestid
• revert/pr750
• revert/pr756
• revisions/check-fix
• save-api
• server/body-parse
• siteinfo/fix
• skip-updates
• summary-test
• summary/https-fix
• summary/v1.1.1
• switch-to-proton
• template/improvements
• templated-subspecs
• test/cass-0.8.14
• test/content-type
• test/with-cass-076
• tests/electron-url
• tests/enable-graphoid
• tests/frontend-no-skip-res
• tests/graphoid
• tests/https
• tests/last-rev
• tests/router-logging-fix
• title-revs
• transform/limits
• translation-api
• travis/cass-3.11.4
• travis/iojs-2-5
• travis/node10
• travis/node8
• update-deps
• user-agent
• v0-7-12/update_deps
• validator/fix
• ve-req
• wikidata/math
• wt2lint/bug-fix

Build # 517

Build Type

push

travis-ci

Committed by d00rman

Commit Message

Feeds: Split the aggregated feed request and cache it

Instead of requesting the whole response of the aggregated feed endpoint
in MCS, it is much faster to request its individual pieces and then
assemble them in RESTBase. The process happens in parallel, so different
MCS workers get to work on a part of the response.

Moreover, in order to decrease the amount of requests to MCS, store the
response using the `ttl` revision retention policy. Currently, we begin
with the conservative ttl of 10 minutes, but it is likely we can
increase it in the future. The random component needs to change more
frequently than that, so whenever we have a cache hit, we re-request a
random title from MCS and incorporate it in the new response. We also
tell Varnish to cache the response for 30 seconds. This means that the
number of requests will be:
- 1 request for the random title every 30 seconds
- 5 requests for the various response components every 10 minutes

Run Details

1281 of 1402 relevant lines covered (91.37%)

64.19 hits per line