mozilla / blurts-server / #12904

19%

push

circleci

Prevent search engines from indexing non-prod envs

Previously we provided a robots.txt, telling search engines and
other bots not to proactively visit our stage and dev sites. That's
good for reducing the load, but won't tell them not to add our dev
and stage sites to their search indexes. The X-Robots-Tag: noindex
header does.

Note that this also required removing the robots.txt, since
otherwise that would tell search engines not to crawl those
headers, apparently.

See also:
https://developers.google.com/search/docs/crawling-indexing/block-indexing

282 of 1529 branches covered (18.44%)

Branch coverage included in aggregate %.

6 of 6 new or added lines in 2 files covered. (100.0%)

959 of 4153 relevant lines covered (23.09%)

1.92 hits per line