os-autoinst / openQA / 1887
100%

DEFAULT BRANCH: master
Ran
Jobs 1
Files 58
Run time 2min
Badge
Embed ▾
README BADGES
x

If you need to use a raster PNG badge, change the '.svg' to '.png' in the link

Markdown

Textile

RDoc

HTML

Rst
1887

Pull #535

travis-ci

Adam Williamson 
Treat kernels / initrds as assets, allow download of all assets

this is to make it possible to do 'remote scheduling' (where
the scheduler runs on some machine which does not have access
to the openQA 'factory' directory) of jobs that boot using a
kernel and/or initrd and/or hdd image, just as the 'ISOURL'
feature made it possible for jobs that boot using an ISO. I had
to change a few other things to be happy with it, though.

The convention now is you can request openQA download for any
asset type, by adding _URL to the appropriate setting name. To
download the ISO set 'ISO_URL', to download a kernel set
'KERNEL_URL', and so on. The POST will error out if you try to
download something that is *not* an asset type. This is because
we can't really guess where we should store non-assets.

As with the old ISOURL code, if you set the normal setting
(ISO, KERNEL etc.), that value will be used as the filename of
the downloaded file, otherwise the original filename will be
split out from the URL and used.

Of course, in order to achieve the original goal, I had to make
kernels and initrds be treated as assets :) In garretraziel's
initial patch they were not. We have another good reason to
treat them as assets, though - it will allow us to clean them
up with the limit_assets gru task (this patch does not change
that yet, but it's necessary to make it possible). For now we
treat kernels and initrds as 'other' assets, they could be
given a new asset type instead if we think that's a good idea.
HDDs of course already have an asset type.

It seemed clean to factor the 'figure out the asset type for
a given setting' code in parse_assets_from_settings, because we
need to do that same thing in job_schedule_iso now, to decide
where the asset should be downloaded to (we need to know its
type). So this adds 'asset_type_from_setting' to Utils, and
has both parse_assets_from_settings and job_schedule_iso use
it.

It's also necessary to define OTHER_DIR (like HDD_DIR and
ISO_DIR) in Common.pm so isotovideo can use it. The code for
attaching the appropriate path to the KERNEL and INITRD values
is almost identical to the code for ISO, but it's not easy to
reuse unfortunately.

As suggested by @aaannz, this also adds whitelisting for asset
download domains. Allowing asset download from anywhere on the
internet adds a potential attack vector by compromising the
credentials (especially the API key / secret) of an admin:
an attacker could craft an ISO or other asset to try and break
out of a worker VM and attack the worker host (which may also
be the server, in a small deployment). We mitigate this with
the download_domains config option, which specifies the domains
from which assets can be downloaded; asset download requests
for files from any other domain will be rejected. The check is
done twice: when the API handles the iso POST request (with a
nice error message returned to the user if the check fails) and
also directly by download_asset, as a safety measure in case
someone somehow finds a way to bypass the API and schedule a GRU
task directly.

Finally I added a bit more handling of potential error cases
in download_asset (renamed from download_iso) to try and avoid
triggering the Eternal Gru Failure Loop Of Pain (it now checks
if it can write to the target path, and catches errors when
moving the temporary file to the final location, since we found
out that was possible).
Pull Request #535:

2619 of 4276 relevant lines covered (61.25%)

38.41 hits per line

Jobs
ID Job ID Ran Files Coverage
1 1887.1 58
61.25
 Travis Job 1887.1
Source Files on build 1887
Coverage ∆ File Lines Relevant Covered Missed Hits/Line