xolox / python-rotate-backups / 64

pending completion

Build # 64

Build Type

push

travis-ci

Committed by

xolox

Commit Message

<a href="https://github.com/xolox/python-rotate-backups/commit/<a class=hub.com/xolox/python-rotate-backups/commit/<a class="double-link" href="https://git"><a class=hub.com/xolox/python-rotate-backups/commit/<a class="double-link" href="https://git"><a class=hub.com/xolox/python-rotate-backups/commit/<a class="double-link" href="https://git"><a class=hub.com/xolox/python-rotate-backups/commit/508d1d8404c4a59175743aa785c742d5e63b4ff7">508d1d840">&lt;a href=&quot;https://github.com/xolox/python-rotate-backups/commit/</a><a class="double-link" href="https://github.com/xolox/python-rotate-backups/commit/&lt;a class=&quot;double-link&quot; href=&quot;https://git">&lt;a class=</a>hub.com/xolox/python-rotate-backups/commit/&lt;a class=&quot;double-link&quot; href=&quot;https://git">&lt;a class=</a>hub.com/xolox/python-rotate-backups/commit/508d1d8404c4a59175743aa785c742d5e63b4ff7">508d1d840</a><a href="https://github.com/xolox/python-rotate-backups/commit/508d1d8404c4a59175743aa785c742d5e63b4ff7">&quot;&gt;&amp;lt;a href=&amp;quot;https://github.com/xolox/python-rotate-backups/commit/&lt;/a&gt;&lt;a class=&quot;double-link&quot; href=&quot;https://github.com/xolox/python-rotate-backups/commit/&amp;lt;a class=&amp;quot;double-link&amp;quot; href=&amp;quot;https://git&quot;&gt;&amp;lt;a class=&lt;/a&gt;hub.com/xolox/python-rotate-backups/commit/&amp;lt;a class=&amp;quot;double-link&amp;quot; href=&amp;quot;https://git&quot;&gt;&amp;lt;a class=&lt;/a&gt;hub.com/xolox/python-rotate-backups/commit/&amp;lt;a class=&amp;quot;double-link&amp;quot; href=&amp;quot;https://git&quot;&gt;&amp;lt;a class=&lt;/a&gt;hub.com/xolox/python-rotate-backups/commit/&amp;lt;a class=&amp;quot;double-link&amp;quot; href=&amp;quot;https://git&quot;&gt;&amp;lt;a class=&lt;/a&gt;hub.com/xolox/python-rotate-backups/commit/&amp;lt;a class=&amp;quot;double-link&amp;quot; href=&amp;quot;https://git&quot;&gt;&amp;lt;a class=&lt;/a&gt;hub.com/xolox/python-rotate-backups/commit/&amp;lt;a class=&amp;quot;double-link&amp;quot; href=&amp;quot;https://git&quot;&gt;&amp;lt;a class=&lt;/a&gt;hub.com/xolox/python-rotate-backups/commit/508d1d8404c4a59175743aa785c742d5e63b4ff7&quot;&gt;508d1d840&lt;/a&gt;&lt;a href=&quot;https://github.com/xolox/python-rotate-backups/commit/508d1d8404c4a59175743aa785c742d5e63b4ff7&quot;&gt;&amp;quot;&amp;g... (continued)

Run Details

1 of 1 new or added line in 1 file covered. (100.0%)

297 of 322 relevant lines covered (92.24%)

5.5 hits per line

Source File
Press 'n' to go to next uncovered line, 'b' for previous

87.8

/rotate_backups/cli.py

# rotate-backups: Simple command line interface for backup rotation.
#
# Author: Peter Odding <peter@peterodding.com>
# Last Change: March 25, 2018
# URL: https://github.com/xolox/python-rotate-backups

"""
Usage: rotate-backups [OPTIONS] [DIRECTORY, ..]

Easy rotation of backups based on the Python package by the same name.

To use this program you specify a rotation scheme via (a combination of) the
--hourly, --daily, --weekly, --monthly and/or --yearly options and the
directory (or directories) containing backups to rotate as one or more
positional arguments.

You can rotate backups on a remote system over SSH by prefixing a DIRECTORY
with an SSH alias and separating the two with a colon (similar to how rsync
accepts remote locations).

Instead of specifying directories and a rotation scheme on the command line you
can also add them to a configuration file. For more details refer to the online
documentation (see also the --config option).

Please use the --dry-run option to test the effect of the specified rotation
scheme before letting this program loose on your precious backups! If you don't
test the results using the dry run mode and this program eats more backups than
intended you have no right to complain ;-).

Supported options:

  -M, --minutely=COUNT

    In a literal sense this option sets the number of "backups per minute" to
    preserve during rotation. For most use cases that doesn't make a lot of
    sense :-) but you can combine the --minutely and --relaxed options to
    preserve more than one backup per hour.  Refer to the usage of the -H,
    --hourly option for details about COUNT.

  -H, --hourly=COUNT

    Set the number of hourly backups to preserve during rotation:

    - If COUNT is a number it gives the number of hourly backups to preserve,
      starting from the most recent hourly backup and counting back in time.
    - Alternatively you can provide an expression that will be evaluated to get
      a number (e.g. if COUNT is `7 * 2' the result would be 14).
    - You can also pass `always' for COUNT, in this case all hourly backups are
      preserved.
    - By default no hourly backups are preserved.

  -d, --daily=COUNT

    Set the number of daily backups to preserve during rotation. Refer to the
    usage of the -H, --hourly option for details about COUNT.

  -w, --weekly=COUNT

    Set the number of weekly backups to preserve during rotation. Refer to the
    usage of the -H, --hourly option for details about COUNT.

  -m, --monthly=COUNT

    Set the number of monthly backups to preserve during rotation. Refer to the
    usage of the -H, --hourly option for details about COUNT.

  -y, --yearly=COUNT

    Set the number of yearly backups to preserve during rotation. Refer to the
    usage of the -H, --hourly option for details about COUNT.

  -I, --include=PATTERN

    Only process backups that match the shell pattern given by PATTERN. This
    argument can be repeated. Make sure to quote PATTERN so the shell doesn't
    expand the pattern before it's received by rotate-backups.

  -x, --exclude=PATTERN

    Don't process backups that match the shell pattern given by PATTERN. This
    argument can be repeated. Make sure to quote PATTERN so the shell doesn't
    expand the pattern before it's received by rotate-backups.

  -j, --parallel

    Remove backups in parallel, one backup per mount point at a time. The idea
    behind this approach is that parallel rotation is most useful when the
    files to be removed are on different disks and so multiple devices can be
    utilized at the same time.

    Because mount points are per system the -j, --parallel option will also
    parallelize over backups located on multiple remote systems.

  -p, --prefer-recent

    By default the first (oldest) backup in each time slot is preserved. If
    you'd prefer to keep the most recent backup in each time slot instead then
    this option is for you.

  -r, --relaxed

    By default the time window for each rotation scheme is enforced (this is
    referred to as strict rotation) but the -r, --relaxed option can be used
    to alter this behavior. The easiest way to explain the difference between
    strict and relaxed rotation is using an example:

    - When using strict rotation and the number of hourly backups to preserve
      is three, only backups created in the relevant time window (the hour of
      the most recent backup and the two hours leading up to that) will match
      the hourly frequency.

    - When using relaxed rotation the three most recent backups will all match
      the hourly frequency (and thus be preserved), regardless of the
      calculated time window.

    If the explanation above is not clear enough, here's a simple way to decide
    whether you want to customize this behavior or not:

    - If your backups are created at regular intervals and you never miss an
      interval then strict rotation (the default) is probably the best choice.

    - If your backups are created at irregular intervals then you may want to
      use the -r, --relaxed option in order to preserve more backups.

  -i, --ionice=CLASS

    Use the `ionice' program to set the I/O scheduling class and priority of
    the `rm' invocations used to remove backups. CLASS is expected to be one of
    the values `idle', `best-effort' or `realtime'. Refer to the man page of
    the `ionice' program for details about these values.

  -c, --config=FILENAME

    Load configuration from FILENAME. If this option isn't given the following
    default locations are searched for configuration files:

    - /etc/rotate-backups.ini and /etc/rotate-backups.d/*.ini
    - ~/.rotate-backups.ini and ~/.rotate-backups.d/*.ini
    - ~/.config/rotate-backups.ini and ~/.config/rotate-backups.d/*.ini

    Any available configuration files are loaded in the order given above, so
    that sections in user-specific configuration files override sections by the
    same name in system-wide configuration files. For more details refer to the
    online documentation.

  -u, --use-sudo

    Enable the use of `sudo' to rotate backups in directories that are not
    readable and/or writable for the current user (or the user logged in to a
    remote system over SSH).

  -n, --dry-run

    Don't make any changes, just print what would be done. This makes it easy
    to evaluate the impact of a rotation scheme without losing any backups.

  -v, --verbose

    Increase logging verbosity (can be repeated).

  -q, --quiet

    Decrease logging verbosity (can be repeated).

  -h, --help

    Show this message and exit.
"""

# Standard library modules.
import getopt
import sys

# External dependencies.
import coloredlogs
from executor import validate_ionice_class
from humanfriendly import parse_path, pluralize
from humanfriendly.terminal import usage
from verboselogs import VerboseLogger

# Modules included in our package.
from rotate_backups import (
    RotateBackups,
    coerce_location,
    coerce_retention_period,
    load_config_file,
)

# Initialize a logger.
logger = VerboseLogger(__name__)


def main():
    """Command line interface for the ``rotate-backups`` program."""
    coloredlogs.install(syslog=True)
    # Command line option defaults.
    rotation_scheme = {}
    kw = dict(include_list=[], exclude_list=[])
    parallel = False
    use_sudo = False
    # Internal state.
    selected_locations = []
    # Parse the command line arguments.
    try:
        options, arguments = getopt.getopt(sys.argv[1:], 'M:H:d:w:m:y:I:x:jpri:c:r:unvqh', [
            'minutely=', 'hourly=', 'daily=', 'weekly=', 'monthly=', 'yearly=',
            'include=', 'exclude=', 'parallel', 'prefer-recent', 'relaxed',
            'ionice=', 'config=', 'use-sudo', 'dry-run', 'verbose', 'quiet',
            'help',
        ])
        for option, value in options:
            if option in ('-M', '--minutely'):
                rotation_scheme['minutely'] = coerce_retention_period(value)
            elif option in ('-H', '--hourly'):
                rotation_scheme['hourly'] = coerce_retention_period(value)
            elif option in ('-d', '--daily'):
                rotation_scheme['daily'] = coerce_retention_period(value)
            elif option in ('-w', '--weekly'):
                rotation_scheme['weekly'] = coerce_retention_period(value)
            elif option in ('-m', '--monthly'):
                rotation_scheme['monthly'] = coerce_retention_period(value)
            elif option in ('-y', '--yearly'):
                rotation_scheme['yearly'] = coerce_retention_period(value)
            elif option in ('-I', '--include'):
                kw['include_list'].append(value)
            elif option in ('-x', '--exclude'):
                kw['exclude_list'].append(value)
            elif option in ('-j', '--parallel'):
                parallel = True
            elif option in ('-p', '--prefer-recent'):
                kw['prefer_recent'] = True
            elif option in ('-r', '--relaxed'):
                kw['strict'] = False
            elif option in ('-i', '--ionice'):
                value = validate_ionice_class(value.lower().strip())
                kw['io_scheduling_class'] = value
            elif option in ('-c', '--config'):
                kw['config_file'] = parse_path(value)
            elif option in ('-u', '--use-sudo'):
                use_sudo = True
            elif option in ('-n', '--dry-run'):
                logger.info("Performing a dry run (because of %s option) ..", option)
                kw['dry_run'] = True
            elif option in ('-v', '--verbose'):
                coloredlogs.increase_verbosity()
            elif option in ('-q', '--quiet'):
                coloredlogs.decrease_verbosity()
            elif option in ('-h', '--help'):
                usage(__doc__)
                return
            else:
                assert False, "Unhandled option! (programming error)"
        if rotation_scheme:
            logger.verbose("Rotation scheme defined on command line: %s", rotation_scheme)
        if arguments:
            # Rotation of the locations given on the command line.
            location_source = 'command line arguments'
            selected_locations.extend(coerce_location(value, sudo=use_sudo) for value in arguments)
        else:
            # Rotation of all configured locations.
            location_source = 'configuration file'
            selected_locations.extend(
                location
                for location, rotation_scheme, options
                in load_config_file(kw.get('config_file'))
            )
        # Inform the user which location(s) will be rotated.
        if selected_locations:
            logger.verbose("Selected %s based on %s:",
                           pluralize(len(selected_locations), "location"),
                           location_source)
            for number, location in enumerate(selected_locations, start=1):
                logger.verbose(" %i. %s", number, location)
        else:
            # Show the usage message when no directories are given nor configured.
            logger.verbose("No location(s) to rotate selected.")
            usage(__doc__)
            return
    except Exception as e:
        logger.error("%s", e)
        sys.exit(1)
    # Rotate the backups in the selected directories.
    program = RotateBackups(rotation_scheme, **kw)
    if parallel:
        program.rotate_concurrent(*selected_locations)
    else:
        for location in selected_locations:
            program.rotate_backups(location)

1	# rotate-backups: Simple command line interface for backup rotation.
2	#
3	# Author: Peter Odding <peter@peterodding.com>
4	# Last Change: March 25, 2018
5	# URL: https://github.com/xolox/python-rotate-backups
6
7	"""	6✔
8	Usage: rotate-backups [OPTIONS] [DIRECTORY, ..]
9
10	Easy rotation of backups based on the Python package by the same name.
11
12	To use this program you specify a rotation scheme via (a combination of) the
13	--hourly, --daily, --weekly, --monthly and/or --yearly options and the
14	directory (or directories) containing backups to rotate as one or more
15	positional arguments.
16
17	You can rotate backups on a remote system over SSH by prefixing a DIRECTORY
18	with an SSH alias and separating the two with a colon (similar to how rsync
19	accepts remote locations).
20
21	Instead of specifying directories and a rotation scheme on the command line you
22	can also add them to a configuration file. For more details refer to the online
23	documentation (see also the --config option).
24
25	Please use the --dry-run option to test the effect of the specified rotation
26	scheme before letting this program loose on your precious backups! If you don't
27	test the results using the dry run mode and this program eats more backups than
28	intended you have no right to complain ;-).
29
30	Supported options:
31
32	-M, --minutely=COUNT
33
34	In a literal sense this option sets the number of "backups per minute" to
35	preserve during rotation. For most use cases that doesn't make a lot of
36	sense :-) but you can combine the --minutely and --relaxed options to
37	preserve more than one backup per hour. Refer to the usage of the -H,
38	--hourly option for details about COUNT.
39
40	-H, --hourly=COUNT
41
42	Set the number of hourly backups to preserve during rotation:
43
44	- If COUNT is a number it gives the number of hourly backups to preserve,
45	starting from the most recent hourly backup and counting back in time.
46	- Alternatively you can provide an expression that will be evaluated to get
47	a number (e.g. if COUNT is `7 * 2' the result would be 14).
48	- You can also pass `always' for COUNT, in this case all hourly backups are
49	preserved.
50	- By default no hourly backups are preserved.
51
52	-d, --daily=COUNT
53
54	Set the number of daily backups to preserve during rotation. Refer to the
55	usage of the -H, --hourly option for details about COUNT.
56
57	-w, --weekly=COUNT
58
59	Set the number of weekly backups to preserve during rotation. Refer to the
60	usage of the -H, --hourly option for details about COUNT.
61
62	-m, --monthly=COUNT
63
64	Set the number of monthly backups to preserve during rotation. Refer to the
65	usage of the -H, --hourly option for details about COUNT.
66
67	-y, --yearly=COUNT
68
69	Set the number of yearly backups to preserve during rotation. Refer to the
70	usage of the -H, --hourly option for details about COUNT.
71
72	-I, --include=PATTERN
73
74	Only process backups that match the shell pattern given by PATTERN. This
75	argument can be repeated. Make sure to quote PATTERN so the shell doesn't
76	expand the pattern before it's received by rotate-backups.
77
78	-x, --exclude=PATTERN
79
80	Don't process backups that match the shell pattern given by PATTERN. This
81	argument can be repeated. Make sure to quote PATTERN so the shell doesn't
82	expand the pattern before it's received by rotate-backups.
83
84	-j, --parallel
85
86	Remove backups in parallel, one backup per mount point at a time. The idea
87	behind this approach is that parallel rotation is most useful when the
88	files to be removed are on different disks and so multiple devices can be
89	utilized at the same time.
90
91	Because mount points are per system the -j, --parallel option will also
92	parallelize over backups located on multiple remote systems.
93
94	-p, --prefer-recent
95
96	By default the first (oldest) backup in each time slot is preserved. If
97	you'd prefer to keep the most recent backup in each time slot instead then
98	this option is for you.
99
100	-r, --relaxed
101
102	By default the time window for each rotation scheme is enforced (this is
103	referred to as strict rotation) but the -r, --relaxed option can be used
104	to alter this behavior. The easiest way to explain the difference between
105	strict and relaxed rotation is using an example:
106
107	- When using strict rotation and the number of hourly backups to preserve
108	is three, only backups created in the relevant time window (the hour of
109	the most recent backup and the two hours leading up to that) will match
110	the hourly frequency.
111
112	- When using relaxed rotation the three most recent backups will all match
113	the hourly frequency (and thus be preserved), regardless of the
114	calculated time window.
115
116	If the explanation above is not clear enough, here's a simple way to decide
117	whether you want to customize this behavior or not:
118
119	- If your backups are created at regular intervals and you never miss an
120	interval then strict rotation (the default) is probably the best choice.
121
122	- If your backups are created at irregular intervals then you may want to
123	use the -r, --relaxed option in order to preserve more backups.
124
125	-i, --ionice=CLASS
126
127	Use the `ionice' program to set the I/O scheduling class and priority of
128	the `rm' invocations used to remove backups. CLASS is expected to be one of
129	the values `idle', `best-effort' or `realtime'. Refer to the man page of
130	the `ionice' program for details about these values.
131
132	-c, --config=FILENAME
133
134	Load configuration from FILENAME. If this option isn't given the following
135	default locations are searched for configuration files:
136
137	- /etc/rotate-backups.ini and /etc/rotate-backups.d/*.ini
138	- ~/.rotate-backups.ini and ~/.rotate-backups.d/*.ini
139	- ~/.config/rotate-backups.ini and ~/.config/rotate-backups.d/*.ini
140
141	Any available configuration files are loaded in the order given above, so
142	that sections in user-specific configuration files override sections by the
143	same name in system-wide configuration files. For more details refer to the
144	online documentation.
145
146	-u, --use-sudo
147
148	Enable the use of `sudo' to rotate backups in directories that are not
149	readable and/or writable for the current user (or the user logged in to a
150	remote system over SSH).
151
152	-n, --dry-run
153
154	Don't make any changes, just print what would be done. This makes it easy
155	to evaluate the impact of a rotation scheme without losing any backups.
156
157	-v, --verbose
158
159	Increase logging verbosity (can be repeated).
160
161	-q, --quiet
162
163	Decrease logging verbosity (can be repeated).
164
165	-h, --help
166
167	Show this message and exit.
168	"""
169
170	# Standard library modules.
171	import getopt	6✔
172	import sys	6✔
173
174	# External dependencies.
175	import coloredlogs	6✔
176	from executor import validate_ionice_class	6✔
177	from humanfriendly import parse_path, pluralize	6✔
178	from humanfriendly.terminal import usage	6✔
179	from verboselogs import VerboseLogger	6✔
180
181	# Modules included in our package.
182	from rotate_backups import (	6✔
183	RotateBackups,
184	coerce_location,
185	coerce_retention_period,
186	load_config_file,
187	)
188
189	# Initialize a logger.
190	logger = VerboseLogger(__name__)	6✔
191
192
193	def main():	6✔
194	"""Command line interface for the ``rotate-backups`` program."""
195	coloredlogs.install(syslog=True)	6✔
196	# Command line option defaults.
197	rotation_scheme = {}	6✔
198	kw = dict(include_list=[], exclude_list=[])	6✔
199	parallel = False	6✔
200	use_sudo = False	6✔
201	# Internal state.
202	selected_locations = []	6✔
203	# Parse the command line arguments.
204	try:	6✔
205	options, arguments = getopt.getopt(sys.argv[1:], 'M:H:d:w:m:y:I:x:jpri:c:r:unvqh', [	6✔
206	'minutely=', 'hourly=', 'daily=', 'weekly=', 'monthly=', 'yearly=',
207	'include=', 'exclude=', 'parallel', 'prefer-recent', 'relaxed',
208	'ionice=', 'config=', 'use-sudo', 'dry-run', 'verbose', 'quiet',
209	'help',
210	])
211	for option, value in options:	6✔
212	if option in ('-M', '--minutely'):	6✔
213	rotation_scheme['minutely'] = coerce_retention_period(value)	6✔
214	elif option in ('-H', '--hourly'):	6✔
215	rotation_scheme['hourly'] = coerce_retention_period(value)	6✔
216	elif option in ('-d', '--daily'):	6✔
217	rotation_scheme['daily'] = coerce_retention_period(value)	6✔
218	elif option in ('-w', '--weekly'):	6✔
219	rotation_scheme['weekly'] = coerce_retention_period(value)	6✔
220	elif option in ('-m', '--monthly'):	6✔
221	rotation_scheme['monthly'] = coerce_retention_period(value)	6✔
222	elif option in ('-y', '--yearly'):	6✔
223	rotation_scheme['yearly'] = coerce_retention_period(value)	6✔
224	elif option in ('-I', '--include'):	6✔
225	kw['include_list'].append(value)	6✔
226	elif option in ('-x', '--exclude'):	6✔
227	kw['exclude_list'].append(value)	6✔
228	elif option in ('-j', '--parallel'):	6✔
229	parallel = True	6✔
230	elif option in ('-p', '--prefer-recent'):	6✔
231	kw['prefer_recent'] = True	6✔
232	elif option in ('-r', '--relaxed'):	6✔
233	kw['strict'] = False	6✔
234	elif option in ('-i', '--ionice'):	6✔
235	value = validate_ionice_class(value.lower().strip())	6✔
236	kw['io_scheduling_class'] = value	6✔
237	elif option in ('-c', '--config'):	6✔
238	kw['config_file'] = parse_path(value)	6✔
239	elif option in ('-u', '--use-sudo'):	6✔
240	use_sudo = True	×
241	elif option in ('-n', '--dry-run'):	6✔
242	logger.info("Performing a dry run (because of %s option) ..", option)	6✔
243	kw['dry_run'] = True	6✔
244	elif option in ('-v', '--verbose'):	6✔
245	coloredlogs.increase_verbosity()	6✔
246	elif option in ('-q', '--quiet'):	×
247	coloredlogs.decrease_verbosity()	×
248	elif option in ('-h', '--help'):	×
249	usage(__doc__)	×
250	return	×
251	else:
252	assert False, "Unhandled option! (programming error)"	×
253	if rotation_scheme:	6✔
254	logger.verbose("Rotation scheme defined on command line: %s", rotation_scheme)	6✔
255	if arguments:	6✔
256	# Rotation of the locations given on the command line.
257	location_source = 'command line arguments'	6✔
258	selected_locations.extend(coerce_location(value, sudo=use_sudo) for value in arguments)	6✔
259	else:
260	# Rotation of all configured locations.
261	location_source = 'configuration file'	6✔
262	selected_locations.extend(	6✔
263	location
264	for location, rotation_scheme, options
265	in load_config_file(kw.get('config_file'))
266	)
267	# Inform the user which location(s) will be rotated.
268	if selected_locations:	6✔
269	logger.verbose("Selected %s based on %s:",	6✔
270	pluralize(len(selected_locations), "location"),
271	location_source)
272	for number, location in enumerate(selected_locations, start=1):	6✔
273	logger.verbose(" %i. %s", number, location)	6✔
274	else:
275	# Show the usage message when no directories are given nor configured.
276	logger.verbose("No location(s) to rotate selected.")	×
277	usage(__doc__)	×
278	return	×
279	except Exception as e:	6✔
280	logger.error("%s", e)	6✔
281	sys.exit(1)	6✔
282	# Rotate the backups in the selected directories.
283	program = RotateBackups(rotation_scheme, **kw)	6✔
284	if parallel:	6✔
285	program.rotate_concurrent(*selected_locations)	6✔
286	else:
287	for location in selected_locations:	6✔
288	program.rotate_backups(location)	6✔

xolox / python-rotate-backups / 64

Source File Press 'n' to go to next uncovered line, 'b' for previous

Source File
Press 'n' to go to next uncovered line, 'b' for previous