ICRAR / ngas

69%

master: 69%

LAST BUILD ON BRANCH issue-51

branch: issue-51

CHANGE BRANCH

x

Reset

• issue-51
• 98-query-ignore
• NGAS-163
• aiosmtpd
• avoid-installing-bsddb3-in-py27
• bbcp_fixes
• c_client_warning_fixes
• centos7-compatibility-changes
• clone-fix
• clone-order-fixes
• correct-janitor-init
• correct-logging-c
• coverage-stability
• direct_sql
• doc-updates
• docs-fix
• document-future-removal-of-27
• fits_optional_arcfile
• fix-requests-dependency-specs
• fs-monitor
• fs-monitor-fixes
• github-actions
• improve-register-error-message
• issue-103
• issue-13
• issue-38
• issue-44
• issue-50
• janitor-thread-startup
• macos-hostnames
• macos-travis-fix
• macos-travis-update
• master
• netifaces-fix
• next
• no-create-venv
• none-reg-parameters
• partner_sites
• plugins-importing
• py311
• py39
• python-client-improvements
• relaxed-netifaces
• release-12.0
• remove-2.7-builds
• remove-test-generic-registration-plugin
• requests-pinning
• richer-archiving-event
• separate-test-servers
• servers-flag
• small-improvements
• smtp-testing
• stream-proxy-request
• subscription-def
• subscription_thread_ng
• travis-errors
• travis-fixes
• travis-focal
• update-travis-matrix
• uwe-fixes
• v11
• v12
• v12.0
• various_improvements
• volumes

Move bad-files under each volume directory

The bad-files directory, currently sitting directly under the NGAS root
directory and not within any of the volume directories, contains files
that for one reason or another could not be archived automatically
(server shutdown, client disconnected, etc). When files are moved into
the bad-files directory, their original location is always the staging/
directory in the target volume where the final archiving takes place.
Because volumes are usually mapped to different filesystems, this often
results on slow, expensive data copies across filesystems instead of
fast, simpler renames within the single filesystem. Moreover, files
under bad-files are never automatically removed or picked out in any
way, and therefore they simply accumulate until they are dealt with
(most commonly removed).

This commit moves the bad-files directory to exist within each of the
volumes rather than directly under the NGAS root. This should speed up
greatly the movement of files into such directories, while keeping the
information of the disk they originated from.

This issue was reported in #51.

Signed-off-by: Rodrigo Tobar <rtobar@icrar.org>