• Home
  • Features
  • Pricing
  • Docs
  • Announcements
  • Sign In

MITLibraries / geo-harvester
100%
main: 100%

Build:
Build:
LAST BUILD BRANCH: IN-1246-pip-audit
DEFAULT BRANCH: main
Repo Added 21 Dec 2023 09:08PM UTC
Files 26
Badge
Embed ▾
README BADGES
x

If you need to use a raster PNG badge, change the '.svg' to '.png' in the link

Markdown

Textile

RDoc

HTML

Rst

LAST BUILD ON BRANCH GDT-230-add-alma-marc-cli-cmd
branch: GDT-230-add-alma-marc-cli-cmd
CHANGE BRANCH
x
Reset
  • GDT-230-add-alma-marc-cli-cmd
  • GDT-109-no-cdn-write-ogm
  • GDT-109-normalize-ogm-records
  • GDT-116-fix-wkt-coords
  • GDT-149-updates-from-full-mit-harvest
  • GDT-156-memory-performance
  • GDT-159-make-bbox-optional-and-add-quality-validation
  • GDT-159-remove-bbox-requirement-from-mitaardvark-and-add-wkt-validator
  • GDT-170-remove-sqs-check
  • GDT-195-normalize-format-resource-type
  • GDT-226-alma-marc-fetch-and-filter
  • GDT-227-refactor-source-format-classes
  • GDT-227-transform-alma-marc-optional-fields
  • GDT-227-transform-alma-marc-required-fields
  • GDT-241-post-normalize-data-cleanup
  • GDT-267-no-records-skip-output-file
  • GDT-269-ogm-incremental-use-rss-vs-api
  • GDT-275-sqs-missing-file
  • GDT-290-pool-and-dedupe-eb-events
  • GDT-323-resolve-container-oom-error-for-cdn-writes
  • GDT-329-skip-ogm-supressed
  • GDT-68-update-external-url
  • GDT-85-ogm-fetch-records
  • GDT-85-ogm-github-api
  • GDT-85-ogm-incremental-harvest-improvements
  • GDT-87-send-eb-events
  • IN-1088-maintenance-2024-11
  • IN-1246-pip-audit
  • ISS-59-fgdc-mapping-bug
  • bug-modify-failed-record-logging
  • cdn-url-fix
  • dependabot/pip/aiohttp-3.9.2
  • dependabot/pip/attrs-23.2.0
  • dependabot/pip/black-24.2.0
  • dependabot/pip/black-24.3.0
  • dependabot/pip/black-24.4.1
  • dependabot/pip/black-24.4.2
  • dependabot/pip/boto3-1.34.10
  • dependabot/pip/boto3-1.34.11
  • dependabot/pip/boto3-1.34.12
  • dependabot/pip/boto3-1.34.13
  • dependabot/pip/boto3-1.34.14
  • dependabot/pip/boto3-1.34.15
  • dependabot/pip/boto3-1.34.16
  • dependabot/pip/boto3-1.34.17
  • dependabot/pip/boto3-1.34.19
  • dependabot/pip/boto3-1.34.20
  • dependabot/pip/boto3-1.34.21
  • dependabot/pip/boto3-1.34.22
  • dependabot/pip/boto3-1.34.23
  • dependabot/pip/boto3-1.34.25
  • dependabot/pip/boto3-1.34.26
  • dependabot/pip/boto3-1.34.27
  • dependabot/pip/boto3-1.34.28
  • dependabot/pip/boto3-1.34.29
  • dependabot/pip/boto3-1.34.30
  • dependabot/pip/boto3-1.34.31
  • dependabot/pip/boto3-1.34.32
  • dependabot/pip/boto3-1.34.33
  • dependabot/pip/boto3-1.34.34
  • dependabot/pip/boto3-1.34.35
  • dependabot/pip/boto3-1.34.36
  • dependabot/pip/boto3-1.34.37
  • dependabot/pip/boto3-1.34.38
  • dependabot/pip/boto3-1.34.39
  • dependabot/pip/boto3-1.34.40
  • dependabot/pip/boto3-1.34.41
  • dependabot/pip/boto3-1.34.42
  • dependabot/pip/boto3-1.34.43
  • dependabot/pip/boto3-1.34.44
  • dependabot/pip/boto3-1.34.45
  • dependabot/pip/boto3-1.34.46
  • dependabot/pip/boto3-1.34.47
  • dependabot/pip/boto3-1.34.48
  • dependabot/pip/boto3-1.34.49
  • dependabot/pip/boto3-1.34.50
  • dependabot/pip/boto3-1.34.51
  • dependabot/pip/boto3-1.34.52
  • dependabot/pip/boto3-1.34.53
  • dependabot/pip/boto3-1.34.54
  • dependabot/pip/boto3-1.34.55
  • dependabot/pip/boto3-1.34.56
  • dependabot/pip/boto3-1.34.57
  • dependabot/pip/boto3-1.34.58
  • dependabot/pip/boto3-1.34.59
  • dependabot/pip/boto3-1.34.6
  • dependabot/pip/boto3-1.34.60
  • dependabot/pip/boto3-1.34.61
  • dependabot/pip/boto3-1.34.62
  • dependabot/pip/boto3-1.34.63
  • dependabot/pip/boto3-1.34.64
  • dependabot/pip/boto3-1.34.65
  • dependabot/pip/boto3-1.34.66
  • dependabot/pip/boto3-1.34.67
  • dependabot/pip/boto3-1.34.68
  • dependabot/pip/boto3-1.34.69
  • dependabot/pip/boto3-1.34.7
  • dependabot/pip/boto3-1.34.70
  • dependabot/pip/boto3-1.34.71
  • dependabot/pip/boto3-1.34.72
  • dependabot/pip/boto3-1.34.73
  • dependabot/pip/boto3-1.34.74
  • dependabot/pip/boto3-1.34.75
  • dependabot/pip/boto3-1.34.76
  • dependabot/pip/boto3-1.34.77
  • dependabot/pip/boto3-1.34.78
  • dependabot/pip/boto3-1.34.79
  • dependabot/pip/boto3-1.34.8
  • dependabot/pip/boto3-1.34.80
  • dependabot/pip/boto3-1.34.81
  • dependabot/pip/boto3-1.34.82
  • dependabot/pip/boto3-1.34.83
  • dependabot/pip/boto3-1.34.84
  • dependabot/pip/boto3-1.34.85
  • dependabot/pip/boto3-1.34.86
  • dependabot/pip/boto3-1.34.87
  • dependabot/pip/boto3-1.34.88
  • dependabot/pip/boto3-1.34.89
  • dependabot/pip/boto3-1.34.9
  • dependabot/pip/boto3-1.34.90
  • dependabot/pip/boto3-1.34.91
  • dependabot/pip/boto3-1.34.92
  • dependabot/pip/boto3-1.34.93
  • dependabot/pip/boto3-1.34.94
  • dependabot/pip/boto3-1.34.95
  • dependabot/pip/boto3-1.34.96
  • dependabot/pip/boto3-1.34.97
  • dependabot/pip/boto3-1.34.98
  • dependabot/pip/boto3-stubs-1.34.10
  • dependabot/pip/boto3-stubs-1.34.11
  • dependabot/pip/boto3-stubs-1.34.12
  • dependabot/pip/boto3-stubs-1.34.13
  • dependabot/pip/boto3-stubs-1.34.14
  • dependabot/pip/boto3-stubs-1.34.15
  • dependabot/pip/boto3-stubs-1.34.16
  • dependabot/pip/boto3-stubs-1.34.17
  • dependabot/pip/boto3-stubs-1.34.19
  • dependabot/pip/boto3-stubs-1.34.20
  • dependabot/pip/boto3-stubs-1.34.21
  • dependabot/pip/boto3-stubs-1.34.22
  • dependabot/pip/boto3-stubs-1.34.23
  • dependabot/pip/boto3-stubs-1.34.25
  • dependabot/pip/boto3-stubs-1.34.26
  • dependabot/pip/boto3-stubs-1.34.27
  • dependabot/pip/boto3-stubs-1.34.28
  • dependabot/pip/boto3-stubs-1.34.29
  • dependabot/pip/boto3-stubs-1.34.30
  • dependabot/pip/boto3-stubs-1.34.31
  • dependabot/pip/boto3-stubs-1.34.32
  • dependabot/pip/boto3-stubs-1.34.33
  • dependabot/pip/boto3-stubs-1.34.34
  • dependabot/pip/boto3-stubs-1.34.36
  • dependabot/pip/boto3-stubs-1.34.37
  • dependabot/pip/boto3-stubs-1.34.38
  • dependabot/pip/boto3-stubs-1.34.39
  • dependabot/pip/boto3-stubs-1.34.40
  • dependabot/pip/boto3-stubs-1.34.41
  • dependabot/pip/boto3-stubs-1.34.42
  • dependabot/pip/boto3-stubs-1.34.43
  • dependabot/pip/boto3-stubs-1.34.44
  • dependabot/pip/boto3-stubs-1.34.45
  • dependabot/pip/boto3-stubs-1.34.46
  • dependabot/pip/boto3-stubs-1.34.47
  • dependabot/pip/boto3-stubs-1.34.48
  • dependabot/pip/boto3-stubs-1.34.49
  • dependabot/pip/boto3-stubs-1.34.50
  • dependabot/pip/boto3-stubs-1.34.51
  • dependabot/pip/boto3-stubs-1.34.52
  • dependabot/pip/boto3-stubs-1.34.53
  • dependabot/pip/boto3-stubs-1.34.54
  • dependabot/pip/boto3-stubs-1.34.55
  • dependabot/pip/boto3-stubs-1.34.56
  • dependabot/pip/boto3-stubs-1.34.57
  • dependabot/pip/boto3-stubs-1.34.58
  • dependabot/pip/boto3-stubs-1.34.59
  • dependabot/pip/boto3-stubs-1.34.6
  • dependabot/pip/boto3-stubs-1.34.60
  • dependabot/pip/boto3-stubs-1.34.61
  • dependabot/pip/boto3-stubs-1.34.62
  • dependabot/pip/boto3-stubs-1.34.63
  • dependabot/pip/boto3-stubs-1.34.64
  • dependabot/pip/boto3-stubs-1.34.65
  • dependabot/pip/boto3-stubs-1.34.66
  • dependabot/pip/boto3-stubs-1.34.67
  • dependabot/pip/boto3-stubs-1.34.68
  • dependabot/pip/boto3-stubs-1.34.69
  • dependabot/pip/boto3-stubs-1.34.7
  • dependabot/pip/boto3-stubs-1.34.70
  • dependabot/pip/boto3-stubs-1.34.71
  • dependabot/pip/boto3-stubs-1.34.72
  • dependabot/pip/boto3-stubs-1.34.73
  • dependabot/pip/boto3-stubs-1.34.74
  • dependabot/pip/boto3-stubs-1.34.75
  • dependabot/pip/boto3-stubs-1.34.76
  • dependabot/pip/boto3-stubs-1.34.77
  • dependabot/pip/boto3-stubs-1.34.78
  • dependabot/pip/boto3-stubs-1.34.79
  • dependabot/pip/boto3-stubs-1.34.8
  • dependabot/pip/boto3-stubs-1.34.80
  • dependabot/pip/boto3-stubs-1.34.81
  • dependabot/pip/boto3-stubs-1.34.82
  • dependabot/pip/boto3-stubs-1.34.83
  • dependabot/pip/boto3-stubs-1.34.84
  • dependabot/pip/boto3-stubs-1.34.85
  • dependabot/pip/boto3-stubs-1.34.86
  • dependabot/pip/boto3-stubs-1.34.87
  • dependabot/pip/boto3-stubs-1.34.88
  • dependabot/pip/boto3-stubs-1.34.89
  • dependabot/pip/boto3-stubs-1.34.9
  • dependabot/pip/boto3-stubs-1.34.90
  • dependabot/pip/boto3-stubs-1.34.91
  • dependabot/pip/boto3-stubs-1.34.92
  • dependabot/pip/boto3-stubs-1.34.93
  • dependabot/pip/boto3-stubs-1.34.94
  • dependabot/pip/boto3-stubs-1.34.95
  • dependabot/pip/boto3-stubs-1.34.96
  • dependabot/pip/boto3-stubs-1.34.97
  • dependabot/pip/boto3-stubs-1.34.98
  • dependabot/pip/cryptography-42.0.4
  • dependabot/pip/freezegun-1.5.0
  • dependabot/pip/idna-3.7
  • dependabot/pip/ipython-8.19.0
  • dependabot/pip/ipython-8.20.0
  • dependabot/pip/ipython-8.21.0
  • dependabot/pip/jinja2-3.1.3
  • dependabot/pip/jinja2-3.1.4
  • dependabot/pip/moto-5.0.1
  • dependabot/pip/moto-5.0.3
  • dependabot/pip/moto-5.0.4
  • dependabot/pip/moto-5.0.5
  • dependabot/pip/mypy-1.10.0
  • dependabot/pip/mypy-1.8.0
  • dependabot/pip/pre-commit-3.6.1
  • dependabot/pip/pygit2-1.14.1
  • dependabot/pip/ruff-0.1.11
  • dependabot/pip/ruff-0.1.13
  • dependabot/pip/ruff-0.1.14
  • dependabot/pip/ruff-0.1.15
  • dependabot/pip/ruff-0.1.9
  • dependabot/pip/ruff-0.2.0
  • dependabot/pip/ruff-0.2.1
  • dependabot/pip/sentry-sdk-1.40.1
  • dependabot/pip/sentry-sdk-1.40.2
  • dependabot/pip/sentry-sdk-1.40.3
  • dependabot/pip/sentry-sdk-1.40.4
  • dependabot/pip/sentry-sdk-1.40.5
  • dependabot/pip/sentry-sdk-1.40.6
  • dependabot/pip/sentry-sdk-1.41.0
  • dependabot/pip/sentry-sdk-1.42.0
  • dependabot/pip/sentry-sdk-1.43.0
  • dependabot/pip/sentry-sdk-1.44.0
  • dependabot/pip/sentry-sdk-1.44.1
  • dependabot/pip/sentry-sdk-1.45.0
  • dependabot/pip/shapely-2.0.3
  • dependabot/pip/types-python-dateutil-2.8.19.20240311
  • dependabot/pip/types-python-dateutil-2.9.0.20240315
  • dependabot/pip/types-python-dateutil-2.9.0.20240316
  • dependabot/pip/types-requests-2.31.0.20240406
  • dependabot/pip/werkzeug-3.0.3
  • fix-tests-from-merges
  • handle-ogm-missing-data
  • main
  • refs/tags/v1.0.0
  • refs/tags/v1.1.0
  • refs/tags/v1.2.0
  • refs/tags/v1.2.1
  • refs/tags/v1.3.0
  • v1.3.1

16 Apr 2024 03:19PM UTC coverage: 99.664% (+0.2%) from 99.466%
8708241698

Pull #271

github

jonavellecuerdo
Add CLI command and workflow for Alma harvests

Why these changes are being introduced:
* Geo-Harvester will be used to retrieve geospatial metadata records from
MIT Alma exports.

How this addresses that need:
* Add 'alma' harvest command to cli.py
* Add test invoking 'alma' harvest command in test_cli.py
* Add test to verify error is raised when 'latest full run date' cannot be identified
* Refactor MITAlmaHarvester methods to support 'full' and 'incremental'
harvests

Side effects of this change:
* None

Relevant ticket(s):
* https://mitlibraries.atlassian.net/browse/GDT-230
Pull Request #271: Add CLI command and workflow for Alma harvests

51 of 51 new or added lines in 3 files covered. (100.0%)

2078 of 2085 relevant lines covered (99.66%)

1.0 hits per line

Relevant lines Covered
Build:
Build:
2085 RELEVANT LINES 2078 COVERED LINES
1.0 HITS PER LINE
Source Files on GDT-230-add-alma-marc-cli-cmd
  • Tree
  • List 29
  • Changed 7
  • Source Changed 0
  • Coverage Changed 7
Coverage ∆ File Lines Relevant Covered Missed Hits/Line

Recent builds

Builds Branch Commit Type Ran Committer Via Coverage
8708241698 GDT-230-add-alma-marc-cli-cmd Add CLI command and workflow for Alma harvests Why these changes are being introduced: * Geo-Harvester will be used to retrieve geospatial metadata records from MIT Alma exports. How this addresses that need: * Add 'alma' harvest command to cli.... Pull #271 16 Apr 2024 03:21PM UTC jonavellecuerdo github
99.66
8706645670 GDT-230-add-alma-marc-cli-cmd Address comments * Move raising of error to MITAlmaHarvester.full_harvest_get_source_records * Clean up MITAlmaHarvester.get_latest_full_run_date Pull #271 16 Apr 2024 01:37PM UTC jonavellecuerdo github
99.66
8706631850 GDT-230-add-alma-marc-cli-cmd Consolidate methods for filtering filepaths by dates push 16 Apr 2024 01:36PM UTC jonavellecuerdo github
99.66
8667439778 GDT-230-add-alma-marc-cli-cmd Address comments * Move raising of error to MITAlmaHarvester.full_harvest_get_source_records * Clean up MITAlmaHarvester.get_latest_full_run_date Pull #271 12 Apr 2024 08:12PM UTC jonavellecuerdo github
99.47
8665961115 GDT-230-add-alma-marc-cli-cmd Address comments * Update docstring for MITAlmaHarvester.full_harvest_get_source_records * Address mypy errors to enable return of NoneType by MITAlmaHarvester._get_date_from_filepath * Add test to verify error is raised when 'latest full run dat... Pull #271 12 Apr 2024 05:47PM UTC jonavellecuerdo github
99.47
8665907637 GDT-230-add-alma-marc-cli-cmd Address comments * Update docstring for MITAlmaHarvester.full_harvest_get_source_records * Address mypy errors to enable return of NoneType by MITAlmaHarvester._get_date_from_filepath * Add test to verify error is raised when 'latest full run dat... Pull #271 12 Apr 2024 05:41PM UTC jonavellecuerdo github
99.47
8653104039 GDT-230-add-alma-marc-cli-cmd Add CLI command and workflow for Alma harvests Why these changes are being introduced: * Geo-Harvester will be used to retrieve geospatial metadata records from MIT Alma exports. How this addresses that need: * Add 'alma' harvest command to cli.... Pull #271 11 Apr 2024 08:53PM UTC jonavellecuerdo github
99.43
8651439429 GDT-230-add-alma-marc-cli-cmd Add CLI command and workflow for Alma harvests Why these changes are being introduced: * Geo-Harvester will be used to retrieve geospatial metadata records from MIT Alma exports. How this addresses that need: * Add 'alma' harvest command to cli.... push 11 Apr 2024 06:22PM UTC jonavellecuerdo github
99.43
See All Builds (389)
  • Repo on GitHub
STATUS · Troubleshooting · Open an Issue · Sales · Support · CAREERS · ENTERPRISE · START FREE · SCHEDULE DEMO
ANNOUNCEMENTS · TWITTER · TOS & SLA · Supported CI Services · What's a CI service? · Automated Testing

© 2025 Coveralls, Inc