16021803183

Committed 02 Jul 2025 09:46AM UTC coverage: 38.627% (-0.03%) from 38.661%

Build # 16021803183

Build Type

Pull #176

github

Committed by

web-flow

Commit Message

Merge 562ab0019 into ac34a7dec

Pull Request Pull Request #176: Hotfix ckan helper to handle expected keys missing

Run Details

0 of 2 new or added lines in 1 file covered. (0.0%)

878 of 2273 relevant lines covered (38.63%)

0.39 hits per line

Source File
Press 'n' to go to next uncovered line, 'b' for previous

0.0

/helpers/ckan.py

import json
import os
from collections import defaultdict

out = defaultdict(dict)
licenses = {}

# The ckan directory is the one produced by https://github.com/Bjwebb/IATI-Registry-Refresher/tree/save_ckan_json

for publisher in os.listdir('ckan'):
    with open(os.path.join('ckan', publisher)) as fp:
        try:
            for package in json.load(fp)['result']['results']:
                if package['resources']:
                    extras = dict((x['key'], x['value']) for x in package['extras'])
                    out[publisher][package['name']] = {
                        'title': package['title'],
                        'extras': extras,
                        'license_id': package['license_id'],
                        'resource': package['resources'][0],
                    }
                    if package['license_id']:
                        licenses[package['license_id']] = {
                            'name': package['license_title'],
                            'url': package.get('license_url'),
                        }
        except ValueError:
            print('{0} is not valid JSON'.format(publisher))
        except KeyError:
            print('{0} does not have the expected structure'.format(publisher))

with open('ckan.json', 'w') as fp:
    json.dump(out, fp, indent=2, sort_keys=True)
with open('licenses.json', 'w') as fp:
    json.dump(licenses, fp, indent=2, sort_keys=True)

1	import json	×
2	import os	×
3	from collections import defaultdict	×
4
5	out = defaultdict(dict)	×
6	licenses = {}	×
7
8	# The ckan directory is the one produced by https://github.com/Bjwebb/IATI-Registry-Refresher/tree/save_ckan_json
9
10	for publisher in os.listdir('ckan'):	×
11	with open(os.path.join('ckan', publisher)) as fp:	×
12	try:	×
13	for package in json.load(fp)['result']['results']:	×
14	if package['resources']:	×
15	extras = dict((x['key'], x['value']) for x in package['extras'])	×
16	out[publisher][package['name']] = {	×
17	'title': package['title'],
18	'extras': extras,
19	'license_id': package['license_id'],
20	'resource': package['resources'][0],
21	}
22	if package['license_id']:	×
23	licenses[package['license_id']] = {	×
24	'name': package['license_title'],
25	'url': package.get('license_url'),
26	}
27	except ValueError:	×
28	print('{0} is not valid JSON'.format(publisher))	×
NEW 29	except KeyError:	×
NEW 30	print('{0} does not have the expected structure'.format(publisher))	×
31
32	with open('ckan.json', 'w') as fp:	×
33	json.dump(out, fp, indent=2, sort_keys=True)	×
34	with open('licenses.json', 'w') as fp:	×
35	json.dump(licenses, fp, indent=2, sort_keys=True)	×

IATI / IATI-Stats / 16021803183

Source File Press 'n' to go to next uncovered line, 'b' for previous

Source File
Press 'n' to go to next uncovered line, 'b' for previous