GavinHuttley / ensembl_tui / 12921907602

89%

push

github

MAINT: refactor homology installation

[CHANGED] split the homology installation into loading, aggregating, and
    writing. This does increase memory footprint, but it greatly improves
    performance.

[CHANGED] HomologyAggregator.add_record() directly inserts into the
    homology_groups_attr table. We now assume the input records are in
    their final form. This means we can delete nearly all attributes
    and methods. It also means we no longer need the
    HomologyAggregator.commit() method.

[CHANGED] _ingest_homology.group_related() takes raw data from sql query,
    which significantly improves performance

[NEW] _ingest_homology.merge_grouped() allows us to merge the homolog_group
    data in python, rather than looking things up in duckdb -- significant
    performance improvement

[CHANGED] don't compress result of load_homologies based on whether running
    in multiprocess mode. Has little benefit in terms of memory footprint.
    Revisit this issue later.

48 of 49 new or added lines in 3 files covered. (97.96%)

2632 of 3028 relevant lines covered (86.92%)

7.61 hits per line