openbraininstitute / neurodamus / 38563660628

91%

push

github

Load .h5 morphologies without leaking memory for every 3D point (#592)

Loading an `.h5` morphology (`morphio_read`, which the emodel templates
and `Cell.hoc` call) runs one hoc command per 3D point, from a hoc loop
over the Python list: `execute(cmds._[i], cell)`. Every command hoc
reads out of the list stays allocated, about 80 bytes each, and
`clear_model` doesn't get it back. For L1 cells of a rat somatosensory
circuit (about 15,000 points each) that is 1.2 MB per cell, about half
of what a built cell costs. The same commands run from a Python loop
leak nothing.

`MorphIOWrapper.instantiate(cell)` now builds the morphology. The few
`create` and subset commands still go through hoc. The sections are
connected, and their points added, through NEURON's Python API
(`pt3dadd` with vectors), rounded to the 8 significant digits the
commands carry. `morphio_read` uses it. The two other hoc loops over
Python lists, `execute_commands` in `Cell.hoc` (used by `AddHocMorph`)
and `execute_commands_from_pylist` in `MorphIO.hoc`, now loop in Python.

Measured in the OBI neurodamus image, building the 291 cells of an L1
circuit on one rank (BlueRecording `write_weights`):

| | main | This PR |
|---|---|---|
| Memory for the cells | 446 MB | 216 MB |
| Run time | 37 s | 22 s |

- The geometry is bit-identical: every section, point, diameter and
parent of 32 morphologies (448,100 points), and BlueRecording's weights
files.
- Building and clearing the same 30 cells again: memory grew 37 MB per
round on main and stays flat with this PR.
- `instantiate` loads those 32 morphologies in 1.0 s; the hoc loop takes
2.5 s.
- In that image, `tests/unit` gives the same results as main, plus a new
test that `instantiate` matches the hoc commands.
`tests/integration-e2e` and `tests/scientific` pass.

26 of 26 new or added lines in 1 file covered. (100.0%)

8030 of 8811 relevant lines covered (91.14%)

9.12 hits per line