SwissDataScienceCenter / renku-data-services / 16621300066

87%

main: 86%

Build Type

push

github

Committed by web-flow

Commit Message

fix: refactor get_all_project_copies for performance reasons (#940)

The implementation had some issues that make it slow:

- It doesn't implement any paging, so we are potentially transporting
  a lot of data. This is not fixed in this PR.

- At very first, it retrieves *all* copied projects for a given
  template_id and then filters it in memory to retain only the
  accessible ones of the current user.

  This is done by next retrieving all project ids the user has access
  to, including all public ones. This is potentially returning a lot
  of data.

  The filtering then continues to loop through the database objects
  and checks each id for membership in the `project_ids` list. A
  python list has O(n) time for membership tests, so depending on the
  number of results this gets slow quite quickly.

Measuring this call shows that by far the longest time is spent
filtering:
```python
[p for p in project_orms if p.id in project_ids]
```
It can quickly get to multi-seconds response time, if the data is a
bit larger.

The improvements involve the following:

- Exchange order of retrieval: first get all accessible project ids,
  except those for public projects. This can reduce the resulting set of
  ids to check against as it doesn't contain public entities.

- With these ids, the SQL query is constructed to only return projects
  whose ids are in this set. If the scope is read, then the condition
  includes all public ones and the ones the user has more specific
  access to.

In my measurements on my machine with a generated data set of 2000
project copies, these changes resulted in 10x faster response times.

Run Details

9 of 9 new or added lines in 1 file covered. (100.0%)

2 existing lines in 1 file now uncovered.

21644 of 24817 relevant lines covered (87.21%)

1.53 hits per line