SwissDataScienceCenter / renku-data-services
87%
main: 86%

LAST BUILD BRANCH: eikek/multiple-jobs-per-launcher
DEFAULT BRANCH: main
Repo Added
Files 375
Badge
Embed ▾
README BADGES
x

If you need to use a raster PNG badge, change the '.svg' to '.png' in the link

Markdown

Textile

RDoc

HTML

Rst

LAST BUILD ON BRANCH v0.52.0
branch: v0.52.0
CHANGE BRANCH
x

coverage: 87.214% (+0.001%) from 87.213%
16621300066

push

github

web-flow 
fix: refactor get_all_project_copies for performance reasons (#940)

The implementation had some issues that make it slow:

- It doesn't implement any paging, so we are potentially transporting
  a lot of data. This is not fixed in this PR.

- At very first, it retrieves *all* copied projects for a given
  template_id and then filters it in memory to retain only the
  accessible ones of the current user.

  This is done by next retrieving all project ids the user has access
  to, including all public ones. This is potentially returning a lot
  of data.

  The filtering then continues to loop through the database objects
  and checks each id for membership in the `project_ids` list. A
  python list has O(n) time for membership tests, so depending on the
  number of results this gets slow quite quickly.

Measuring this call shows that by far the longest time is spent
filtering:
```python
[p for p in project_orms if p.id in project_ids]
```
It can quickly get to multi-seconds response time, if the data is a
bit larger.

The improvements involve the following:

- Exchange order of retrieval: first get all accessible project ids,
  except those for public projects. This can reduce the resulting set of
  ids to check against as it doesn't contain public entities.

- With these ids, the SQL query is constructed to only return projects
  whose ids are in this set. If the scope is read, then the condition
  includes all public ones and the ones the user has more specific
  access to.

In my measurements on my machine with a generated data set of 2000
project copies, these changes resulted in 10x faster response times.

21644 of 24817 relevant lines covered (87.21%)

1.53 hits per line

Relevant lines Covered
24817 RELEVANT LINES 21644 COVERED LINES
1.53 HITS PER LINE
Source Files on v0.52.0

Recent builds

Builds Branch Commit Type Ran Committer Via Coverage
16621300066 v0.52.0 fix: refactor get_all_project_copies for performance reasons (#940) The implementation had some issues that make it slow: - It doesn't implement any paging, so we are potentially transporting a lot of data. This is not fixed in this PR. - At ... push web-flow github
87.21
See All Builds (3605)