3017863014

Build Type

github

Committed by GitHub

Commit Message

<a href="https://github.com/kobotoolbox/kpi/commit/<a class=hub.com/kobotoolbox/kpi/commit/<a class="double-link" href="https://git"><a class=hub.com/kobotoolbox/kpi/commit/<a class="double-link" href="https://git"><a class=hub.com/kobotoolbox/kpi/commit/<a class="double-link" href="https://git"><a class=hub.com/kobotoolbox/kpi/commit/cf9f73427049be3ccfda7d9ad686deabc26b5677">cf9f73427">&lt;a href=&quot;https://github.com/kobotoolbox/kpi/commit/</a><a class="double-link" href="https://github.com/kobotoolbox/kpi/commit/&lt;a class=&quot;double-link&quot; href=&quot;https://git">&lt;a class=</a>hub.com/kobotoolbox/kpi/commit/&lt;a class=&quot;double-link&quot; href=&quot;https://git">&lt;a class=</a>hub.com/kobotoolbox/kpi/commit/&lt;a class=&quot;double-link&quot; href=&quot;https://git">&lt;a class=</a>hub.com/kobotoolbox/kpi/commit/&lt;a class=&quot;double-link&quot; href=&quot;https://git">&lt;a class=</a>hub.com/kobotoolbox/kpi/commit/cf9f73427049be3ccfda7d9ad686deabc26b5677">cf9f73427</a><a href="https://github.com/kobotoolbox/kpi/commit/cf9f73427049be3ccfda7d9ad686deabc26b5677">&quot;&gt;&amp;lt;a href=&amp;quot;https://github.com/kobotoolbox/kpi/commit/&lt;/a&gt;&lt;a class=&quot;double-link&quot; href=&quot;https://github.com/kobotoolbox/kpi/commit/&amp;lt;a class=&amp;quot;double-link&amp;quot; href=&amp;quot;https://git&quot;&gt;&amp;lt;a class=&lt;/a&gt;hub.com/kobotoolbox/kpi/commit/&amp;lt;a class=&amp;quot;double-link&amp;quot; href=&amp;quot;https://git&quot;&gt;&amp;lt;a class=&lt;/a&gt;hub.com/kobotoolbox/kpi/commit/&amp;lt;a class=&amp;quot;double-link&amp;quot; href=&amp;quot;https://git&quot;&gt;&amp;lt;a class=&lt;/a&gt;hub.com/kobotoolbox/kpi/commit/&amp;lt;a class=&amp;quot;double-link&amp;quot; href=&amp;quot;https://git&quot;&gt;&amp;lt;a class=&lt;/a&gt;hub.com/kobotoolbox/kpi/commit/cf9f73427049be3ccfda7d9ad686deabc26b5677&quot;&gt;cf9f73427&lt;/a&gt;&lt;a href=&quot;https://github.com/kobotoolbox/kpi/commit/cf9f73427049be3ccfda7d9ad686deabc26b5677&quot;&gt;&amp;quot;&amp;gt;&amp;amp;lt;a href=&amp;amp;quot;https://github.com/kobotoolbox/kpi/commit/&amp;lt;/a&amp;gt;&amp;lt;a class=&amp;quot;double-link&amp;quot; href=&amp;quot;https://github.com/kobotoolbox/kpi/commit/&amp;amp;lt;a class=&amp;amp;quot;double-link&am... (continued)

Pull Request Pull Request #3989: Move background tasks for REST Services to their own queue

Run Details

2155 of 3562 branches covered (60.5%)

2 of 2 new or added lines in 2 files covered. (100.0%)

9270 of 12185 relevant lines covered (76.08%)

1.52 hits per line

Source File
Press 'n' to go to next uncovered line, 'b' for previous

50.91

/kobo/apps/hook/views/v2/hook.py

# coding: utf-8
from datetime import timedelta

import constance
from django.db.models import Q
from django.utils import timezone
from django.utils.translation import gettext as t
from rest_framework import viewsets, status
from rest_framework.decorators import action
from rest_framework.response import Response
from rest_framework_extensions.mixins import NestedViewSetMixin

from kobo.apps.hook.constants import HOOK_LOG_FAILED, HOOK_LOG_PENDING
from kobo.apps.hook.models import Hook, HookLog
from kobo.apps.hook.serializers.v2.hook import HookSerializer
from kobo.apps.hook.tasks import retry_all_task
from kpi.permissions import AssetEditorSubmissionViewerPermission
from kpi.utils.viewset_mixins import AssetNestedObjectViewsetMixin


class HookViewSet(AssetNestedObjectViewsetMixin, NestedViewSetMixin,
                  viewsets.ModelViewSet):
    """

    ## External services

    Lists the external services endpoints accessible to requesting user

    <pre class="prettyprint">
    <b>GET</b> /api/v2/assets/{asset_uid}/hooks/
    </pre>

    > Example
    >
    >       curl -X GET https://[kpi-url]/api/v2/assets/a9PkXcgVgaDXuwayVeAuY5/hooks/

    ## CRUD

    * `asset_uid` - is the unique identifier of a specific asset
    * `uid` - is the unique identifier of a specific external service

    #### Retrieves an external service
    <pre class="prettyprint">
    <b>GET</b> /api/v2/assets/<code>{asset_uid}</code>/hooks/<code>{uid}</code>
    </pre>


    > Example
    >
    >       curl -X GET https://[kpi-url]/api/v2/assets/a9PkXcgVgaDXuwayVeAuY5/hooks/hfgha2nxBdoTVcwohdYNzb

    #### Add an external service to asset.
    <pre class="prettyprint">
    <b>POST</b> /api/v2/assets/<code>{asset_uid}</code>/hooks/
    </pre>


    > Example
    >
    >       curl -X POST https://[kpi-url]/api/v2/assets/a9PkXcgVgaDXuwayVeAuY5/hooks/


    > **Payload to create a new external service**
    >
    >        {
    >           "name": {string},
    >           "endpoint": {string},
    >           "active": {boolean},
    >           "email_notification": {boolean},
    >           "export_type": {string},
    >           "subset_fields": [{string}],
    >           "auth_level": {string},
    >           "settings": {
    >               "username": {string},
    >               "password": {string},
    >               "custom_headers": {
    >                   {string}: {string}
    >                   ...
    >                   {string}: {string}
    >               }
    >           },
    >           "payload_template": {string}
    >        }

    where

    * `name` and `endpoint` are required
    * `active` is True by default
    * `export_type` must be one these values:

        1. `json` (_default_)
        2. `xml`

    * `email_notification` is a boolean. If true, User will be notified when request to remote server has failed.
    * `auth_level` must be one these values:

        1. `no_auth` (_default_)
        2. `basic_auth`

    * `subset_fields` is the list of fields of the form definition. Only these fields should be present in data sent to remote server
    * `settings`.`custom_headers` is dictionary of `custom header`: `value`

    For example:
    >           "settings": {
    >               "customer_headers": {
    >                   "Authorization" : "Token 1af538baa9045a84c0e889f672baf83ff24"
    >               }

    * `payload_template` is a custom wrapper around `%SUBMISSION%` when sending data to remote server.
       It can be used only with JSON submission format.

    For example:
    >           "payload_template": '{"fields": %SUBMISSION%}'

    #### Update an external service.
    <pre class="prettyprint">
    <b>PATCH</b> /api/v2/assets/<code>{asset_uid}</code>/hooks/{uid}
    </pre>


    > Example
    >
    >       curl -X PATCH https://[kpi-url]/api/v2/assets/a9PkXcgVgaDXuwayVeAuY5/hooks/hfgha2nxBdoTVcwohdYNzb


    Only specify properties to update in the payload. See above for payload structure

    #### Delete an external service.
    <pre class="prettyprint">
    <b>DELETE</b> /api/v2/assets/<code>{asset_uid}</code>/hooks/{uid}
    </pre>


    > Example
    >
    >       curl -X DELETE https://[kpi-url]/api/v2/assets/a9PkXcgVgaDXuwayVeAuY5/hooks/hfgha2nxBdoTVcwohdYNzb

    #### Retries all failed attempts
    <pre class="prettyprint">
    <b>PATCH</b> /api/v2/assets/<code>{asset_uid}</code>/hooks/<code>{hook_uid}</code>/retry/
    </pre>

    **This call is asynchronous. Job is sent to Celery to be run in background**

    > Example
    >
    >       curl -X PATCH https://[kpi-url]/api/v2/assets/a9PkXcgVgaDXuwayVeAuY5/hooks/hfgha2nxBdoTVcwohdYNzb/retry/

    It returns all logs `uid`s that are being retried.

    ### CURRENT ENDPOINT
    """

    model = Hook
    lookup_field = "uid"
    serializer_class = HookSerializer
    permission_classes = (AssetEditorSubmissionViewerPermission,)

    def get_queryset(self):
        queryset = self.model.objects.filter(asset__uid=self.asset.uid)
        # Even though we only need 'uid', `select_related('asset__uid')`
        # actually pulled in the entire `kpi_asset` table under Django 1.8. In
        # Django 1.9, "select_related() prohibits non-relational fields for
        # nested relations."
        queryset = queryset.select_related('asset')
        return queryset

    def perform_create(self, serializer):
        serializer.save(asset=self.asset)

    @action(detail=True, methods=["PATCH"])
    def retry(self, request, uid=None, *args, **kwargs):
        hook = self.get_object()
        response = {"detail": t("Task successfully scheduled")}
        status_code = status.HTTP_200_OK
        if hook.active:
            seconds = HookLog.get_elapsed_seconds(constance.config.HOOK_MAX_RETRIES)
            threshold = timezone.now() - timedelta(seconds=seconds)

            records = hook.logs.filter(Q(date_modified__lte=threshold,
                                         status=HOOK_LOG_PENDING) |
                                       Q(status=HOOK_LOG_FAILED)). \
                values_list("id", "uid").distinct()
            # Prepare lists of ids
            hooklogs_ids = []
            hooklogs_uids = []
            for record in records:
                hooklogs_ids.append(record[0])
                hooklogs_uids.append(record[1])

            if len(records) > 0:
                # Mark all logs as PENDING
                HookLog.objects.filter(id__in=hooklogs_ids).update(status=HOOK_LOG_PENDING)
                # Delegate to Celery
                retry_all_task.apply_async(
                    queue='kpi_low_priority_queue', args=(hooklogs_ids,)
                )
                response.update({
                    "pending_uids": hooklogs_uids
                })

            else:
                response["detail"] = t("No data to retry")
                status_code = status.HTTP_304_NOT_MODIFIED
        else:
            response["detail"] = t("Can not retry on disabled hooks")
            status_code = status.HTTP_400_BAD_REQUEST

        return Response(response, status=status_code)

1	# coding: utf-8
2	from datetime import timedelta	2✔
3
4	import constance	2✔
5	from django.db.models import Q	2✔
6	from django.utils import timezone	2✔
7	from django.utils.translation import gettext as t	2✔
8	from rest_framework import viewsets, status	2✔
9	from rest_framework.decorators import action	2✔
10	from rest_framework.response import Response	2✔
11	from rest_framework_extensions.mixins import NestedViewSetMixin	2✔
12
13	from kobo.apps.hook.constants import HOOK_LOG_FAILED, HOOK_LOG_PENDING	2✔
14	from kobo.apps.hook.models import Hook, HookLog	2✔
15	from kobo.apps.hook.serializers.v2.hook import HookSerializer	2✔
16	from kobo.apps.hook.tasks import retry_all_task	2✔
17	from kpi.permissions import AssetEditorSubmissionViewerPermission	2✔
18	from kpi.utils.viewset_mixins import AssetNestedObjectViewsetMixin	2✔
19
20
21	class HookViewSet(AssetNestedObjectViewsetMixin, NestedViewSetMixin,	2✔
22	viewsets.ModelViewSet):
23	"""
24
25	## External services
26
27	Lists the external services endpoints accessible to requesting user
28
29	<pre class="prettyprint">
30	<b>GET</b> /api/v2/assets/{asset_uid}/hooks/
31	</pre>
32
33	> Example
34	>
35	> curl -X GET https://[kpi-url]/api/v2/assets/a9PkXcgVgaDXuwayVeAuY5/hooks/
36
37	## CRUD
38
39	* `asset_uid` - is the unique identifier of a specific asset
40	* `uid` - is the unique identifier of a specific external service
41
42	#### Retrieves an external service
43	<pre class="prettyprint">
44	<b>GET</b> /api/v2/assets/<code>{asset_uid}</code>/hooks/<code>{uid}</code>
45	</pre>
46
47
48	> Example
49	>
50	> curl -X GET https://[kpi-url]/api/v2/assets/a9PkXcgVgaDXuwayVeAuY5/hooks/hfgha2nxBdoTVcwohdYNzb
51
52	#### Add an external service to asset.
53	<pre class="prettyprint">
54	<b>POST</b> /api/v2/assets/<code>{asset_uid}</code>/hooks/
55	</pre>
56
57
58	> Example
59	>
60	> curl -X POST https://[kpi-url]/api/v2/assets/a9PkXcgVgaDXuwayVeAuY5/hooks/
61
62
63	> Payload to create a new external service
64	>
65	> {
66	> "name": {string},
67	> "endpoint": {string},
68	> "active": {boolean},
69	> "email_notification": {boolean},
70	> "export_type": {string},
71	> "subset_fields": [{string}],
72	> "auth_level": {string},
73	> "settings": {
74	> "username": {string},
75	> "password": {string},
76	> "custom_headers": {
77	> {string}: {string}
78	> ...
79	> {string}: {string}
80	> }
81	> },
82	> "payload_template": {string}
83	> }
84
85	where
86
87	* `name` and `endpoint` are required
88	* `active` is True by default
89	* `export_type` must be one these values:
90
91	1. `json` (_default_)
92	2. `xml`
93
94	* `email_notification` is a boolean. If true, User will be notified when request to remote server has failed.
95	* `auth_level` must be one these values:
96
97	1. `no_auth` (_default_)
98	2. `basic_auth`
99
100	* `subset_fields` is the list of fields of the form definition. Only these fields should be present in data sent to remote server
101	* `settings`.`custom_headers` is dictionary of `custom header`: `value`
102
103	For example:
104	> "settings": {
105	> "customer_headers": {
106	> "Authorization" : "Token 1af538baa9045a84c0e889f672baf83ff24"
107	> }
108
109	* `payload_template` is a custom wrapper around `%SUBMISSION%` when sending data to remote server.
110	It can be used only with JSON submission format.
111
112	For example:
113	> "payload_template": '{"fields": %SUBMISSION%}'
114
115	#### Update an external service.
116	<pre class="prettyprint">
117	<b>PATCH</b> /api/v2/assets/<code>{asset_uid}</code>/hooks/{uid}
118	</pre>
119
120
121	> Example
122	>
123	> curl -X PATCH https://[kpi-url]/api/v2/assets/a9PkXcgVgaDXuwayVeAuY5/hooks/hfgha2nxBdoTVcwohdYNzb
124
125
126	Only specify properties to update in the payload. See above for payload structure
127
128	#### Delete an external service.
129	<pre class="prettyprint">
130	<b>DELETE</b> /api/v2/assets/<code>{asset_uid}</code>/hooks/{uid}
131	</pre>
132
133
134	> Example
135	>
136	> curl -X DELETE https://[kpi-url]/api/v2/assets/a9PkXcgVgaDXuwayVeAuY5/hooks/hfgha2nxBdoTVcwohdYNzb
137
138	#### Retries all failed attempts
139	<pre class="prettyprint">
140	<b>PATCH</b> /api/v2/assets/<code>{asset_uid}</code>/hooks/<code>{hook_uid}</code>/retry/
141	</pre>
142
143	This call is asynchronous. Job is sent to Celery to be run in background
144
145	> Example
146	>
147	> curl -X PATCH https://[kpi-url]/api/v2/assets/a9PkXcgVgaDXuwayVeAuY5/hooks/hfgha2nxBdoTVcwohdYNzb/retry/
148
149	It returns all logs `uid`s that are being retried.
150
151	### CURRENT ENDPOINT
152	"""
153
154	model = Hook	2✔
155	lookup_field = "uid"	2✔
156	serializer_class = HookSerializer	2✔
157	permission_classes = (AssetEditorSubmissionViewerPermission,)	2✔
158
159	def get_queryset(self):	2✔
160	queryset = self.model.objects.filter(asset__uid=self.asset.uid)	2✔
161	# Even though we only need 'uid', `select_related('asset__uid')`
162	# actually pulled in the entire `kpi_asset` table under Django 1.8. In
163	# Django 1.9, "select_related() prohibits non-relational fields for
164	# nested relations."
165	queryset = queryset.select_related('asset')	2✔
166	return queryset	2✔
167
168	def perform_create(self, serializer):	2✔
169	serializer.save(asset=self.asset)	2✔
170
171	@action(detail=True, methods=["PATCH"])	2✔
172	def retry(self, request, uid=None, args, *kwargs):	2✔
173	hook = self.get_object()	×
174	response = {"detail": t("Task successfully scheduled")}	×
175	status_code = status.HTTP_200_OK	×
176	if hook.active:	×
177	seconds = HookLog.get_elapsed_seconds(constance.config.HOOK_MAX_RETRIES)	×
178	threshold = timezone.now() - timedelta(seconds=seconds)	×
179
180	records = hook.logs.filter(Q(date_modified__lte=threshold,	×
181	status=HOOK_LOG_PENDING) \|
182	Q(status=HOOK_LOG_FAILED)). \
183	values_list("id", "uid").distinct()
184	# Prepare lists of ids
185	hooklogs_ids = []	×
186	hooklogs_uids = []	×
187	for record in records:	×
188	hooklogs_ids.append(record[0])	×
189	hooklogs_uids.append(record[1])	×
190
191	if len(records) > 0:	×
192	# Mark all logs as PENDING
193	HookLog.objects.filter(id__in=hooklogs_ids).update(status=HOOK_LOG_PENDING)	×
194	# Delegate to Celery
195	retry_all_task.apply_async(	×
196	queue='kpi_low_priority_queue', args=(hooklogs_ids,)
197	)
198	response.update({	×
199	"pending_uids": hooklogs_uids
200	})
201
202	else:
203	response["detail"] = t("No data to retry")	×
204	status_code = status.HTTP_304_NOT_MODIFIED	×
205	else:
206	response["detail"] = t("Can not retry on disabled hooks")	×
207	status_code = status.HTTP_400_BAD_REQUEST	×
208
209	return Response(response, status=status_code)	×

kobotoolbox / kpi / 3017863014

Source File Press 'n' to go to next uncovered line, 'b' for previous

Source File
Press 'n' to go to next uncovered line, 'b' for previous