• Home
  • Features
  • Pricing
  • Docs
  • Announcements
  • Sign In

tomasnorre / crawler / 43449756443

13 Nov 2026 03:09PM UTC coverage: 67.299% (-4.0%) from 71.322%
43449756443

push

github

web-flow
!!! [FEATURE] Process the crawler queue with the TYPO3 message bus (#1322)

The crawler queue is no longer processed by the extension's own process
management. Each queue entry is dispatched as a CrawlUrlMessage to the
TYPO3 message bus (symfony/messenger) and crawled by CrawlUrlHandler in
a worker started with "typo3 messenger:consume".

Queue processing:
- Queue entries are still stored in tx_crawler_queue. The message only
  carries the queue id and is dispatched when the entry is added from the
  backend module, the crawler:buildQueue command or on page save.
- Scheduled entries are delayed with a DelayStamp.
- The Before/AfterQueueItemAddedEvent are dispatched by the handler.
- Re-crawling from the log module dispatches a forced message.

Priority crawling:
- Three Doctrine transports crawler_high, crawler and crawler_low.
- The transport is chosen by the sitemap_priority of the page (EXT:seo),
  re-crawls from the log module use the high priority transport.
- Run the worker with "messenger:consume crawler_high crawler crawler_low"
  to crawl high priority pages first.
- If the routing of CrawlUrlMessage is changed, e.g. to the synchronous
  "default" transport, the routing is respected.

Breaking changes:
- The crawler:processQueue command, the process backend module, the
  process status AJAX route and the table tx_crawler_process are removed,
  together with the process related extension settings.
- Removed classes: Crawler, ProcessService, ProcessRepository, the
  Process models, cleaners and manager, the Sleeper helpers, HookUtility,
  BackendModuleLinkService, AfterQueueItemAddedEventListener,
  Domain\Model\Configuration, Domain\Model\Queue, Value\CrawlAction,
  CrawlerObjectException and TimeStampException.
- Removed methods: CrawlerController::readUrl(), readUrlFromArray() and
  the deprecated drawURLs_PIfilter(), as well as the process related
  methods of QueueRepository.
- The extension requires symfony/messenger and psr/event-dispatc... (continued)

74 of 91 new or added lines in 10 files covered. (81.32%)

6 existing lines in 1 file now uncovered.

1278 of 1899 relevant lines covered (67.3%)

4.99 hits per line

Source File
Press 'n' to go to next uncovered line, 'b' for previous

0.0
/Classes/Controller/Backend/BackendModuleCrawlerLogController.php
1
<?php
2

3
declare(strict_types=1);
4

5
namespace AOE\Crawler\Controller\Backend;
6

7
/*
8
 * (c) 2022-     Tomas Norre Mikkelsen <tomasnorre@gmail.com>
9
 *
10
 * This file is part of the TYPO3 Crawler Extension.
11
 *
12
 * It is free software; you can redistribute it and/or modify it under
13
 * the terms of the GNU General Public License, either version 2
14
 * of the License, or any later version.
15
 *
16
 * For the full copyright and license information, please read the
17
 * LICENSE.txt file that was distributed with this source code.
18
 *
19
 * The TYPO3 project - inspiring people to share!
20
 */
21

22
use AOE\Crawler\Controller\Backend\Helper\RequestHelper;
23
use AOE\Crawler\Controller\Backend\Helper\ResultHandler;
24
use AOE\Crawler\Controller\Backend\Helper\UrlBuilder;
25
use AOE\Crawler\Domain\Repository\QueueRepository;
26
use AOE\Crawler\Queue\CrawlPriority;
27
use AOE\Crawler\Queue\CrawlUrlMessageDispatcher;
28
use AOE\Crawler\Queue\Message\CrawlUrlMessage;
29
use AOE\Crawler\Service\BackendModuleLogService;
30
use AOE\Crawler\Service\BackendModuleScriptUrlService;
31
use AOE\Crawler\Utility\MessageUtility;
32
use AOE\Crawler\Value\QueueFilter;
33
use AOE\Crawler\Writer\FileWriter\CsvWriter\CsvWriterInterface;
34
use Psr\Http\Message\ResponseInterface;
35
use Psr\Http\Message\ServerRequestInterface;
36
use Psr\Http\Message\UriInterface;
37
use Symfony\Contracts\Service\Attribute\Required;
38
use TYPO3\CMS\Backend\Routing\Exception\RouteNotFoundException;
39
use TYPO3\CMS\Backend\Template\ModuleTemplate;
40
use TYPO3\CMS\Backend\Tree\View\PageTreeView;
41
use TYPO3\CMS\Backend\Utility\BackendUtility;
42
use TYPO3\CMS\Core\Database\ConnectionPool;
43
use TYPO3\CMS\Core\Database\Query\QueryBuilder;
44
use TYPO3\CMS\Core\Imaging\IconFactory;
45
use TYPO3\CMS\Core\Imaging\IconSize;
46
use TYPO3\CMS\Core\Utility\DebugUtility;
47
use TYPO3\CMS\Core\Utility\GeneralUtility;
48

49
/**
50
 * @internal since v12.0.0
51
 */
52
final class BackendModuleCrawlerLogController extends AbstractBackendModuleController implements BackendModuleControllerInterface
53
{
54
    public const BACKEND_MODULE = 'web_site_crawler_log';
55

56
    private QueryBuilder $queryBuilder;
57
    private bool $CSVExport = false;
58
    private readonly array $backendModuleMenu;
59
    private int $setId;
60
    private string $quiPath;
61
    private string $logDisplay;
62
    private int $itemsPerPage;
63
    private string $showResultLog;
64
    private string $showFeVars;
65
    private int $showSetId;
66
    private string $logDepth;
67
    /**
68
     * @var mixed|string|null
69
     */
70
    private mixed $queueId;
71

72
    public function __construct(
73
        private readonly QueueRepository $queueRepository,
74
        private readonly CsvWriterInterface $csvWriter,
75
        private readonly IconFactory $iconFactory,
76
        private readonly CrawlUrlMessageDispatcher $crawlUrlMessageDispatcher,
77
        private readonly BackendModuleLogService $backendModuleLogService,
78
        private readonly BackendModuleScriptUrlService $backendModuleScriptUrlService,
79
    ) {
80
        $this->backendModuleMenu = $this->getModuleMenu();
×
81
    }
82

83
    #[Required]
84
    public function setQueryBuilder(): void
85
    {
86
        $this->queryBuilder = GeneralUtility::makeInstance(ConnectionPool::class)->getQueryBuilderForTable(
×
87
            QueueRepository::TABLE_NAME
×
88
        );
×
89
    }
90

91
    #[\Override]
92
    public function handleRequest(ServerRequestInterface $request): ResponseInterface
93
    {
94
        $this->setPropertiesBasedOnPostVars($request);
×
95
        $this->moduleTemplate = $this->setupView($request, $this->pageUid);
×
96
        $this->moduleTemplate = $this->moduleTemplate->makeDocHeaderModuleMenu([
×
97
            'id' => $this->pageUid,
×
98
        ]);
×
99

100
        if (!$this->pageUid) {
×
101
            $this->moduleTemplate->assign('noPageSelected', true);
×
102
            return $this->moduleTemplate->renderResponse('Backend/ShowLog');
×
103
        }
104
        $this->moduleTemplate = $this->assignValues($request);
×
105
        return $this->moduleTemplate->renderResponse('Backend/ShowLog');
×
106
    }
107

108
    private function getQueueEntry(mixed $queueId): array
109
    {
110
        $q_entry = $this->queryBuilder
×
111
            ->from(QueueRepository::TABLE_NAME)
×
112
            ->select('*')->where(
×
113
                $this->queryBuilder->expr()->eq('qid', $this->queryBuilder->createNamedParameter($queueId))
×
114
            )->executeQuery()
×
115
            ->fetchAssociative();
×
116

117
        // Explode values
118
        $q_entry['parameters'] = json_decode((string) $q_entry['parameters'], true);
×
119
        $q_entry['result_data'] = json_decode((string) $q_entry['result_data'], true);
×
120
        $resStatus = ResultHandler::getResStatus($q_entry['result_data']);
×
121
        if (is_array($q_entry['result_data'])) {
×
122
            $q_entry['result_data']['content'] = json_decode((string) $q_entry['result_data']['content'], true);
×
123
            if (!$this->showResultLog) {
×
124
                if (is_array($q_entry['result_data']['content'])) {
×
125
                    unset($q_entry['result_data']['content']['log']);
×
126
                }
127
            }
128
        }
129
        return [$q_entry, $resStatus];
×
130
    }
131

132
    /**
133
     * @throws RouteNotFoundException
134
     */
135
    private function assignValues(ServerRequestInterface $request): ModuleTemplate
136
    {
137
        // Look for set ID sent - if it is, we will display contents of that set:
138
        $this->showSetId = RequestHelper::getIntFromRequest($request, 'setID');
×
139
        $this->CSVExport = RequestHelper::getBoolFromRequest($request, '_csv');
×
140
        $logEntriesPerPage = [];
×
141
        $csvData = [];
×
142

143
        $quidRead = RequestHelper::getIntFromRequest($request, 'qid_read');
×
144
        if ($quidRead) {
×
145
            // Re-crawl an already crawled entry, requested by an editor and therefore with high priority
NEW
146
            $this->crawlUrlMessageDispatcher->dispatch(
×
NEW
147
                new CrawlUrlMessage($quidRead, true),
×
NEW
148
                priority: CrawlPriority::High
×
NEW
149
            );
×
150
        }
151

152
        if ($this->queueId) {
×
153
            // Get entry record:
154
            [$q_entry, $resStatus] = $this->getQueueEntry($this->queueId);
×
155
            $this->moduleTemplate->assignMultiple([
×
156
                'queueStatus' => $resStatus,
×
157
                'queueDetails' => DebugUtility::viewArray($q_entry),
×
158
            ]);
×
159
        } else {
160
            // Show list
161
            // Drawing tree:
162
            $tree = GeneralUtility::makeInstance(PageTreeView::class);
×
163
            $perms_clause = $GLOBALS['BE_USER']->getPagePermsClause(1);
×
164
            $tree->init('AND ' . $perms_clause);
×
165

166
            // Set root row:
167
            $pageinfo = BackendUtility::readPageAccess($this->pageUid, $perms_clause);
×
168

169
            if (is_array($pageinfo)) {
×
170
                $HTML = $this->iconFactory->getIconForRecord('pages', $pageinfo, IconSize::SMALL)->render();
×
171
                $tree->tree[] = [
×
172
                    'row' => $pageinfo,
×
173
                    'HTML' => $HTML,
×
174
                ];
×
175
            }
176

177
            // Get branch beneath:
178
            if ($this->logDepth) {
×
179
                $tree->getTree($this->pageUid, (int) $this->logDepth);
×
180
            }
181

182
            // If Flush button is pressed, flush tables instead of selecting entries:
183
            if ($request->getParsedBody()['_flush'] ?? false) {
×
184
                $doFlush = true;
×
185
            } elseif ($request->getParsedBody()['_flush_all'] ?? false) {
×
186
                $doFlush = true;
×
187
                $this->logDisplay = 'all';
×
188
            } else {
189
                $doFlush = false;
×
190
            }
191

192
            $queueFilter = new QueueFilter($this->logDisplay);
×
193

194
            if ($doFlush) {
×
195
                $this->queueRepository->flushQueue($queueFilter);
×
196
            }
197

198
            // Traverse page tree:
199
            $count = 0;
×
200

201
            foreach ($tree->tree as $data) {
×
202
                $logEntriesOfPage = $this->queueRepository->getQueueEntriesForPageId(
×
203
                    (int) $data['row']['uid'],
×
204
                    $this->itemsPerPage,
×
205
                    $queueFilter
×
206
                );
×
207

208
                [$logEntriesPerPage[], $row] = $this->backendModuleLogService->addRows(
×
209
                    $logEntriesOfPage,
×
210
                    RequestHelper::getIntFromRequest($request, 'setID'),
×
211
                    $data['HTML'] . BackendUtility::getRecordTitle('pages', $data['row'], true),
×
212
                    $this->showResultLog,
×
213
                    $this->showFeVars,
×
214
                    $this->CSVExport
×
215
                );
×
216
                $csvData[] = $row;
×
217

218
                if (++$count === 1000) {
×
219
                    break;
×
220
                }
221
            }
222

223
            $this->moduleTemplate->assign('logEntriesPerPage', $logEntriesPerPage);
×
224
        }
225

226
        if ($this->CSVExport) {
×
227
            $this->outputCsvFile($csvData);
×
228
        }
229

230
        $queryParams = [
×
231
            'setID' => $this->setId,
×
232
            'displayLog' => $this->logDisplay,
×
233
            'itemsPerPage' => $this->itemsPerPage,
×
234
            'ShowFeVars' => $this->showFeVars,
×
235
            'ShowResultLog' => $this->showResultLog,
×
236
            'logDepth' => $this->logDepth,
×
237
        ];
×
238

239
        return $this->moduleTemplate->assignMultiple([
×
240
            'backendModuleUrl' => $this->getBackendModuleUrl(),
×
241
            'actionUrl' => '',
×
242
            'queueId' => $this->queueId,
×
243
            'setId' => $this->showSetId,
×
244
            'noPageSelected' => false,
×
245
            'logEntriesPerPage' => $logEntriesPerPage,
×
246
            'showResultLog' => $this->showResultLog,
×
247
            'showFeVars' => $this->showFeVars,
×
248
            'displayActions' => 1,
×
249
            'displayLogFilterConfig' => [
×
250
                'name' => 'displayLog',
×
251
                'currentValue' => $this->logDisplay,
×
252
                'menuItems' => $this->backendModuleMenu['displayLog'],
×
253
                'scriptUrl' => $this->backendModuleScriptUrlService->buildScriptUrl(
×
254
                    $request,
×
255
                    'displayLog',
×
256
                    $this->pageUid,
×
257
                    $queryParams
×
258
                ),
×
259
            ],
×
260
            'itemPerPageConfig' => [
×
261
                'name' => 'itemsPerPage',
×
262
                'currentValue' => $this->itemsPerPage,
×
263
                'menuItems' => $this->backendModuleMenu['itemsPerPage'],
×
264
                'scriptUrl' => $this->backendModuleScriptUrlService->buildScriptUrl(
×
265
                    $request,
×
266
                    'itemsPerPage',
×
267
                    $this->pageUid,
×
268
                    $queryParams
×
269
                ),
×
270
            ],
×
271
            'showResultLogConfig' => [
×
272
                'name' => 'ShowResultLog',
×
273
                'currentValue' => $this->showResultLog,
×
274
                'scriptUrl' => $this->backendModuleScriptUrlService->buildScriptUrl(
×
275
                    $request,
×
276
                    'ShowResultLog',
×
277
                    $this->pageUid,
×
278
                    $queryParams,
×
279
                    $this->quiPath
×
280
                ),
×
281
            ],
×
282
            'showFeVarsConfig' => [
×
283
                'name' => 'ShowFeVars',
×
284
                'currentValue' => $this->showFeVars,
×
285
                'scriptUrl' => $this->backendModuleScriptUrlService->buildScriptUrl(
×
286
                    $request,
×
287
                    'ShowFeVars',
×
288
                    $this->pageUid,
×
289
                    $queryParams,
×
290
                    $this->quiPath
×
291
                ),
×
292
            ],
×
293
            'depthDropDownConfig' => [
×
294
                'name' => 'logDepth',
×
295
                'currentValue' => $this->logDepth,
×
296
                'menuItems' => $this->backendModuleMenu['logDepth'],
×
297
                'scriptUrl' => $this->backendModuleScriptUrlService->buildScriptUrl(
×
298
                    $request,
×
299
                    'logDepth',
×
300
                    $this->pageUid,
×
301
                    $queryParams
×
302
                ),
×
303
            ],
×
304
        ]);
×
305
    }
306

307
    private function setPropertiesBasedOnPostVars(ServerRequestInterface $request): void
308
    {
309
        $this->pageUid = (int) ($request->getQueryParams()['id'] ?? -1);
×
310
        $this->setId = RequestHelper::getIntFromRequest($request, 'setIDÍ„');
×
311
        $quidDetails = $request->getParsedBody()['qid_details'] ?? $request->getQueryParams()['qid_details'] ?? null;
×
312
        $this->quiPath = $quidDetails ? '&qid_details=' . (int) $quidDetails : '';
×
313
        $this->queueId = $quidDetails ?? null;
×
314
        $this->logDisplay = RequestHelper::getStringFromRequest($request, 'displayLog', 'all');
×
315
        $this->itemsPerPage = RequestHelper::getIntFromRequest($request, 'itemsPerPage', 10);
×
316
        $this->showResultLog = RequestHelper::getStringFromRequest($request, 'ShowResultLog', '0');
×
317
        $this->showFeVars = RequestHelper::getStringFromRequest($request, 'ShowFeVars', '0');
×
318
        $this->logDepth = RequestHelper::getStringFromRequest($request, 'logDepth', '0');
×
319
    }
320

321
    /**
322
     * Outputs the CSV file and sets the correct headers
323
     */
324
    private function outputCsvFile(array $csvData): void
325
    {
326
        if (!count($csvData)) {
×
327
            MessageUtility::addWarningMessage(
×
328
                $this->getLanguageService()->sL('crawler.messages:message.canNotExportEmptyQueueToCsvText')
×
329
            );
×
330
            return;
×
331
        }
332

333
        $csvString = $this->csvWriter->arrayToCsv($csvData);
×
334

335
        header('Content-Type: application/octet-stream');
×
336
        header('Content-Disposition: attachment; filename=CrawlerLog.csv');
×
337
        echo $csvString;
×
338

339
        exit;
×
340
    }
341

342
    private function getBackendModuleUrl(): UriInterface
343
    {
344
        return GeneralUtility::makeInstance(UrlBuilder::class)->getBackendModuleUrl([], self::BACKEND_MODULE);
×
345
    }
346
}
STATUS · Troubleshooting · Open an Issue · Sales · Support · CAREERS · ENTERPRISE · START FREE TRIAL · SCHEDULE DEMO
ANNOUNCEMENTS · TWITTER · TOS & SLA · Supported CI Services · What's a CI service? · Automated Testing

© 2026 Coveralls, Inc