• Home
  • Features
  • Pricing
  • Docs
  • Announcements
  • Sign In

tomasnorre / crawler / 43449756443

13 Nov 2026 03:09PM UTC coverage: 67.299% (-4.0%) from 71.322%
43449756443

push

github

web-flow
!!! [FEATURE] Process the crawler queue with the TYPO3 message bus (#1322)

The crawler queue is no longer processed by the extension's own process
management. Each queue entry is dispatched as a CrawlUrlMessage to the
TYPO3 message bus (symfony/messenger) and crawled by CrawlUrlHandler in
a worker started with "typo3 messenger:consume".

Queue processing:
- Queue entries are still stored in tx_crawler_queue. The message only
  carries the queue id and is dispatched when the entry is added from the
  backend module, the crawler:buildQueue command or on page save.
- Scheduled entries are delayed with a DelayStamp.
- The Before/AfterQueueItemAddedEvent are dispatched by the handler.
- Re-crawling from the log module dispatches a forced message.

Priority crawling:
- Three Doctrine transports crawler_high, crawler and crawler_low.
- The transport is chosen by the sitemap_priority of the page (EXT:seo),
  re-crawls from the log module use the high priority transport.
- Run the worker with "messenger:consume crawler_high crawler crawler_low"
  to crawl high priority pages first.
- If the routing of CrawlUrlMessage is changed, e.g. to the synchronous
  "default" transport, the routing is respected.

Breaking changes:
- The crawler:processQueue command, the process backend module, the
  process status AJAX route and the table tx_crawler_process are removed,
  together with the process related extension settings.
- Removed classes: Crawler, ProcessService, ProcessRepository, the
  Process models, cleaners and manager, the Sleeper helpers, HookUtility,
  BackendModuleLinkService, AfterQueueItemAddedEventListener,
  Domain\Model\Configuration, Domain\Model\Queue, Value\CrawlAction,
  CrawlerObjectException and TimeStampException.
- Removed methods: CrawlerController::readUrl(), readUrlFromArray() and
  the deprecated drawURLs_PIfilter(), as well as the process related
  methods of QueueRepository.
- The extension requires symfony/messenger and psr/event-dispatc... (continued)

74 of 91 new or added lines in 10 files covered. (81.32%)

6 existing lines in 1 file now uncovered.

1278 of 1899 relevant lines covered (67.3%)

4.99 hits per line

Source File
Press 'n' to go to next uncovered line, 'b' for previous

0.0
/Classes/Controller/Backend/BackendModuleStartCrawlingController.php
1
<?php
2

3
declare(strict_types=1);
4

5
namespace AOE\Crawler\Controller\Backend;
6

7
/*
8
 * (c) 2022-     Tomas Norre Mikkelsen <tomasnorre@gmail.com>
9
 *
10
 * This file is part of the TYPO3 Crawler Extension.
11
 *
12
 * It is free software; you can redistribute it and/or modify it under
13
 * the terms of the GNU General Public License, either version 2
14
 * of the License, or any later version.
15
 *
16
 * For the full copyright and license information, please read the
17
 * LICENSE.txt file that was distributed with this source code.
18
 *
19
 * The TYPO3 project - inspiring people to share!
20
 */
21

22
use AOE\Crawler\Controller\Backend\Helper\RequestHelper;
23
use AOE\Crawler\Controller\Backend\Helper\UrlBuilder;
24
use AOE\Crawler\Controller\CrawlerController;
25
use AOE\Crawler\Domain\Model\Reason;
26
use AOE\Crawler\Event\InvokeQueueChangeEvent;
27
use AOE\Crawler\Utility\MessageUtility;
28
use Psr\Http\Message\ResponseInterface;
29
use Psr\Http\Message\ServerRequestInterface;
30
use Psr\Http\Message\UriInterface;
31
use Symfony\Contracts\Service\Attribute\Required;
32
use TYPO3\CMS\Backend\Routing\Exception\RouteNotFoundException;
33
use TYPO3\CMS\Backend\Routing\UriBuilder;
34
use TYPO3\CMS\Backend\Template\ModuleTemplate;
35
use TYPO3\CMS\Core\EventDispatcher\EventDispatcher;
36
use TYPO3\CMS\Core\Utility\GeneralUtility;
37

38
/**
39
 * @internal since v12.0.0
40
 */
41
class BackendModuleStartCrawlingController extends AbstractBackendModuleController implements BackendModuleControllerInterface
42
{
43
    private const BACKEND_MODULE = 'web_site_crawler_start';
44
    private const LINE_FEED = 10;
45
    private const CARRIAGE_RETURN = 13;
46
    private int $reqMinute = 1000;
47
    private EventDispatcher $eventDispatcher;
48

49
    /**
50
     * @var array holds the selection of configuration from the configuration selector box
51
     */
52
    private $incomingConfigurationSelection = [];
53

54
    public function __construct(
55
        private readonly CrawlerController $crawlerController,
56
        private readonly UriBuilder $backendUriBuilder,
57
    ) {
58
    }
×
59

60
    #[Required]
61
    public function setEventDispatcher(): void
62
    {
63
        $this->eventDispatcher = GeneralUtility::makeInstance(EventDispatcher::class);
×
64
    }
65

66
    #[\Override]
67
    public function handleRequest(ServerRequestInterface $request): ResponseInterface
68
    {
NEW
69
        $this->checkPhpBinaryForDirectRequests($this->extensionSettings);
×
70
        $this->pageUid = (int) ($request->getQueryParams()['id'] ?? -1);
×
71
        $this->moduleTemplate = $this->setupView($request, $this->pageUid);
×
72
        $this->moduleTemplate->makeDocHeaderModuleMenu([
×
73
            'id' => $request->getQueryParams()['id'] ?? -1,
×
74
        ]);
×
75

76
        $this->assignValues($request);
×
77
        return $this->moduleTemplate->renderResponse('Backend/ShowCrawlerInformation');
×
78
    }
79

80
    private function assignValues(ServerRequestInterface $request): ModuleTemplate
81
    {
82
        $logUrl = $this->backendUriBuilder->buildUriFromRoute('web_site_crawler_log', [
×
83
            'id' => $this->pageUid,
×
84
        ]);
×
85

86
        $crawlingDepth = RequestHelper::getIntFromRequest($request, 'crawlingDepth');
×
87
        $submitCrawlUrls = RequestHelper::getBoolFromRequest($request, '_crawl');
×
88
        $downloadCrawlUrls = RequestHelper::getBoolFromRequest($request, '_download');
×
89
        $scheduledTime = $this->getScheduledTime(RequestHelper::getStringFromRequest($request, 'tstamp', 'now'));
×
90

91
        $this->incomingConfigurationSelection = RequestHelper::getArrayFromRequest($request, 'configurationSelection');
×
92

93
        //$this->crawlerController = $this->getCrawlerController();
94
        $this->crawlerController->setID = GeneralUtility::md5int(microtime());
×
95

96
        $queueRows = '';
×
97
        $noConfigurationSelected = $this->isNoConfigurationSelected();
×
98
        if ($noConfigurationSelected) {
×
99
            MessageUtility::addWarningMessage(
×
100
                $this->getLanguageService()->sL('crawler.messages:labels.noConfigSelected')
×
101
            );
×
102
        } else {
103
            if ($submitCrawlUrls) {
×
104
                $reason = new Reason();
×
105
                $reason->setReason(Reason::REASON_GUI_SUBMIT);
×
106
                $reason->setDetailText(
×
107
                    'The user ' . $GLOBALS['BE_USER']->user['username'] . ' added pages to the crawler queue manually'
×
108
                );
×
109
                $this->eventDispatcher->dispatch(new InvokeQueueChangeEvent($reason));
×
110
            }
111

112
            $queueRows = $this->crawlerController->getPageTreeAndUrls(
×
113
                $this->pageUid,
×
114
                $crawlingDepth,
×
115
                $scheduledTime,
×
116
                $this->reqMinute,
×
117
                $submitCrawlUrls,
×
118
                $downloadCrawlUrls,
×
119
                // Do not filter any processing instructions
120
                [],
×
121
                $this->incomingConfigurationSelection
×
122
            );
×
123
        }
124

125
        // Download Urls to crawl:
126
        $downloadUrls = $this->crawlerController->downloadUrls;
×
127
        if ($downloadCrawlUrls) {
×
128
            // Creating output header:
129
            header('Content-Type: application/octet-stream');
×
130
            header('Content-Disposition: attachment; filename=CrawlerUrls.txt');
×
131

132
            // Printing the content of the CSV lines:
133
            echo implode(chr(self::CARRIAGE_RETURN) . chr(self::LINE_FEED), $downloadUrls);
×
134
            exit;
×
135
        }
136

137
        return $this->moduleTemplate->assignMultiple([
×
138
            'currentPageId' => $this->pageUid,
×
139
            'noConfigurationSelected' => $noConfigurationSelected,
×
140
            'submitCrawlUrls' => $submitCrawlUrls,
×
141
            'amountOfUrls' => count($this->crawlerController->duplicateTrack),
×
142
            'selectors' => $this->generateConfigurationSelectors($this->pageUid, (string) $crawlingDepth, $request),
×
143
            'queueRows' => $queueRows,
×
144
            'displayActions' => 0,
×
145
            'actionUrl' => $this->getActionUrl(),
×
146
            'logUrl' => $logUrl,
×
147
        ]);
×
148
    }
149

150
    /**
151
     * Generates the configuration selectors for compiling URLs:
152
     */
153
    private function generateConfigurationSelectors(
154
        int $pageId,
155
        string $crawlingDepth,
156
        ServerRequestInterface $request
157
    ): array {
158
        $selectors = [];
×
159
        $selectors['depth'] = $this->selectorBox(
×
160
            [
×
161
                0 => $this->getLanguageService()->sL('core.core:labels.depth_0'),
×
162
                1 => $this->getLanguageService()->sL('core.core:labels.depth_1'),
×
163
                2 => $this->getLanguageService()->sL('core.core:labels.depth_2'),
×
164
                3 => $this->getLanguageService()->sL('core.core:labels.depth_3'),
×
165
                4 => $this->getLanguageService()->sL('core.core:labels.depth_4'),
×
166
                99 => $this->getLanguageService()->sL('core.core:labels.depth_infi'),
×
167
            ],
×
168
            'crawlingDepth',
×
169
            $crawlingDepth,
×
170
            false
×
171
        );
×
172

173
        // Configurations
174
        $availableConfigurations = $this->crawlerController->getConfigurationsForBranch(
×
175
            $pageId,
×
176
            (int) $crawlingDepth,
×
177
        );
×
178
        $selectors['configurations'] = $this->selectorBox(
×
179
            empty($availableConfigurations) ? [] : array_combine($availableConfigurations, $availableConfigurations),
×
180
            'configurationSelection',
×
181
            $this->incomingConfigurationSelection,
×
182
            true
×
183
        );
×
184

185
        // Scheduled time:
186
        $selectors['scheduled'] = $this->selectorBox(
×
187
            [
×
188
                'now' => $this->getLanguageService()->sL('crawler.messages:labels.time.now'),
×
189
                'midnight' => $this->getLanguageService()->sL('crawler.messages:labels.time.midnight'),
×
190
                '04:00' => $this->getLanguageService()->sL('crawler.messages:labels.time.4am'),
×
191
            ],
×
192
            'tstamp',
×
193
            RequestHelper::getStringFromRequest($request, 'tstamp', 'now'),
×
194
            false
×
195
        );
×
196

197
        return $selectors;
×
198
    }
199

200
    /**
201
     * Create selector box
202
     *
203
     * @param array $optArray Options key(value) => label pairs
204
     * @param string $name Selector box name
205
     * @param string|array|null $value Selector box value (array for multiple...)
206
     * @param boolean $multiple If set, will draw multiple box.
207
     *
208
     * @return string HTML select element
209
     */
210
    private function selectorBox($optArray, $name, string|array|null $value, bool $multiple): string
211
    {
212
        if (!is_string($value) && !is_array($value)) {
×
213
            $value = '';
×
214
        }
215

216
        $options = [];
×
217
        foreach ($optArray as $key => $val) {
×
218
            $selected = (
×
219
                !$multiple && is_string($value) && !strcmp($value, (string) $key)
×
220
            ) || (
×
221
                $multiple && in_array($key, (array) $value, true)
×
222
            );
×
223
            $options[] = '
×
224
                <option value="' . $key . '" ' . ($selected ? ' selected="selected"' : '') . '>' . htmlspecialchars(
×
225
                (string) $val,
×
226
                ENT_QUOTES | ENT_HTML5
×
227
            ) . '</option>';
×
228
        }
229

230
        return '<select class="form-select" name="' . htmlspecialchars(
×
231
            $name . ($multiple ? '[]' : ''),
×
232
            ENT_QUOTES | ENT_HTML5
×
233
        ) . '"' . ($multiple ? ' multiple' : '') . '>' . implode('', $options) . '</select>';
×
234
    }
235

236
    private function getScheduledTime(string $time): int
237
    {
238
        $scheduledTime = match ($time) {
×
239
            'midnight' => mktime(0, 0, 0),
×
240
            '04:00' => mktime(0, 0, 0) + 4 * 3600,
×
241
            default => time(),
×
242
        };
×
243

244
        if (!$scheduledTime) {
×
245
            return time();
×
246
        }
247

248
        return $scheduledTime;
×
249
    }
250

251
    /**
252
     * @throws RouteNotFoundException
253
     */
254
    private function getActionUrl(): UriInterface
255
    {
256
        return GeneralUtility::makeInstance(UrlBuilder::class)->getBackendModuleUrl([], self::BACKEND_MODULE);
×
257
    }
258

259
    private function isNoConfigurationSelected(): bool
260
    {
261
        return empty($this->incomingConfigurationSelection)
×
262
            || (count($this->incomingConfigurationSelection) === 1 && empty($this->incomingConfigurationSelection[0]));
×
263
    }
264
}
STATUS · Troubleshooting · Open an Issue · Sales · Support · CAREERS · ENTERPRISE · START FREE TRIAL · SCHEDULE DEMO
ANNOUNCEMENTS · TWITTER · TOS & SLA · Supported CI Services · What's a CI service? · Automated Testing

© 2026 Coveralls, Inc