• Home
  • Features
  • Pricing
  • Docs
  • Announcements
  • Sign In

tomasnorre / crawler / 43449756443

13 Nov 2026 03:09PM UTC coverage: 67.299% (-4.0%) from 71.322%
43449756443

push

github

web-flow
!!! [FEATURE] Process the crawler queue with the TYPO3 message bus (#1322)

The crawler queue is no longer processed by the extension's own process
management. Each queue entry is dispatched as a CrawlUrlMessage to the
TYPO3 message bus (symfony/messenger) and crawled by CrawlUrlHandler in
a worker started with "typo3 messenger:consume".

Queue processing:
- Queue entries are still stored in tx_crawler_queue. The message only
  carries the queue id and is dispatched when the entry is added from the
  backend module, the crawler:buildQueue command or on page save.
- Scheduled entries are delayed with a DelayStamp.
- The Before/AfterQueueItemAddedEvent are dispatched by the handler.
- Re-crawling from the log module dispatches a forced message.

Priority crawling:
- Three Doctrine transports crawler_high, crawler and crawler_low.
- The transport is chosen by the sitemap_priority of the page (EXT:seo),
  re-crawls from the log module use the high priority transport.
- Run the worker with "messenger:consume crawler_high crawler crawler_low"
  to crawl high priority pages first.
- If the routing of CrawlUrlMessage is changed, e.g. to the synchronous
  "default" transport, the routing is respected.

Breaking changes:
- The crawler:processQueue command, the process backend module, the
  process status AJAX route and the table tx_crawler_process are removed,
  together with the process related extension settings.
- Removed classes: Crawler, ProcessService, ProcessRepository, the
  Process models, cleaners and manager, the Sleeper helpers, HookUtility,
  BackendModuleLinkService, AfterQueueItemAddedEventListener,
  Domain\Model\Configuration, Domain\Model\Queue, Value\CrawlAction,
  CrawlerObjectException and TimeStampException.
- Removed methods: CrawlerController::readUrl(), readUrlFromArray() and
  the deprecated drawURLs_PIfilter(), as well as the process related
  methods of QueueRepository.
- The extension requires symfony/messenger and psr/event-dispatc... (continued)

74 of 91 new or added lines in 10 files covered. (81.32%)

6 existing lines in 1 file now uncovered.

1278 of 1899 relevant lines covered (67.3%)

4.99 hits per line

Source File
Press 'n' to go to next uncovered line, 'b' for previous

80.45
/Classes/Controller/CrawlerController.php
1
<?php
2

3
declare(strict_types=1);
4

5
namespace AOE\Crawler\Controller;
6

7
/*
8
 * (c) 2021 Tomas Norre Mikkelsen <tomasnorre@gmail.com>
9
 *
10
 * This file is part of the TYPO3 Crawler Extension.
11
 *
12
 * It is free software; you can redistribute it and/or modify it under
13
 * the terms of the GNU General Public License, either version 2
14
 * of the License, or any later version.
15
 *
16
 * For the full copyright and license information, please read the
17
 * LICENSE.txt file that was distributed with this source code.
18
 *
19
 * The TYPO3 project - inspiring people to share!
20
 */
21

22
use AOE\Crawler\Configuration\ExtensionConfigurationProvider;
23
use AOE\Crawler\Domain\Repository\ConfigurationRepository;
24
use AOE\Crawler\Domain\Repository\QueueRepository;
25
use AOE\Crawler\Event\AfterUrlAddedToQueueEvent;
26
use AOE\Crawler\Queue\CrawlUrlMessageDispatcher;
27
use AOE\Crawler\Queue\Message\CrawlUrlMessage;
28
use AOE\Crawler\Service\ConfigurationService;
29
use AOE\Crawler\Service\PageService;
30
use AOE\Crawler\Service\ProcessInstructionService;
31
use AOE\Crawler\Service\UrlService;
32
use AOE\Crawler\Value\QueueRow;
33
use Psr\Http\Message\UriInterface;
34
use Psr\Log\LoggerAwareInterface;
35
use Psr\Log\LoggerAwareTrait;
36
use TYPO3\CMS\Backend\Tree\View\PageTreeView;
37
use TYPO3\CMS\Backend\Utility\BackendUtility;
38
use TYPO3\CMS\Core\Authentication\BackendUserAuthentication;
39
use TYPO3\CMS\Core\Authentication\GroupResolver;
40
use TYPO3\CMS\Core\Core\Bootstrap;
41
use TYPO3\CMS\Core\Domain\Repository\PageRepository;
42
use TYPO3\CMS\Core\EventDispatcher\EventDispatcher;
43
use TYPO3\CMS\Core\Imaging\IconFactory;
44
use TYPO3\CMS\Core\Imaging\IconSize;
45
use TYPO3\CMS\Core\Type\Bitmask\Permission;
46
use TYPO3\CMS\Core\Utility\DebugUtility;
47
use TYPO3\CMS\Core\Utility\GeneralUtility;
48
use TYPO3\CMS\Core\Utility\MathUtility;
49

50
/**
51
 * @package AOE\Crawler\Controller
52
 * @internal since v12.0.0
53
 */
54
class CrawlerController implements LoggerAwareInterface
55
{
56
    use LoggerAwareTrait;
57

58
    public int $setID = 0;
59
    public array $duplicateTrack = [];
60
    public array $downloadUrls = [];
61
    public array $incomingProcInstructions = [];
62
    public array $incomingConfigurationSelection = [];
63
    public bool $registerQueueEntriesInternallyOnly = false;
64
    public array $queueEntries = [];
65
    public array $urlList = [];
66
    public array $extensionSettings = [];
67

68
    /**
69
     * Mount Point
70
     */
71
    public ?string $MP = null;
72
    protected QueueRepository $queueRepository;
73
    protected ConfigurationRepository $configurationRepository;
74
    protected int $maximumUrlsToCompile = 10000;
75
    protected IconFactory $iconFactory;
76

77
    /**
78
     * @var BackendUserAuthentication|null
79
     */
80
    private $backendUser;
81
    private int $scheduledTime = 0;
82
    private int $reqMinute = 0;
83
    private bool $submitCrawlUrls = false;
84
    private bool $downloadCrawlUrls = false;
85
    private PageRepository $pageRepository;
86
    private ConfigurationService $configurationService;
87
    private UrlService $urlService;
88
    private EventDispatcher $eventDispatcher;
89
    private CrawlUrlMessageDispatcher $crawlUrlMessageDispatcher;
90

91
    /************************************
92
     *
93
     * Getting URLs based on Page TSconfig
94
     *
95
     ************************************/
96

97
    public function __construct()
98
    {
99
        $this->queueRepository = GeneralUtility::makeInstance(QueueRepository::class);
17 ✔
100
        $this->configurationRepository = GeneralUtility::makeInstance(ConfigurationRepository::class);
17 ✔
101
        $this->pageRepository = GeneralUtility::makeInstance(PageRepository::class);
17 ✔
102
        $this->eventDispatcher = GeneralUtility::makeInstance(EventDispatcher::class);
17 ✔
103
        $this->iconFactory = GeneralUtility::makeInstance(IconFactory::class);
17 ✔
104
        $this->configurationService = GeneralUtility::makeInstance(
17 ✔
105
            ConfigurationService::class,
17 ✔
106
            GeneralUtility::makeInstance(UrlService::class),
17 ✔
107
            $this->configurationRepository,
17 ✔
108
            GeneralUtility::makeInstance(GroupResolver::class),
17 ✔
109
        );
17 ✔
110
        $this->urlService = GeneralUtility::makeInstance(UrlService::class);
17 ✔
111

112
        $configurationProvider = GeneralUtility::makeInstance(ExtensionConfigurationProvider::class);
17 ✔
113
        $this->extensionSettings = $configurationProvider->getExtensionConfiguration();
17 ✔
114
        $this->setMaximumUrlsToCompile(
17 ✔
115
            MathUtility::forceIntegerInRange($this->extensionSettings['maxCompileUrls'], 1, 1000000000, 10000)
17 ✔
116
        );
17 ✔
117
    }
118

119
    public function injectCrawlUrlMessageDispatcher(CrawlUrlMessageDispatcher $crawlUrlMessageDispatcher): void
120
    {
121
        $this->crawlUrlMessageDispatcher = $crawlUrlMessageDispatcher;
16 ✔
122
    }
123

124
    public function setMaximumUrlsToCompile(int $maximumUrlsToCompile): void
125
    {
126
        $this->maximumUrlsToCompile = $maximumUrlsToCompile;
17 ✔
127
    }
128

129
    /**
130
     * Sets the extensions settings (unserialized pendant of $TYPO3_CONF_VARS['EXT']['extConf']['crawler']).
131
     */
132
    public function setExtensionSettings(array $extensionSettings): void
133
    {
134
        $this->extensionSettings = $extensionSettings;
1 ✔
135
    }
136

137
    /**
138
     * Wrapper method for getUrlsForPageId()
139
     * It returns an array of configurations and no urls!
140
     *
141
     * @param array $pageRow Page record with at least dok-type and uid columns.
142
     * @see getUrlsForPageId()
143
     */
144
    public function getUrlsForPageRow(array $pageRow, string &$skipMessage = ''): array
145
    {
146
        if (!isset($pageRow['uid'])) {
16 ✔
147
            $skipMessage = "pageRow['uid'] is missing";
1 ✔
148
            return [];
1 ✔
149
        }
150

151
        $pageRowUid = intval($pageRow['uid']);
15 ✔
152
        if (!$pageRowUid) {
15 ✔
153
            $skipMessage = 'PageUid "' . $pageRow['uid'] . '" was not an integer';
2 ✔
154
            return [];
2 ✔
155
        }
156

157
        $message = $this->getPageService()->checkIfPageShouldBeSkipped($pageRow);
13 ✔
158
        if ($message === false) {
13 ✔
159
            $res = $this->getUrlsForPageId($pageRowUid);
12 ✔
160
            $skipMessage = '';
12 ✔
161
        } else {
162
            $skipMessage = $message;
2 ✔
163
            $res = [];
2 ✔
164
        }
165

166
        return $res;
13 ✔
167
    }
168

169
    /**
170
     * Creates a list of URLs from input array (and submits them to queue if asked for)
171
     * See Web > Info module script + "indexed_search"'s crawler hook-client using this!
172
     *
173
     * @param array $vv Information about URLs from pageRow to crawl.
174
     * @param array $pageRow Page row
175
     * @param int $scheduledTime Unix time to schedule indexing to, typically time()
176
     * @param int $reqMinute Number of requests per minute (creates the interleave between requests)
177
     * @param bool $submitCrawlUrls If set, submits the URLs to queue
178
     * @param bool $downloadCrawlUrls If set (and submitcrawlUrls is false) will fill $downloadUrls with entries)
179
     * @param array $duplicateTrack Array which is passed by reference and contains the an id per url to secure we will not crawl duplicates
180
     * @param array $downloadUrls Array which will be filled with URLS for download if flag is set.
181
     * @param array $incomingProcInstructions Array of processing instructions
182
     * @return string List of URLs (meant for display in backend module)
183
     */
184
    public function urlListFromUrlArray(
185
        array $vv,
186
        array $pageRow,
187
        int $scheduledTime,
188
        int $reqMinute,
189
        bool $submitCrawlUrls,
190
        bool $downloadCrawlUrls,
191
        array &$duplicateTrack,
192
        array &$downloadUrls,
193
        array $incomingProcInstructions
194
    ): string {
195
        if (!is_array($vv['URLs'])) {
10 ✔
196
            return 'ERROR - no URL generated';
×
197
        }
198
        $urlLog = [];
10 ✔
199
        $pageId = (int) $pageRow['uid'];
10 ✔
200
        $configurationHash = $this->getConfigurationHash($vv);
10 ✔
201
        $skipInnerCheck = $this->queueRepository->noUnprocessedQueueEntriesForPageWithConfigurationHashExist(
10 ✔
202
            $pageId,
10 ✔
203
            $configurationHash
10 ✔
204
        );
10 ✔
205

206
        $processInstructionService = new ProcessInstructionService();
10 ✔
207

208
        foreach ($vv['URLs'] as $urlQuery) {
10 ✔
209
            if (!$processInstructionService->isAllowed(
10 ✔
210
                $vv['subCfg']['procInstrFilter'] ?? '',
10 ✔
211
                $incomingProcInstructions
10 ✔
212
            )) {
10 ✔
213
                continue;
×
214
            }
215
            $url = $this->urlService->getUrlFromPageAndQueryParameters(
10 ✔
216
                $pageId,
10 ✔
217
                $urlQuery,
10 ✔
218
                $vv['subCfg']['baseUrl'] ?? null,
10 ✔
219
                (int) ($vv['subCfg']['force_ssl'] ?? 0)
10 ✔
220
            );
10 ✔
221

222
            if (!$url instanceof UriInterface) {
10 ✔
223
                continue;
×
224
            }
225

226
            $url = (string) $url;
10 ✔
227

228
            // Create key by which to determine unique-ness:
229
            $uKey = $url . '|' . ($vv['subCfg']['userGroups'] ?? '') . '|' . ($vv['subCfg']['procInstrFilter'] ?? '');
10 ✔
230

231
            if (isset($duplicateTrack[$uKey])) {
10 ✔
232
                //if the url key is registered just display it and do not resubmit is
233
                $urlLog[] = '<em><span class="text-muted">' . htmlspecialchars($url) . '</span></em>';
×
234
            } else {
235
                // Scheduled time:
236
                $schTime = $scheduledTime + round(count($duplicateTrack) * (60 / $reqMinute));
10 ✔
237
                $schTime = intval($schTime / 60) * 60;
10 ✔
238
                $formattedDate = BackendUtility::datetime($schTime);
10 ✔
239
                $this->urlList[] = '[' . $formattedDate . '] ' . $url;
10 ✔
240
                $urlList = '[' . $formattedDate . '] ' . htmlspecialchars($url);
10 ✔
241

242
                // Submit for crawling!
243
                if ($submitCrawlUrls) {
10 ✔
244
                    $added = $this->addUrl(
9 ✔
245
                        $pageId,
9 ✔
246
                        $url,
9 ✔
247
                        $vv['subCfg'],
9 ✔
248
                        $scheduledTime,
9 ✔
249
                        $configurationHash,
9 ✔
250
                        $skipInnerCheck
9 ✔
251
                    );
9 ✔
252
                    if ($added === false) {
9 ✔
253
                        $urlList .= ' (URL already existed)';
4 ✔
254
                    }
255
                } elseif ($downloadCrawlUrls) {
1 ✔
256
                    $downloadUrls[$url] = $url;
1 ✔
257
                }
258
                $urlLog[] = $urlList;
10 ✔
259
            }
260
            $duplicateTrack[$uKey] = true;
10 ✔
261
        }
262

263
        // Todo: Find a better option to have this correct in both backend (<br>) and cli (<new line>)
264
        return implode('<br>', $urlLog);
10 ✔
265
    }
266

267
    public function getPageTSconfigForId(int $id): array
268
    {
269
        if (!$this->MP) {
11 ✔
270
            $pageTSconfig = BackendUtility::getPagesTSconfig($id);
11 ✔
271
        } else {
272
            [, $mountPointId] = explode('-', $this->MP);
×
273
            $pageTSconfig = BackendUtility::getPagesTSconfig((int) $mountPointId);
×
274
        }
275

276
        // Call a hook to alter configuration
277
        if (
278
            isset($GLOBALS['TYPO3_CONF_VARS']['EXTCONF']['crawler']['getPageTSconfigForId'])
11 ✔
279
            && is_array($GLOBALS['TYPO3_CONF_VARS']['EXTCONF']['crawler']['getPageTSconfigForId'])
11 ✔
280
        ) {
281
            $params = [
×
282
                'pageId' => $id,
×
283
                'pageTSConfig' => &$pageTSconfig,
×
284
            ];
×
285
            foreach ($GLOBALS['TYPO3_CONF_VARS']['EXTCONF']['crawler']['getPageTSconfigForId'] as $userFunc) {
×
286
                GeneralUtility::callUserFunction($userFunc, $params, $this);
×
287
            }
288
        }
289
        return $pageTSconfig;
11 ✔
290
    }
291

292
    /**
293
     * This method returns an array of configurations.
294
     * Adds no urls!
295
     */
296
    public function getUrlsForPageId(int $pageId): array
297
    {
298
        // Get page TSconfig for page ID
299
        $pageTSconfig = $this->getPageTSconfigForId($pageId);
10 ✔
300

301
        $mountPoint = $this->MP ?? '';
10 ✔
302

303
        $res = [];
10 ✔
304

305
        // Fetch Crawler Configuration from pageTSConfig
306
        $res = $this->configurationService->getConfigurationFromPageTS($pageTSconfig, $pageId, $res, $mountPoint);
10 ✔
307

308
        // Get configuration from tx_crawler_configuration records up the rootline
309
        $res = $this->configurationService->getConfigurationFromDatabase($pageId, $res);
10 ✔
310

311
        foreach ($GLOBALS['TYPO3_CONF_VARS']['EXTCONF']['crawler']['processUrls'] ?? [] as $func) {
10 ✔
312
            $params = [
×
313
                'res' => &$res,
×
314
            ];
×
315
            GeneralUtility::callUserFunction($func, $params, $this);
×
316
        }
317
        return $res;
10 ✔
318
    }
319

320
    /**
321
     * Find all configurations of subpages of a page
322
     * TODO: Write Functional Tests
323
     */
324
    public function getConfigurationsForBranch(int $rootid, int $depth): array
325
    {
326
        $configurationsForBranch = [];
1 ✔
327
        $pageTSconfig = $this->getPageTSconfigForId($rootid);
1 ✔
328
        $sets = $pageTSconfig['tx_crawler.']['crawlerCfg.']['paramSets.'] ?? [];
1 ✔
329
        foreach ($sets as $key => $value) {
1 ✔
330
            if (!is_array($value)) {
×
331
                continue;
×
332
            }
333
            $configurationsForBranch[] = str_ends_with((string) $key, '.') ? substr((string) $key, 0, -1) : $key;
×
334
        }
335
        $pids = [];
1 ✔
336
        $rootLine = BackendUtility::BEgetRootLine($rootid);
1 ✔
337
        foreach ($rootLine as $node) {
1 ✔
338
            $pids[] = $node['uid'];
1 ✔
339
        }
340
        $tree = GeneralUtility::makeInstance(PageTreeView::class);
1 ✔
341
        $perms_clause = $this->getBackendUser()->getPagePermsClause(Permission::PAGE_SHOW);
1 ✔
342
        $tree->init(empty($perms_clause) ? '' : ('AND ' . $perms_clause));
1 ✔
343
        $tree->getTree($rootid, $depth, '');
1 ✔
344
        foreach ($tree->tree as $node) {
1 ✔
345
            $pids[] = $node['row']['uid'];
×
346
        }
347

348
        $configurations = $this->configurationRepository->getCrawlerConfigurationRecordsFromRootLine($rootid, $pids);
1 ✔
349

350
        foreach ($configurations as $configuration) {
1 ✔
351
            $configurationsForBranch[] = $configuration['name'];
1 ✔
352
        }
353
        return $configurationsForBranch;
1 ✔
354
    }
355

356
    /************************************
357
     *
358
     * Crawler log
359
     *
360
     ************************************/
361

362
    /**
363
     * Adding call back entries to log (called from hooks typically, see indexed search class "class.crawler.php"
364
     *
365
     * @param integer $setId Set ID
366
     * @param array $params Parameters to pass to call back function
367
     * @param string $callBack Call back object reference, eg. 'EXT:indexed_search/class.crawler.php:&tx_indexedsearch_crawler'
368
     * @param integer $page_id Page ID to attach it to
369
     * @param integer $schedule Time at which to activate
370
     *
371
     * @deprecated since 12.0.5 will be removed in 14.x
372
     */
373
    public function addQueueEntry_callBack(
374
        int $setId,
375
        array $params,
376
        string $callBack,
377
        int $page_id = 0,
378
        int $schedule = 0
379
    ): void {
380
        $params['_CALLBACKOBJ'] = $callBack;
×
NEW
381
        $scheduled = $schedule ?: $this->getCurrentTime();
×
382

NEW
383
        $queueId = $this->queueRepository->add([
×
NEW
384
            'page_id' => $page_id,
×
NEW
385
            'parameters' => json_encode($params),
×
NEW
386
            'scheduled' => $scheduled,
×
NEW
387
            'exec_time' => 0,
×
NEW
388
            'set_id' => $setId,
×
NEW
389
            'result_data' => '',
×
NEW
390
        ]);
×
NEW
391
        $this->dispatchCrawlUrlMessage($queueId, $page_id, $scheduled);
×
392
    }
393

394
    /************************************
395
     *
396
     * URL setting
397
     *
398
     ************************************/
399

400
    /**
401
     * Setting a URL for crawling:
402
     *
403
     * @param integer $id Page ID
404
     * @param string $url Complete URL
405
     * @param array $subCfg Sub configuration array (from TS config)
406
     * @param integer $tstamp Scheduled-time
407
     * @param string $configurationHash (optional) configuration hash
408
     * @param bool $skipInnerDuplicationCheck (optional) skip inner duplication check
409
     * @return bool
410
     */
411
    public function addUrl(
412
        $id,
413
        $url,
414
        array $subCfg,
415
        $tstamp,
416
        $configurationHash = '',
417
        $skipInnerDuplicationCheck = false
418
    ) {
419
        $urlAdded = false;
14 ✔
420
        $rows = [];
14 ✔
421

422
        // Creating parameters:
423
        $parameters = [
14 ✔
424
            'url' => $url,
14 ✔
425
        ];
14 ✔
426

427
        // fe user group simulation:
428
        $uGs = implode(',', array_unique(GeneralUtility::intExplode(',', $subCfg['userGroups'] ?? '', true)));
14 ✔
429
        if ($uGs) {
14 ✔
430
            $parameters['feUserGroupList'] = $uGs;
1 ✔
431
        }
432

433
        // Setting processing instructions
434
        $parameters['procInstructions'] = GeneralUtility::trimExplode(',', $subCfg['procInstrFilter'] ?? '');
14 ✔
435
        if (is_array($subCfg['procInstrParams.'] ?? false)) {
14 ✔
436
            $parameters['procInstrParams'] = $subCfg['procInstrParams.'];
10 ✔
437
        }
438

439
        // Compile value array:
440
        $parameters_serialized = json_encode($parameters) ?: '';
14 ✔
441
        $fieldArray = [
14 ✔
442
            'page_id' => (int) $id,
14 ✔
443
            'parameters' => $parameters_serialized,
14 ✔
444
            'parameters_hash' => md5($parameters_serialized),
14 ✔
445
            'configuration_hash' => $configurationHash,
14 ✔
446
            'scheduled' => $tstamp,
14 ✔
447
            'exec_time' => 0,
14 ✔
448
            'set_id' => $this->setID,
14 ✔
449
            'result_data' => '',
14 ✔
450
            'configuration' => $subCfg['key'],
14 ✔
451
        ];
14 ✔
452

453
        if ($this->registerQueueEntriesInternallyOnly) {
14 ✔
454
            //the entries will only be registered and not stored to the database
455
            $this->queueEntries[] = $fieldArray;
3 ✔
456
        } else {
457
            if (!$skipInnerDuplicationCheck) {
11 ✔
458
                // check if there is already an equal entry
459
                $rows = $this->queueRepository->getDuplicateQueueItemsIfExists(
6 ✔
460
                    (bool) $this->extensionSettings['enableTimeslot'],
6 ✔
461
                    $tstamp,
6 ✔
462
                    $this->getCurrentTime(),
6 ✔
463
                    $fieldArray['page_id'],
6 ✔
464
                    $fieldArray['parameters_hash']
6 ✔
465
                );
6 ✔
466
            }
467
            if ($rows === []) {
11 ✔
468
                $uid = $this->queueRepository->add($fieldArray);
10 ✔
469
                $urlAdded = true;
10 ✔
470

471
                $this->eventDispatcher->dispatch(new AfterUrlAddedToQueueEvent((string) $uid, $fieldArray));
10 ✔
472
                $this->dispatchCrawlUrlMessage($uid, (int) $id, (int) $tstamp);
10 ✔
473
            }
474
        }
475

476
        return $urlAdded;
14 ✔
477
    }
478

479
    /**
480
     * Returns the current system time
481
     *
482
     * @return int
483
     */
484
    public function getCurrentTime()
485
    {
486
        return time();
10 ✔
487
    }
488

489
    /************************************
490
     *
491
     * URL reading
492
     *
493
     ************************************/
494
    /*****************************
495
     *
496
     * Compiling URLs to crawl - tools
497
     *
498
     *****************************/
499

500
    /**
501
     * This draws the pageTree with URLs for e.g the Backend Log Module
502
     *
503
     * @param integer $id Root page id to start from.
504
     * @param integer $depth Depth of tree, 0=only id-page, 1= on sublevel, 99 = infinite
505
     * @param integer $scheduledTime Unix Time when the URL is timed to be visited when put in queue
506
     * @param integer $reqMinute Number of requests per minute (creates the interleave between requests)
507
     * @param boolean $submitCrawlUrls If set, submits the URLs to queue in database (real crawling)
508
     * @param boolean $downloadCrawlUrls If set (and submitcrawlUrls is false) will fill $downloadUrls with entries)
509
     * @param array $incomingProcInstructions Array of processing instructions
510
     * @param array $configurationSelection Array of configuration keys
511
     * @return array
512
     */
513
    public function getPageTreeAndUrls(
514
        $id,
515
        $depth,
516
        $scheduledTime,
517
        $reqMinute,
518
        $submitCrawlUrls,
519
        $downloadCrawlUrls,
520
        array $incomingProcInstructions,
521
        array $configurationSelection
522
    ) {
523
        $this->scheduledTime = $scheduledTime;
7 ✔
524
        $this->reqMinute = $reqMinute;
7 ✔
525
        $this->submitCrawlUrls = $submitCrawlUrls;
7 ✔
526
        $this->downloadCrawlUrls = $downloadCrawlUrls;
7 ✔
527
        $this->incomingProcInstructions = $incomingProcInstructions;
7 ✔
528
        $this->incomingConfigurationSelection = $configurationSelection;
7 ✔
529

530
        $this->duplicateTrack = [];
7 ✔
531
        $this->downloadUrls = [];
7 ✔
532

533
        // Drawing tree:
534
        $tree = GeneralUtility::makeInstance(PageTreeView::class);
7 ✔
535
        $perms_clause = $this->getBackendUser()->getPagePermsClause(Permission::PAGE_SHOW);
7 ✔
536
        $tree->init(empty($perms_clause) ? '' : ('AND ' . $perms_clause));
7 ✔
537

538
        $pageInfo = BackendUtility::readPageAccess($id, $perms_clause);
7 ✔
539
        if (is_array($pageInfo)) {
7 ✔
540
            // Set root row:
541
            $tree->tree[] = [
7 ✔
542
                'row' => $pageInfo,
7 ✔
543
                'HTML' => $this->iconFactory->getIconForRecord('pages', $pageInfo, IconSize::SMALL),
7 ✔
544
            ];
7 ✔
545
        }
546

547
        // Get branch beneath:
548
        if ($depth) {
7 ✔
549
            $tree->getTree($id, $depth, '');
1 ✔
550
        }
551

552
        $queueRows = [];
7 ✔
553

554
        // Traverse page tree:
555
        foreach ($tree->tree as $data) {
7 ✔
556
            $this->MP = null;
7 ✔
557

558
            // recognize mount points
559
            if ($data['row']['doktype'] === PageRepository::DOKTYPE_MOUNTPOINT) {
7 ✔
560
                $mountpage = $this->pageRepository->getPage($data['row']['uid']);
×
561

562
                // fetch mounted pages
563
                $this->MP = $mountpage['mount_pid'] . '-' . $data['row']['uid'];
×
564

565
                $mountTree = GeneralUtility::makeInstance(PageTreeView::class);
×
566
                $mountTree->init(empty($perms_clause) ? '' : ('AND ' . $perms_clause));
×
567
                $mountTree->getTree($mountpage['mount_pid'], $depth);
×
568

569
                foreach ($mountTree->tree as $mountData) {
×
570
                    $queueRows = array_merge($queueRows, $this->drawURLs_addRowsForPage(
×
571
                        $mountData['row'],
×
572
                        BackendUtility::getRecordTitle('pages', $mountData['row'], true),
×
573
                        (string) $data['HTML']
×
574
                    ));
×
575
                }
576

577
                // replace page when mount_pid_ol is enabled
578
                if ($mountpage['mount_pid_ol']) {
×
579
                    $data['row']['uid'] = $mountpage['mount_pid'];
×
580
                } else {
581
                    // if the mount_pid_ol is not set the MP must not be used for the mountpoint page
582
                    $this->MP = null;
×
583
                }
584
            }
585

586
            $queueRows = array_merge($queueRows, $this->drawURLs_addRowsForPage(
7 ✔
587
                $data['row'],
7 ✔
588
                BackendUtility::getRecordTitle('pages', $data['row'], true),
7 ✔
589
                (string) $data['HTML']
7 ✔
590
            ));
7 ✔
591
        }
592

593
        return $queueRows;
7 ✔
594
    }
595

596
    /**
597
     * Create the rows for display of the page tree
598
     * For each page a number of rows are shown displaying GET variable configuration
599
     */
600
    public function drawURLs_addRowsForPage(array $pageRow, string $pageTitle, string $pageTitleHTML = ''): array
601
    {
602
        $skipMessage = '';
7 ✔
603

604
        // Get list of configurations
605
        $configurations = $this->getUrlsForPageRow($pageRow, $skipMessage);
7 ✔
606
        $configurations = ConfigurationService::removeDisallowedConfigurations(
7 ✔
607
            $this->incomingConfigurationSelection,
7 ✔
608
            $configurations
7 ✔
609
        );
7 ✔
610

611
        // Traverse parameter combinations:
612
        $c = 0;
7 ✔
613

614
        $queueRowCollection = [];
7 ✔
615

616
        if (!empty($configurations)) {
7 ✔
617
            foreach ($configurations as $confKey => $confArray) {
7 ✔
618
                // Title column:
619
                if (!$c) {
7 ✔
620
                    $queueRow = new QueueRow($pageTitle);
7 ✔
621
                } else {
622
                    $queueRow = new QueueRow();
×
623
                }
624
                $queueRow->setPageTitleHTML($pageTitleHTML);
7 ✔
625

626
                if (!in_array(
7 ✔
627
                    $pageRow['uid'],
7 ✔
628
                    $this->configurationService->expandExcludeString($confArray['subCfg']['exclude'] ?? ''),
7 ✔
629
                    true
7 ✔
630
                )) {
7 ✔
631
                    // URL list:
632
                    $urlList = $this->urlListFromUrlArray(
7 ✔
633
                        $confArray,
7 ✔
634
                        $pageRow,
7 ✔
635
                        $this->scheduledTime,
7 ✔
636
                        $this->reqMinute,
7 ✔
637
                        $this->submitCrawlUrls,
7 ✔
638
                        $this->downloadCrawlUrls,
7 ✔
639
                        $this->duplicateTrack,
7 ✔
640
                        $this->downloadUrls,
7 ✔
641
                        // if empty the urls won't be filtered by processing instructions
642
                        $this->incomingProcInstructions
7 ✔
643
                    );
7 ✔
644

645
                    // Expanded parameters:
646
                    $paramExpanded = '';
7 ✔
647
                    $calcAccu = [];
7 ✔
648
                    $calcRes = 1;
7 ✔
649
                    foreach ($confArray['paramExpanded'] as $gVar => $gVal) {
7 ✔
650
                        $paramExpanded .= '
×
651
                            <tr>
652
                                <td>' . htmlspecialchars('&' . $gVar . '=') . '<br/>' .
×
653
                            '(' . count($gVal) . ')' .
×
654
                            '</td>
×
655
                                <td nowrap="nowrap">' . nl2br(
×
656
                                htmlspecialchars(implode(chr(10), $gVal))
×
657
                            ) . '</td>
×
658
                            </tr>
659
                        ';
×
660
                        $calcRes *= count($gVal);
×
661
                        $calcAccu[] = count($gVal);
×
662
                    }
663
                    if (!empty($paramExpanded)) {
7 ✔
664
                        $paramExpanded = "<table>{$paramExpanded}</table>";
×
665
                    }
666
                    $paramExpanded .= 'Comb: ' . implode('*', $calcAccu) . '=' . $calcRes;
7 ✔
667

668
                    // Options
669
                    $queueRowOptionCollection = [];
7 ✔
670
                    if ($confArray['subCfg']['userGroups'] ?? false) {
7 ✔
671
                        $queueRowOptionCollection[] = 'User Groups: ' . $confArray['subCfg']['userGroups'];
×
672
                    }
673
                    if ($confArray['subCfg']['procInstrFilter'] ?? false) {
7 ✔
674
                        $queueRowOptionCollection[] = 'ProcInstr: ' . $confArray['subCfg']['procInstrFilter'];
×
675
                    }
676

677
                    $parameterConfig = nl2br(
7 ✔
678
                        htmlspecialchars(rawurldecode(
7 ✔
679
                            trim(str_replace(
7 ✔
680
                                '&',
7 ✔
681
                                chr(10) . '&',
7 ✔
682
                                GeneralUtility::implodeArrayForUrl('', $confArray['paramParsed'] ?? [])
7 ✔
683
                            ))
7 ✔
684
                        ))
7 ✔
685
                    );
7 ✔
686
                    $queueRow->setValuesExpanded($paramExpanded);
7 ✔
687
                    $queueRow->setConfigurationKey($confKey);
7 ✔
688
                    $queueRow->setUrls($urlList);
7 ✔
689
                    $queueRow->setOptions($queueRowOptionCollection);
7 ✔
690
                    $queueRow->setParameters(DebugUtility::viewArray($confArray['subCfg']['procInstrParams.'] ?? []));
7 ✔
691
                    $queueRow->setParameterConfig($parameterConfig);
7 ✔
692
                } else {
693
                    $queueRow->setConfigurationKey($confKey);
×
694
                    $queueRow->setMessage('(Page is excluded in this configuration)');
×
695
                }
696
                $queueRowCollection[] = $queueRow;
7 ✔
697

698
                $c++;
7 ✔
699
            }
700
        } else {
701
            $message = !empty($skipMessage) ? ' (' . $skipMessage . ')' : '';
1 ✔
702
            $queueRow = new QueueRow($pageTitle);
1 ✔
703
            $queueRow->setPageTitleHTML($pageTitleHTML);
1 ✔
704
            $queueRow->setMessage($message);
1 ✔
705
            $queueRowCollection[] = $queueRow;
1 ✔
706
        }
707

708
        return $queueRowCollection;
7 ✔
709
    }
710

711
    /**
712
     * Returns a md5 hash generated from a serialized configuration array.
713
     *
714
     * @return string
715
     */
716
    protected function getConfigurationHash(array $configuration)
717
    {
718
        unset($configuration['paramExpanded']);
16 ✔
719
        unset($configuration['URLs']);
16 ✔
720
        return md5(serialize($configuration));
16 ✔
721
    }
722

723
    protected function getPageService(): PageService
724
    {
725
        return new PageService();
10 ✔
726
    }
727

728
    /**
729
     * Hands the queue record over to the message bus. Records scheduled in the future are delayed accordingly.
730
     */
731
    private function dispatchCrawlUrlMessage(int $queueId, int $pageId, int $scheduled): void
732
    {
733
        $this->crawlUrlMessageDispatcher->dispatch(
10 ✔
734
            new CrawlUrlMessage($queueId),
10 ✔
735
            $pageId,
10 ✔
736
            $scheduled - $this->getCurrentTime()
10 ✔
737
        );
10 ✔
738
    }
739

740
    /**
741
     * @return BackendUserAuthentication
742
     */
743
    private function getBackendUser()
744
    {
745
        // Make sure the _cli_ user is loaded
746
        Bootstrap::initializeBackendAuthentication();
8 ✔
747
        if ($this->backendUser === null) {
8 ✔
748
            $this->backendUser = $GLOBALS['BE_USER'];
8 ✔
749
        }
750
        return $this->backendUser;
8 ✔
751
    }
752
}
STATUS · Troubleshooting · Open an Issue · Sales · Support · CAREERS · ENTERPRISE · START FREE TRIAL · SCHEDULE DEMO
ANNOUNCEMENTS · TWITTER · TOS & SLA · Supported CI Services · What's a CI service? · Automated Testing

© 2026 Coveralls, Inc