• Home
  • Features
  • Pricing
  • Docs
  • Announcements
  • Sign In

tomasnorre / crawler / 43818818671

16 Nov 2026 06:44AM UTC coverage: 68.855% (-0.07%) from 68.921%
43818818671

push

github

web-flow
!!! [TASK] Keep the state of compiling URLs in a CrawlRun (#1340)

* !!! [TASK] Keep the state of compiling URLs in a CrawlRun

QueueEntryWriter and PageTreeUrlCollector are shared services, but kept
the options and results of a run in public properties: the set id, the
handled URLs, the download URLs, the URL list and the collected queue
entries. Callers had to set and reset them, and state could leak from
one run into the next.

AOE\Crawler\Queue\CrawlRun now holds the options of one run and collects
its results. It is created per run and handed over as parameter, so both
services are stateless.

* QueueEntryWriter::compileUrlList() and addUrl() take a CrawlRun
* PageTreeUrlCollector::collect() and getQueueRowsForPage() take a CrawlRun
* QueueEntryWriter::addCallback() takes the set id as optional parameter
* BackendModuleStartCrawlingController no longer needs the QueueEntryWriter

The new unit tests of compileUrlList() covered code that had no unit tests
before, and the mutants escaping there lowered the covered code MSI.

* CrawlRun builds the key of a handled URL and calculates the scheduled
  time of the next URL, both tested exactly
* compileUrlList() checks the processing instruction filter once instead
  of for every URL, it does not change within a configuration
* Tests for base URL and force_ssl, URLs that cannot be built, user groups
  and processing instruction filters of handled URLs, numeric
  configuration keys and several parsed parameters

63 of 76 new or added lines in 6 files covered. (82.89%)

1269 of 1843 relevant lines covered (68.86%)

5.19 hits per line

Source File
Press 'n' to go to next uncovered line, 'b' for previous

86.44
/Classes/Command/BuildQueueCommand.php
1
<?php
2

3
declare(strict_types=1);
4

5
namespace AOE\Crawler\Command;
6

7
/*
8
 * (c) 2021 Tomas Norre Mikkelsen <tomasnorre@gmail.com>
9
 *
10
 * This file is part of the TYPO3 Crawler Extension.
11
 *
12
 * It is free software; you can redistribute it and/or modify it under
13
 * the terms of the GNU General Public License, either version 2
14
 * of the License, or any later version.
15
 *
16
 * For the full copyright and license information, please read the
17
 * LICENSE.txt file that was distributed with this source code.
18
 *
19
 * The TYPO3 project - inspiring people to share!
20
 */
21

22
use AOE\Crawler\Configuration\ExtensionConfigurationProvider;
23
use AOE\Crawler\Domain\Model\Reason;
24
use AOE\Crawler\Domain\Repository\QueueRepository;
25
use AOE\Crawler\Event\InvokeQueueChangeEvent;
26
use AOE\Crawler\Queue\CrawlRun;
27
use AOE\Crawler\Queue\Handler\CrawlUrlHandler;
28
use AOE\Crawler\Queue\Message\CrawlUrlMessage;
29
use AOE\Crawler\Queue\QueueEntryWriter;
30
use AOE\Crawler\Service\PageTreeUrlCollector;
31
use AOE\Crawler\Utility\MessageUtility;
32
use AOE\Crawler\Value\QueueRow;
33
use Symfony\Component\Console\Attribute\AsCommand;
34
use Symfony\Component\Console\Command\Command;
35
use Symfony\Component\Console\Helper\ProgressBar;
36
use Symfony\Component\Console\Input\InputArgument;
37
use Symfony\Component\Console\Input\InputInterface;
38
use Symfony\Component\Console\Input\InputOption;
39
use Symfony\Component\Console\Output\OutputInterface;
40
use TYPO3\CMS\Core\Domain\Repository\PageRepository;
41
use TYPO3\CMS\Core\EventDispatcher\EventDispatcher;
42
use TYPO3\CMS\Core\Utility\GeneralUtility;
43
use TYPO3\CMS\Core\Utility\MathUtility;
44

45
/**
46
 * @internal since v12.0.0
47
 */
48
#[AsCommand(name: 'crawler:buildQueue', description: 'Create entries in the queue that can be processed at once',)]
49
class BuildQueueCommand extends Command
50
{
51
    public function __construct(
52
        private readonly EventDispatcher $eventDispatcher,
53
        private readonly QueueRepository $queueRepository,
54
        private readonly PageRepository $pageRepository,
55
        private readonly PageTreeUrlCollector $pageTreeUrlCollector,
56
        private readonly QueueEntryWriter $queueEntryWriter,
57
        private readonly CrawlUrlHandler $crawlUrlHandler,
58
    ) {
59
        parent::__construct();
8 ✔
60
    }
61

62
    #[\Override]
63
    protected function configure(): void
64
    {
65
        $this->setDescription('Create entries in the queue that can be processed at once');
8 ✔
66

67
        $this->setHelp(
8 ✔
68
            'Try "typo3 help crawler:buildQueue" to see your options' . chr(10) . chr(10) .
8 ✔
69
            'Works as a CLI interface to some functionality from the Web > Info > Site Crawler module;
8 ✔
70
It can put entries in the queue from command line options, return the list of URLs and even execute
71
all entries right away without having to queue them up - this can be useful for immediate re-cache,
72
re-indexing or static publishing from command line.' . chr(10) . chr(10) .
8 ✔
73
            '
8 ✔
74
            Examples:
75
              --- Re-cache pages from page 7 and two levels down, executed immediately
76
              $ typo3 crawler:buildQueue 7 defaultConfiguration --depth 2 --mode exec
77

78
              --- Put entries for re-caching pages from page 7 into queue, 4 every minute.
79
              $ typo3 crawler:buildQueue 7 defaultConfiguration --depth 0 --mode queue --number 4
80
            '
8 ✔
81
        );
8 ✔
82

83
        $this->addArgument('page', InputArgument::REQUIRED, 'The page from where the queue building should start');
8 ✔
84

85
        $this->addArgument('conf', InputArgument::REQUIRED, 'A comma separated list of crawler configurations');
8 ✔
86

87
        $this->addOption(
8 ✔
88
            'depth',
8 ✔
89
            'd',
8 ✔
90
            InputOption::VALUE_OPTIONAL,
8 ✔
91
            'Tree depth, 0-99\', "How many levels under the \'page_id\' to include.',
8 ✔
92
            '0'
8 ✔
93
        );
8 ✔
94

95
        $this->addOption(
8 ✔
96
            'mode',
8 ✔
97
            'm',
8 ✔
98
            InputOption::VALUE_OPTIONAL,
8 ✔
99
            'Specifies output modes url : Will list URLs which wget could use as input. queue: Will put entries in the queue, to be processed by "typo3 messenger:consume crawler_high crawler crawler_low". exec: Will execute all entries right away!'
8 ✔
100
        );
8 ✔
101

102
        $this->addOption(
8 ✔
103
            'number',
8 ✔
104
            '',
8 ✔
105
            InputOption::VALUE_OPTIONAL,
8 ✔
106
            'Specifies how many items are put in the queue per minute. Only valid for output mode "queue"',
8 ✔
107
            '0'
8 ✔
108
        );
8 ✔
109
    }
110

111
    /**
112
     * Crawler Command - Submitting URLs to be crawled.
113
     *
114
     * Works as a CLI interface to some functionality from the Web > Info > Site Crawler module;
115
     * It can put entries in the queue from command line options, return the list of URLs and even execute
116
     * all entries right away without having to queue them up - this can be useful for immediate re-cache,
117
     * re-indexing or static publishing from command line.
118
     *
119
     * Examples:
120
     *
121
     * --- Re-cache pages from page 7 and two levels down, executed immediately
122
     * $ typo3 crawler:buildQueue 7 defaultConfiguration --depth 2 --mode exec
123
     *
124
     *
125
     * --- Put entries for re-caching pages from page 7 into queue, 4 every minute.
126
     * $ typo3 crawler:buildQueue 7 defaultConfiguration --depth 0 --mode queue --number 4
127
     */
128
    #[\Override]
129
    protected function execute(InputInterface $input, OutputInterface $output): int
130
    {
131
        $mode = $input->getOption('mode') ?? 'queue';
8 ✔
132

133
        $extensionSettings = GeneralUtility::makeInstance(
8 ✔
134
            ExtensionConfigurationProvider::class
8 ✔
135
        )->getExtensionConfiguration();
8 ✔
136

137
        $pageId = MathUtility::forceIntegerInRange((int) $input->getArgument('page'), 0);
8 ✔
138
        if ($pageId === 0 || empty($this->pageRepository->getPage($pageId))) {
8 ✔
139
            $message = "Page {$pageId} is not a valid page, please check you root page id and try again.";
1 ✔
140
            MessageUtility::addErrorMessage($message);
1 ✔
141
            $output->writeln("<info>{$message}</info>");
1 ✔
142
            return Command::FAILURE;
1 ✔
143
        }
144

145
        $configurationKeys = $this->getConfigurationKeys((string) $input->getArgument('conf'));
7 ✔
146

147
        if ($mode === 'queue' || $mode === 'exec') {
7 ✔
148
            $reason = new Reason();
6 ✔
149
            $reason->setReason(Reason::REASON_CLI_SUBMIT);
6 ✔
150
            $reason->setDetailText('The cli script of the crawler added to the queue');
6 ✔
151
            $this->eventDispatcher->dispatch(new InvokeQueueChangeEvent($reason));
6 ✔
152
        }
153

154
        if ($extensionSettings['cleanUpOldQueueEntries']) {
7 ✔
155
            $this->queueRepository->cleanUpOldQueueEntries();
7 ✔
156
        }
157

158
        $run = $this->createCrawlRun($input, $mode, $configurationKeys);
7 ✔
159
        $queueRows = $this->pageTreeUrlCollector->collect(
7 ✔
160
            $pageId,
7 ✔
161
            MathUtility::forceIntegerInRange((int) $input->getOption('depth'), 0, 99),
7 ✔
162
            $run
7 ✔
163
        );
7 ✔
164

165
        match ($mode) {
7 ✔
166
            'url' => $output->writeln('<info>' . implode(PHP_EOL, $run->getDownloadUrls()) . PHP_EOL . '</info>'),
1 ✔
167
            'exec' => $this->outputModeExec($output, $queueRows, $run),
2 ✔
168
            'queue' => $this->outputModeQueue($output, $queueRows, $run),
4 ✔
NEW
169
            default => $this->outputModeDefault($output, $queueRows, $run),
×
170
        };
7 ✔
171

172
        return Command::SUCCESS;
7 ✔
173
    }
174

175
    /**
176
     * Obtains configuration keys from the CLI arguments
177
     */
178
    private function getConfigurationKeys(string $conf): array
179
    {
180
        $parameter = trim($conf);
7 ✔
181
        return $parameter !== '' ? GeneralUtility::trimExplode(',', $parameter) : [];
7 ✔
182
    }
183

184
    private function outputUrls(array $queueRows, OutputInterface $output): void
185
    {
186
        /** @var QueueRow $row */
187
        foreach ($queueRows as $row) {
6 ✔
188
            if (empty($row->message)) {
6 ✔
189
                $output->writeln('<info>' . $row->urls . '</info>');
6 ✔
190
            } else {
191
                $output->writeln(
1 ✔
192
                    '<comment>Page "' . $row->pageTitle . '" is not added to queue' . $row->message . '</comment>'
1 ✔
193
                );
1 ✔
194
            }
195
        }
196
    }
197

198
    private function outputModeDefault(OutputInterface $output, array $queueRows, CrawlRun $run): void
199
    {
200
        $output->writeln(
×
201
            '<info>' . count(
×
NEW
202
                $run->getUrlList()
×
203
            ) . ' entries found for processing. (Use "mode" to decide action):</info>' . PHP_EOL
×
204
        );
×
205
        $this->outputUrls($queueRows, $output);
×
206
    }
207

208
    private function outputModeQueue(OutputInterface $output, array $queueRows, CrawlRun $run): void
209
    {
210
        $output->writeln('<info>Putting ' . count($run->getUrlList()) . ' entries in queue:</info>' . PHP_EOL);
4 ✔
211
        $this->outputUrls($queueRows, $output);
4 ✔
212
    }
213

214
    private function outputModeExec(OutputInterface $output, array $queueRows, CrawlRun $run): void
215
    {
216
        $progressBar = new ProgressBar($output);
2 ✔
217
        $output->writeln('<info>Executing ' . count($run->getUrlList()) . ' requests right away:</info>');
2 ✔
218
        $this->outputUrls($queueRows, $output);
2 ✔
219
        $output->writeln('<info>Processing</info>' . PHP_EOL);
2 ✔
220

221
        foreach ($progressBar->iterate($run->getQueueEntries()) as $queueRec) {
2 ✔
222
            $p = json_decode((string) $queueRec['parameters'], true);
2 ✔
223
            if (is_bool($p)) {
2 ✔
224
                continue;
×
225
            }
226

227
            $progressBar->clear();
2 ✔
228
            if (empty($p['procInstructions'][0])) {
2 ✔
229
                $procInstructionsString = '';
2 ✔
230
            } else {
231
                $procInstructionsString = ' (' . implode(',', $p['procInstructions']) . ')';
×
232
            }
233
            $output->writeln('<info>' . $p['url'] . $procInstructionsString . ' => ' . '</info>');
2 ✔
234
            $progressBar->display();
2 ✔
235

236
            // Bypass the message queue and crawl the entry right away
237
            $result = ($this->crawlUrlHandler)(new CrawlUrlMessage($this->queueRepository->add($queueRec)));
2 ✔
238

239
            $resultContent = $result['content'] ?? '';
2 ✔
240
            $requestResult = json_decode((string) $resultContent, true);
2 ✔
241

242
            $progressBar->clear();
2 ✔
243
            if (is_array($requestResult)) {
2 ✔
244
                $resLog = array_key_exists('log', $requestResult)
2 ✔
245
                && is_array($requestResult['log']) ? chr(9) . chr(9) .
2 ✔
246
                    implode(PHP_EOL . chr(9) . chr(9), $requestResult['log']) : '';
2 ✔
247
                $output->writeln('<info>OK: ' . $resLog . '</info>' . PHP_EOL);
2 ✔
248
            } else {
249
                $output->writeln(
×
250
                    '<error>Error checking Crawler Result:  ' . substr(
×
251
                        (string) preg_replace('/\s+/', ' ', strip_tags((string) $resultContent)),
×
252
                        0,
×
253
                        30000
×
254
                    ) . '...' . PHP_EOL . '</error>' . PHP_EOL
×
255
                );
×
256
            }
257
            $progressBar->display();
2 ✔
258
        }
259
        $output->writeln('');
2 ✔
260
    }
261

262
    private function createCrawlRun(InputInterface $input, mixed $mode, array $configurationKeys): CrawlRun
263
    {
264
        return new CrawlRun(
7 ✔
265
            $this->queueEntryWriter->getCurrentTime(),
7 ✔
266
            MathUtility::forceIntegerInRange((int) $input->getOption('number') ?: 30, 1, 1000),
7 ✔
267
            submitUrls: $mode === 'queue' || $mode === 'exec',
7 ✔
268
            collectDownloadUrls: $mode === 'url',
7 ✔
269
            // Entries of the exec mode are crawled right away, instead of being stored in the queue first
270
            collectQueueEntriesOnly: $mode === 'exec',
7 ✔
271
            configurationSelection: $configurationKeys,
7 ✔
272
        );
7 ✔
273
    }
274
}
STATUS · Troubleshooting · Open an Issue · Sales · Support · CAREERS · ENTERPRISE · START FREE TRIAL · SCHEDULE DEMO
ANNOUNCEMENTS · TWITTER · TOS & SLA · Supported CI Services · What's a CI service? · Automated Testing

© 2026 Coveralls, Inc