PHPackages                             schliesser/sitecrawler - PHPackages - PHPackages  [Skip to content](#main-content)[PHPackages](/)[Directory](/)[Categories](/categories)[Trending](/trending)[Leaderboard](/leaderboard)[Changelog](/changelog)[Analyze](/analyze)[Collections](/collections)[Log in](/login)[Sign up](/register)

1. [Directory](/)
2. /
3. [Utility & Helpers](/categories/utility)
4. /
5. schliesser/sitecrawler

ActiveTypo3-cms-extension[Utility & Helpers](/categories/utility)

schliesser/sitecrawler
======================

TYPO3 sitemap crawler

v3.0.2(5mo ago)1136.0k↓54.8%1[1 PRs](https://github.com/schliesser/sitecrawler/pulls)GPL-2.0-or-laterPHPPHP ^8.1CI passing

Since Feb 25Pushed 3mo ago1 watchersCompare

[ Source](https://github.com/schliesser/sitecrawler)[ Packagist](https://packagist.org/packages/schliesser/sitecrawler)[ RSS](/packages/schliesser-sitecrawler/feed)WikiDiscussions main Synced yesterday

READMEChangelog (7)Dependencies (7)Versions (12)Used By (0)

TYPO3 Sitemap crawler
=====================

[](#typo3-sitemap-crawler)

This extension provides a Symfony command to crawl any sitemap including all sub sitemaps. It gathers all available urls and then calls each url. This way you can warm up the TYPO3 page cache. Any standard sitemap can be crawled: TYPO3, Shopware, ...

You can provide custom request headers in json format. For basic auth you need to base64 encode user:password in the header.

Since version 1.1.0 the sitecrawler can read `robots.txt` files to fetch all defined sitemaps from it.

Version 3.0.0 supports now gzipped sub sitemaps and TYPO3 v13. Dropped support for TYPO3 v11 and older.

Examples
--------

[](#examples)

Composer based:

```
bin/typo3 sitecrawler:crawl 'https://www.example.com/sitemap.xml'

# with custom request headers
bin/typo3 sitecrawler:crawl 'https://www.example.com/sitemap.xml' '{"Authorization": "Basic dXNlcjpwYXNzd29yZA==", "Cache-Control": "no-cache"}'

# Only list all gathered urls
bin/typo3 sitecrawler:crawl 'https://www.example.com/sitemap.xml' --list=txt

# Only list all gathered urls as json
bin/typo3 sitecrawler:crawl 'https://www.example.com/sitemap.xml' --list=json
```

Non composer:

```
typo3/sysext/core/bin/typo3 sitecrawler:crawl 'https://www.example.com/sitemap.xml'
```

Development
-----------

[](#development)

- Clone project and `cd` into the extension folder
- Install dependencies for tests: `composer install`
- Run tests with: `composer run test`

###  Health Score

53

—

FairBetter than 96% of packages

Maintenance77

Regular maintenance activity

Popularity35

Limited adoption so far

Community8

Small or concentrated contributor base

Maturity72

Established project with proven stability

 Bus Factor1

Top contributor holds 100% of commits — single point of failure

How is this calculated?**Maintenance (25%)** — Last commit recency, latest release date, and issue-to-star ratio. Uses a 2-year decay window.

**Popularity (30%)** — Total and monthly downloads, GitHub stars, and forks. Logarithmic scaling prevents top-heavy scores.

**Community (15%)** — Contributors, dependents, forks, watchers, and maintainers. Measures real ecosystem engagement.

**Maturity (30%)** — Project age, version count, PHP version support, and release stability.

###  Release Activity

Cadence

Every ~239 days

Recently: every ~315 days

Total

10

Last Release

167d ago

Major Versions

v1.1.2 → v2.0.02022-08-05

v2.0.1 → v3.0.02024-05-14

PHP version history (2 changes)v2.0.0PHP ^7.4 || ^8.0

v3.0.0PHP ^8.1

### Community

Maintainers

![](https://www.gravatar.com/avatar/1a00ae72b0b16c1f6cec5516a93cac144b5aec6e3ac824d9cf758636412a6736?d=identicon)[schliesser](/maintainers/schliesser)

---

Top Contributors

[![schliesser](https://avatars.githubusercontent.com/u/11655823?v=4)](https://github.com/schliesser "schliesser (41 commits)")

---

Tags

composercrawlerrobots-txtsitemapsitemap-xmltypo3typo3-extensiontypo3-sitemap-crawler

###  Code Quality

Static AnalysisPHPStan

Code StylePHP CS Fixer

Type Coverage Yes

### Embed Badge

![Health badge](/badges/schliesser-sitecrawler/health.svg)

```
[![Health](https://phpackages.com/badges/schliesser-sitecrawler/health.svg)](https://phpackages.com/packages/schliesser-sitecrawler)
```

###  Alternatives

[friendsoftypo3/content-blocks

TYPO3 CMS Content Blocks - Content Types API | Define reusable components via YAML

103519.9k53](/packages/friendsoftypo3-content-blocks)[wazum/sluggi

TYPO3 extension for URL slug management with inline editing, auto-sync, locking, access control, and redirects

40529.5k](/packages/wazum-sluggi)[typo3/cms-scheduler

TYPO3 CMS Scheduler - Schedule tasks to run once or periodically at a specific time.

169.3M231](/packages/typo3-cms-scheduler)[typo3/cms-lowlevel

TYPO3 CMS Lowlevel - Technical analysis of the system. This includes raw database search, checking relations, counting pages and records etc.

178.2M316](/packages/typo3-cms-lowlevel)[typo3/cms-redirects

TYPO3 CMS Redirects - Create manual redirects, list existing redirects and automatically createredirects on slug changes.

167.4M80](/packages/typo3-cms-redirects)[typo3/cms-form

TYPO3 CMS Form - Flexible TYPO3 frontend form framework that comes with a backend editor interface.

147.6M265](/packages/typo3-cms-form)

PHPackages © 2026

[Directory](/)[Categories](/categories)[Trending](/trending)[Changelog](/changelog)[Analyze](/analyze)