PHPackages                             schliesser/sitecrawler - PHPackages - PHPackages  [Skip to content](#main-content)[PHPackages](/)[Directory](/)[Categories](/categories)[Trending](/trending)[Leaderboard](/leaderboard)[Changelog](/changelog)[Analyze](/analyze)[Collections](/collections)[Log in](/login)[Sign up](/register)

1. [Directory](/)
2. /
3. schliesser/sitecrawler

ActiveTypo3-cms-extension

schliesser/sitecrawler
======================

TYPO3 sitemap crawler

v3.0.2(4mo ago)1133.2k↓32.7%1[1 PRs](https://github.com/schliesser/sitecrawler/pulls)GPL-2.0-or-laterPHPPHP ^8.1CI passing

Since Feb 25Pushed 1mo ago1 watchersCompare

[ Source](https://github.com/schliesser/sitecrawler)[ Packagist](https://packagist.org/packages/schliesser/sitecrawler)[ RSS](/packages/schliesser-sitecrawler/feed)WikiDiscussions main Synced 1mo ago

READMEChangelog (7)Dependencies (7)Versions (11)Used By (0)

TYPO3 Sitemap crawler
=====================

[](#typo3-sitemap-crawler)

This extension provides a Symfony command to crawl any sitemap including all sub sitemaps. It gathers all available urls and then calls each url. This way you can warm up the TYPO3 page cache. Any standard sitemap can be crawled: TYPO3, Shopware, ...

You can provide custom request headers in json format. For basic auth you need to base64 encode user:password in the header.

Since version 1.1.0 the sitecrawler can read `robots.txt` files to fetch all defined sitemaps from it.

Version 3.0.0 supports now gzipped sub sitemaps and TYPO3 v13. Dropped support for TYPO3 v11 and older.

Examples
--------

[](#examples)

Composer based:

```
bin/typo3 sitecrawler:crawl 'https://www.example.com/sitemap.xml'

# with custom request headers
bin/typo3 sitecrawler:crawl 'https://www.example.com/sitemap.xml' '{"Authorization": "Basic dXNlcjpwYXNzd29yZA==", "Cache-Control": "no-cache"}'

# Only list all gathered urls
bin/typo3 sitecrawler:crawl 'https://www.example.com/sitemap.xml' --list=txt

# Only list all gathered urls as json
bin/typo3 sitecrawler:crawl 'https://www.example.com/sitemap.xml' --list=json
```

Non composer:

```
typo3/sysext/core/bin/typo3 sitecrawler:crawl 'https://www.example.com/sitemap.xml'
```

Development
-----------

[](#development)

- Clone project and `cd` into the extension folder
- Install dependencies for tests: `composer install`
- Run tests with: `composer run test`

###  Health Score

54

—

FairBetter than 97% of packages

Maintenance85

Actively maintained with recent releases

Popularity35

Limited adoption so far

Community8

Small or concentrated contributor base

Maturity71

Established project with proven stability

 Bus Factor1

Top contributor holds 100% of commits — single point of failure

How is this calculated?**Maintenance (25%)** — Last commit recency, latest release date, and issue-to-star ratio. Uses a 2-year decay window.

**Popularity (30%)** — Total and monthly downloads, GitHub stars, and forks. Logarithmic scaling prevents top-heavy scores.

**Community (15%)** — Contributors, dependents, forks, watchers, and maintainers. Measures real ecosystem engagement.

**Maturity (30%)** — Project age, version count, PHP version support, and release stability.

###  Release Activity

Cadence

Every ~239 days

Recently: every ~315 days

Total

10

Last Release

121d ago

Major Versions

v1.1.2 → v2.0.02022-08-05

v2.0.1 → v3.0.02024-05-14

PHP version history (2 changes)v2.0.0PHP ^7.4 || ^8.0

v3.0.0PHP ^8.1

### Community

Maintainers

![](https://www.gravatar.com/avatar/1a00ae72b0b16c1f6cec5516a93cac144b5aec6e3ac824d9cf758636412a6736?d=identicon)[schliesser](/maintainers/schliesser)

---

Top Contributors

[![schliesser](https://avatars.githubusercontent.com/u/11655823?v=4)](https://github.com/schliesser "schliesser (41 commits)")

---

Tags

composercrawlerrobots-txtsitemapsitemap-xmltypo3typo3-extensiontypo3-sitemap-crawler

###  Code Quality

Static AnalysisPHPStan

Code StylePHP CS Fixer

Type Coverage Yes

### Embed Badge

![Health badge](/badges/schliesser-sitecrawler/health.svg)

```
[![Health](https://phpackages.com/badges/schliesser-sitecrawler/health.svg)](https://phpackages.com/packages/schliesser-sitecrawler)
```

###  Alternatives

[fluidtypo3/vhs

This is a collection of ViewHelpers for performing rendering tasks that are not natively provided by TYPO3's Fluid templating engine.

1954.1M49](/packages/fluidtypo3-vhs)[in2code/powermail

Powermail is a well-known, editor-friendly, powerful and easy to use mailform extension for TYPO3 with a lots of features

982.5M38](/packages/in2code-powermail)[typo3/testing-framework

The TYPO3 testing framework provides base classes for unit, functional and acceptance testing.

675.0M775](/packages/typo3-testing-framework)[fluidtypo3/flux

The flux package from FluidTYPO3

152982.2k20](/packages/fluidtypo3-flux)[netresearch/rte-ckeditor-image

Image support in CKEditor for the TYPO3 ecosystem - by Netresearch

63991.3k4](/packages/netresearch-rte-ckeditor-image)[friendsoftypo3/content-blocks

TYPO3 CMS Content Blocks - Content Types API | Define reusable components via YAML

96374.6k23](/packages/friendsoftypo3-content-blocks)

PHPackages © 2026

[Directory](/)[Categories](/categories)[Trending](/trending)[Changelog](/changelog)[Analyze](/analyze)