PHPackages                             ronappleton/webcrawler - PHPackages - PHPackages  [Skip to content](#main-content)[PHPackages](/)[Directory](/)[Categories](/categories)[Trending](/trending)[Leaderboard](/leaderboard)[Changelog](/changelog)[Analyze](/analyze)[Collections](/collections)[Log in](/login)[Sign up](/register)

1. [Directory](/)
2. /
3. [Utility & Helpers](/categories/utility)
4. /
5. ronappleton/webcrawler

ActiveLibrary[Utility & Helpers](/categories/utility)

ronappleton/webcrawler
======================

Webcrawler for crawling indexed sites i.e. options and indexes

114PHP

Since Oct 6Pushed 7y ago1 watchersCompare

[ Source](https://github.com/ronappleton/webcrawler)[ Packagist](https://packagist.org/packages/ronappleton/webcrawler)[ RSS](/packages/ronappleton-webcrawler/feed)WikiDiscussions master Synced 3d ago

READMEChangelogDependenciesVersions (1)Used By (0)

Web Crawler
===========

[](#web-crawler)

Simple web crawler for retrieving site links

This web crawler package is a simple package, designed for taking websites and extracting the files it can find from the html that the site provides.

It is restricted to the source domain by default, can be altered using the restrict\_domain option of the crawl method.

It was built for handling known self linking sites, although I will add controls to prevent external crawling when required.

It is simple to use, and solves some of the issues other people have had trying to build simple crawlers.

Supported
---------

[](#supported)

- Scanning and retrieving web page.
- Reading and pulling out all links in web page.
- Deducing if link is to another directory or to a file.
- Storing file and directory location (web location)
- Handles relative and non relative urls
- Times crawls
- Provides minimal count statistic
- Exports data collected as array
- Exports data collected as Json

Warning
-------

[](#warning)

Use this at your own risk, please don't crawl sites of people that are not expecting it, the risk is all yours

Simple Test Script
------------------

[](#simple-test-script)

A simple script for testing is included.

###  Health Score

20

—

LowBetter than 13% of packages

Maintenance20

Infrequent updates — may be unmaintained

Popularity7

Limited adoption so far

Community7

Small or concentrated contributor base

Maturity40

Maturing project, gaining track record

 Bus Factor1

Top contributor holds 100% of commits — single point of failure

How is this calculated?**Maintenance (25%)** — Last commit recency, latest release date, and issue-to-star ratio. Uses a 2-year decay window.

**Popularity (30%)** — Total and monthly downloads, GitHub stars, and forks. Logarithmic scaling prevents top-heavy scores.

**Community (15%)** — Contributors, dependents, forks, watchers, and maintainers. Measures real ecosystem engagement.

**Maturity (30%)** — Project age, version count, PHP version support, and release stability.

### Community

Maintainers

![](https://www.gravatar.com/avatar/6e2afdd94a6f21a4ec2c5df569b80c1d47d6f5d0428f866f55c4bc117856b58b?d=identicon)[Ron Appleton](/maintainers/Ron%20Appleton)

---

Top Contributors

[![ronappleton](https://avatars.githubusercontent.com/u/3792420?v=4)](https://github.com/ronappleton "ronappleton (1 commits)")

### Embed Badge

![Health badge](/badges/ronappleton-webcrawler/health.svg)

```
[![Health](https://phpackages.com/badges/ronappleton-webcrawler/health.svg)](https://phpackages.com/packages/ronappleton-webcrawler)
```

###  Alternatives

[ibrahimbougaoua/filament-sort-order

This is my package filament-sort-order

2116.0k](/packages/ibrahimbougaoua-filament-sort-order)

PHPackages © 2026

[Directory](/)[Categories](/categories)[Trending](/trending)[Changelog](/changelog)[Analyze](/analyze)