PHPackages                             zrashwani/arachnid - PHPackages - PHPackages  [Skip to content](#main-content)[PHPackages](/)[Directory](/)[Categories](/categories)[Trending](/trending)[Leaderboard](/leaderboard)[Changelog](/changelog)[Analyze](/analyze)[Collections](/collections)[Log in](/login)[Sign up](/register)

1. [Directory](/)
2. /
3. [Search & Filtering](/categories/search)
4. /
5. zrashwani/arachnid

ActiveLibrary[Search & Filtering](/categories/search)

zrashwani/arachnid
==================

A crawler to find all unique internal pages on a given website

2.2.1(4y ago)25420.2k59[5 issues](https://github.com/zrashwani/arachnid/issues)MITPHPPHP >=7.2.0CI failing

Since Jan 6Pushed 3y ago20 watchersCompare

[ Source](https://github.com/zrashwani/arachnid)[ Packagist](https://packagist.org/packages/zrashwani/arachnid)[ Docs](http://github.com/zrashwani/arachnid)[ RSS](/packages/zrashwani-arachnid/feed)WikiDiscussions master Synced 2mo ago

READMEChangelog (10)Dependencies (10)Versions (17)Used By (0)

Arachnid Web Crawler
====================

[](#arachnid-web-crawler)

This library will crawl all unique internal links found on a given website up to a specified maximum page depth.

This library is using [*symfony/panther*](https://github.com/symfony/panther) & [FriendsOfPHP/Goutte](https://github.com/FriendsOfPHP/Goutte) libraries to scrap site pages and extract main SEO-related info, including: `title`, `h1 elements`, `h2 elements`, `statusCode`, `contentType`, `meta description`, `meta keyword` and `canonicalLink`.

This library is based on the original blog post by Zeid Rashwani here:

Josh Lockhart adapted the original blog post's code (with permission) for Composer and Packagist and updated the syntax to conform with the PSR-2 coding standard.

[![Build Status](https://camo.githubusercontent.com/e50dd4af53c1f64545d0f0d32b2cfe48b5c28c53b1342bf5c28f185b4f25cb68/68747470733a2f2f7472617669732d63692e636f6d2f7a7261736877616e692f61726163686e69642e7376673f6272616e63683d6d6173746572)](https://travis-ci.com/zrashwani/arachnid)[![codecov](https://camo.githubusercontent.com/ea57350b660cbb5a7796d70dd3f997719c40a442cc60850c52855f64bc2d5dda/68747470733a2f2f636f6465636f762e696f2f67682f7a7261736877616e692f61726163686e69642f6272616e63682f6d61737465722f67726170682f62616467652e737667)](https://codecov.io/gh/zrashwani/arachnid)

Sponsored By
------------

[](#sponsored-by)

[![](./oxylabs.png)](https://oxylabs.go2cloud.org/aff_c?offer_id=7&aff_id=447&url_id=32)

How to Install
--------------

[](#how-to-install)

You can install this library with [Composer](http://getcomposer.org/). Drop this into your `composer.json`manifest file:

```
{
    "require": {
        "zrashwani/arachnid": "dev-master"
    }
}

```

Then run `composer install`.

Getting Started
---------------

[](#getting-started)

### Basic Usage:

[](#basic-usage)

Here's a quick demo to crawl a website:

```