PHPackages                             jez500/web-scraper-for-laravel - PHPackages - PHPackages  [Skip to content](#main-content)[PHPackages](/)[Directory](/)[Categories](/categories)[Trending](/trending)[Leaderboard](/leaderboard)[Changelog](/changelog)[Analyze](/analyze)[Collections](/collections)[Log in](/login)[Sign up](/register)

1. [Directory](/)
2. /
3. [Utility & Helpers](/categories/utility)
4. /
5. jez500/web-scraper-for-laravel

ActiveLibrary[Utility & Helpers](/categories/utility)

jez500/web-scraper-for-laravel
==============================

A web scraper for laravel

1.0.12(2mo ago)12.8k↑175.6%21PHPCI passing

Since Jan 27Pushed 2mo ago1 watchersCompare

[ Source](https://github.com/jez500/Web-scraper-for-Laravel)[ Packagist](https://packagist.org/packages/jez500/web-scraper-for-laravel)[ RSS](/packages/jez500-web-scraper-for-laravel/feed)WikiDiscussions master Synced today

READMEChangelog (4)Dependencies (12)Versions (16)Used By (1)

Web Scraper for Laravel
=======================

[](#web-scraper-for-laravel)

A powerful and flexible Laravel package for scraping external web pages and extracting structured data. Whether you need to fetch static HTML content or scrape JavaScript-rendered pages, this package provides a clean, fluent interface for all your web scraping needs.

Key Features
------------

[](#key-features)

- **HTTP Scraping**: Support for standard HTTP requests using Laravel's HTTP client
- **API Scraping**: Support for scraping JavaScript-rendered pages (using [jez500/seleniumbase-scrapper](https://github.com/jez500/seleniumbase-scrapper))
- **Extract Scraping**: Support for article extraction using [jez500/extract](https://github.com/jez500/extract-docker) (Mercury Parser)
- **Multiple Extraction Methods**: Extract data using CSS selectors, XPath expressions, regular expressions, or JSON dot notation
- **Typed Schemas**: Support for type-safe scrape schemas via DTOs and `fromDto(...)`
- **User Agent Rotation**: Built-in support for rotating user agents to avoid being blocked
- **Response Caching**: Cache responses to avoid repeated requests
- **Comprehensive Error Handling**: Robust error handling for network issues and parsing errors

Installation
------------

[](#installation)

```
composer require jez500/web-scraper-for-laravel
```

For JavaScript-rendered page scraping, you'll also need to install:

```
composer require jez500/seleniumbase-scrapper
```

For article extraction, you'll need to run the Extract service:

```
docker run -p 3000:3000 jez500/extract
# or use your own instance and set the URL via setExtractApiBaseUrl()
```

Quick Start
-----------

[](#quick-start)

### Basic Scraping

[](#basic-scraping)

```
use Jez500\WebScraperForLaravel\Facades\WebScraper;

$scraper = WebScraper::http()->from('https://example.com')->get();
$title = $scraper->getSelector('title')->first();
```

### Extract Structured Data

[](#extract-structured-data)

```
$data = WebScraper::http()
    ->from('https://example.com')
    ->get()
    ->getJson('user.email')
    ->first();
```

Usage Examples
--------------

[](#usage-examples)

### Basic HTTP Scraping

[](#basic-http-scraping)

```
use Jez500\WebScraperForLaravel\Facades\WebScraper;

// Get an instance of the scraper with the body of the page loaded
$scraper = WebScraper::http()->from('https://example.com')->get();

// Get the full page body
$body = $scraper->getBody();
```

### CSS Selector Extraction

[](#css-selector-extraction)

```
// Get the first title element
$title = $scraper->getSelector('title')->first();

// Get the content attribute of the first meta tag with property og:image
$image = $scraper->getSelector('meta[property=og:image]|content')->first();

// Get all paragraph innerHtml as an array
$paragraphs = $scraper->getSelector('p')->all();

// Get all links as an array
$links = $scraper->getSelector('a|href')->all();

// Get the first matching element with a specific class
$price = $scraper->getSelector('.product-price')->first();
```

### XPath Extraction

[](#xpath-extraction)

```
// Get the first h1 element
$h1 = $scraper->getXpath('//h1')->first();

// Get the href attribute of the first link
$linkHref = $scraper->getXpath('//a', 'attr', ['href'])->first();

// Get multiple elements with a specific attribute
$images = $scraper->getXpath('//img[@class="product-image"]')->all();

// Complex XPath query
$productTitle = $scraper->getXpath('//div[@class="product"]/h1')->first();
```

### Regular Expression Extraction

[](#regular-expression-extraction)

```
// Extract user from JSON-like string
$author = $scraper->getRegex('~"user"\:"(.*)"~')->first();

// Extract email addresses
$emails = $scraper->getRegex('~[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}~')->all();

// Extract phone numbers
$phones = $scraper->getRegex('~\b\d{3}[-.]?\d{3}[-.]?\d{4}\b~')->all();
```

### JSON Data Extraction

[](#json-data-extraction)

```
// Get nested JSON data from API response
$author = WebScraper::http()
    ->from('https://api.example.com/page.json')
    ->get()
    ->getJson('user.name')
    ->first();

// Extract all items from an array
$products = WebScraper::http()
    ->from('https://api.example.com/products.json')
    ->get()
    ->getJson('products.*.name')
    ->all();

// Extract multiple nested fields
$emails = WebScraper::http()
    ->from('https://api.example.com/users.json')
    ->get()
    ->getJson('users.*.email')
    ->all();
```

### JavaScript-Rendered Page Scraping

[](#javascript-rendered-page-scraping)

```
// Get title from a JavaScript-rendered page
$title = WebScraper::api()
    ->from('https://example.com')
    ->get()
    ->getSelector('title')
    ->first();

// Extract data from a Single Page Application
$data = WebScraper::api()
    ->from('https://example.com/app')
    ->get()
    ->getSelector('[data-content]')
    ->first();
```

### Article Extraction

[](#article-extraction)

Extract clean article content from web pages using Mercury Parser:

```
// Extract article content
$content = WebScraper::extract()
    ->from('https://example.com/article')
    ->get()
    ->getBody();

// With custom headers
$content = WebScraper::extract()
    ->from('https://example.com/article')
    ->setOptions(['headers' => ['Accept-Language' => 'en-US']])
    ->get()
    ->getBody();

// Use custom Extract API endpoint
$content = WebScraper::extract()
    ->setExtractApiBaseUrl('http://my-extract-service:3000/parser')
    ->from('https://example.com/article')
    ->get()
    ->getBody();
```

### Custom Drivers

[](#custom-drivers)

Register a custom driver in a service provider, then resolve it by name:

```
use Illuminate\Support\Facades\Http;
use Jez500\WebScraperForLaravel\AbstractWebScraper;
use Jez500\WebScraperForLaravel\Drivers\WebScraperDriverInterface;
use Jez500\WebScraperForLaravel\Facades\WebScraper;

WebScraper::extend('residential-proxy', function () {
    return new class implements WebScraperDriverInterface {
        public function fetch(AbstractWebScraper $scraper): string
        {
            $response = Http::withToken(config('services.proxy.token'))
                ->post(config('services.proxy.endpoint'), [
                    'url' => $scraper->getUrl(),
                    'country' => 'us',
                ]);

            return data_get($response->json(), 'html', '');
        }
    };
});

$scraper = WebScraper::driver('residential-proxy')
    ->from('https://example.com')
    ->get();

$title = $scraper->getSelector('title')->first();
```

`http()` and `api()` remain available for backwards compatibility, and they now resolve through the same driver layer internally.

### Typed Scrape Schemas

[](#typed-scrape-schemas)

```
use Jez500\WebScraperForLaravel\Dto\FieldExtractionDto;
use Jez500\WebScraperForLaravel\Dto\ScrapeSchemaDto;

// Define a schema for structured data extraction
$schema = ScrapeSchemaDto::fromArray([
    'fields' => [
        'title' => [
            'type' => 'css',
            'value' => 'title',
        ],
        'description' => [
            'type' => 'css',
            'value' => 'meta[name=description]|content',
        ],
        'image' => [
            'type' => 'xpath',
            'value' => '//meta[@property="og:image"]/@content',
        ],
    ],
]);

// Apply schema and get structured data
$data = WebScraper::http()
    ->from('https://example.com')
    ->get()
    ->fromDto($schema);

// $data will be: ['title' => '...', 'description' => '...', 'image' => '...']
```

`fromDto(...)` also accepts a single `FieldExtractionDto` when you only need one extraction rule:

```
$title = FieldExtractionDto::fromArray([
    'type' => 'css',
    'value' => 'title',
]);

$data = WebScraper::http()
    ->from('https://example.com')
    ->get()
    ->fromDto($title);

// $data will be: ['field' => 'Example Domain']
```

### Advanced Features

[](#advanced-features)

#### Custom User Agents

[](#custom-user-agents)

```
// Set a custom user agent
$scraper = WebScraper::http()
    ->from('https://example.com')
    ->withUserAgent('Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36')
    ->get();
```

#### Response Caching

[](#response-caching)

```
// Cache response for 1 hour
$scraper = WebScraper::http()
    ->from('https://example.com')
    ->cacheFor(3600)  // seconds
    ->get();
```

#### Error Handling

[](#error-handling)

```
try {
    $scraper = WebScraper::http()
        ->from('https://example.com')
        ->get();

    $title = $scraper->getSelector('title')->first();
} catch (\Exception $e) {
    // Handle scraping errors
    Log::error('Scraping failed', ['error' => $e->getMessage()]);
}
```

Testing
-------

[](#testing)

When testing scraping logic, use Laravel's HTTP client mocking to avoid making actual network requests:

```
use Illuminate\Support\Facades\Http;
use Jez500\WebScraperForLaravel\Facades\WebScraper;

public function test_scraping_basic_page()
{
    // Mock the HTTP response
    Http::fake([
        'example.com/*' => Http::response('Test Page', 200),
    ]);

    $scraper = WebScraper::http()->from('https://example.com')->get();
    $title = $scraper->getSelector('title')->first();

    $this->assertEquals('Test Page', $title);
}
```

Best Practices
--------------

[](#best-practices)

- **Always cache requests** when data doesn't change frequently to reduce server load
- **Use appropriate delays** between requests to respect rate limits and avoid being blocked
- **Check robots.txt** before scraping to ensure you're allowed to access the content
- **Handle errors gracefully** and implement retry logic for transient failures
- **Use typed schemas** for complex data extraction to ensure consistency and type safety
- **Test scraping logic** with mocked responses to ensure reliability
- **Respect website terms of service** and implement proper rate limiting
- **Consider using API endpoints** when available instead of scraping HTML
- **Validate extracted data** to ensure it meets your expected format
- **Log scraping activities** for debugging and monitoring purposes

Contributing
------------

[](#contributing)

Contributions are welcome! Please follow these guidelines:

1. Fork the repository
2. Create a feature branch (`git checkout -b feature/amazing-feature`)
3. Make your changes
4. Run coding standards and tests locally: ```
    composer analyse
    composer test
    ```
5. Commit your changes with clear, descriptive messages
6. Push to your branch (`git push origin feature/amazing-feature`)
7. Open a Pull Request

When submitting PRs, please ensure:

- Code follows PSR standards
- Tests are included for new features
- Documentation is updated as needed
- All existing tests pass

Support
-------

[](#support)

For bug reports, feature requests, or questions, please open an issue on GitHub.

License
-------

[](#license)

The MIT License (MIT). Please see [LICENSE.md](LICENSE.md) for more information.

Author
------

[](#author)

[Jeremy Graham](https://github.com/jez500)

###  Health Score

46

—

FairBetter than 92% of packages

Maintenance87

Actively maintained with recent releases

Popularity26

Limited adoption so far

Community16

Small or concentrated contributor base

Maturity46

Maturing project, gaining track record

 Bus Factor1

Top contributor holds 82.1% of commits — single point of failure

How is this calculated?**Maintenance (25%)** — Last commit recency, latest release date, and issue-to-star ratio. Uses a 2-year decay window.

**Popularity (30%)** — Total and monthly downloads, GitHub stars, and forks. Logarithmic scaling prevents top-heavy scores.

**Community (15%)** — Contributors, dependents, forks, watchers, and maintainers. Measures real ecosystem engagement.

**Maturity (30%)** — Project age, version count, PHP version support, and release stability.

###  Release Activity

Cadence

Every ~38 days

Recently: every ~16 days

Total

13

Last Release

63d ago

### Community

Maintainers

![](https://www.gravatar.com/avatar/e3c0bb30ee888a2896903c8da900523f7163c2f64191961ec1c3476fa48d3c89?d=identicon)[jez500](/maintainers/jez500)

---

Top Contributors

[![jez500](https://avatars.githubusercontent.com/u/782823?v=4)](https://github.com/jez500 "jez500 (23 commits)")[![agent-neo500](https://avatars.githubusercontent.com/u/272648073?v=4)](https://github.com/agent-neo500 "agent-neo500 (2 commits)")[![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?v=4)](https://github.com/coderabbitai[bot] "coderabbitai[bot] (1 commits)")[![iliakhubuluri](https://avatars.githubusercontent.com/u/13035391?v=4)](https://github.com/iliakhubuluri "iliakhubuluri (1 commits)")[![maksm](https://avatars.githubusercontent.com/u/2431416?v=4)](https://github.com/maksm "maksm (1 commits)")

###  Code Quality

Static AnalysisPHPStan

Code StyleLaravel Pint

### Embed Badge

![Health badge](/badges/jez500-web-scraper-for-laravel/health.svg)

```
[![Health](https://phpackages.com/badges/jez500-web-scraper-for-laravel/health.svg)](https://phpackages.com/packages/jez500-web-scraper-for-laravel)
```

###  Alternatives

[craftcms/cms

Craft CMS

3.6k3.6M3.1k](/packages/craftcms-cms)[spatie/laravel-export

Create a static site bundle from a Laravel app

674146.0k6](/packages/spatie-laravel-export)[blackfire/player

A powerful web crawler and web scraper with Blackfire support

49617.1k](/packages/blackfire-player)[spatie/laravel-pjax

A pjax middleware for Laravel 5

523386.8k11](/packages/spatie-laravel-pjax)[crwlr/crawler

Web crawling and scraping library.

36917.4k2](/packages/crwlr-crawler)[helgesverre/extractor

AI-Powered Data Extraction for your Laravel application.

22340.0k](/packages/helgesverre-extractor)

PHPackages © 2026

[Directory](/)[Categories](/categories)[Trending](/trending)[Changelog](/changelog)[Analyze](/analyze)