PHPackages                             g1k/rolling-curl - PHPackages - PHPackages  [Skip to content](#main-content)[PHPackages](/)[Directory](/)[Categories](/categories)[Trending](/trending)[Leaderboard](/leaderboard)[Changelog](/changelog)[Analyze](/analyze)[Collections](/collections)[Log in](/login)[Sign up](/register)

1. [Directory](/)
2. /
3. [HTTP & Networking](/categories/http)
4. /
5. g1k/rolling-curl

AbandonedArchivedLibrary[HTTP & Networking](/categories/http)

g1k/rolling-curl
================

Rolling-Curl: A non-blocking, non-dos multi-curl library for PHP

3.1.2(8y ago)07Apache-2.0PHPPHP >=5.3.0

Since May 30Pushed 8y ago1 watchersCompare

[ Source](https://github.com/G1K/rolling-curl)[ Packagist](https://packagist.org/packages/g1k/rolling-curl)[ Docs](https://github.com/chuyskywalker/rolling-curl)[ RSS](/packages/g1k-rolling-curl/feed)WikiDiscussions master Synced 1mo ago

READMEChangelogDependencies (1)Versions (7)Used By (0)

RollingCurl
===========

[](#rollingcurl)

A cURL library to fetch a large number of resources while maintaining a consistent number of simultaneous connections

Authors:

- Jeff Minard (jrm.cc)
- Josh Fraser (joshfraser.com)
- Alexander Makarov (rmcreative.ru)

Overview
--------

[](#overview)

RollingCurl is a more efficient implementation of curl\_multi().

curl\_multi is a great way to process multiple HTTP requests in parallel in PHP but suffers from a few faults:

1. The documentation for curl\_multi is very obtuse and, as such, is easy to incorrectly or poorly implement
2. Most curl\_multi examples queue up all requests and execute them all at once

The second point is the most important one for two reasons:

1. If you have to wait on every single request to complete, your program is "blocked" by the longest running request.
2. More importantly, when you run a large number of cURL requests simultaneously you are, essentially, running a DOS attack. If you have to fetch hundreds or even thousands of URLs you're very likely to be blocked by automatic DOS systems. At best, you're not being a very respectful citizen of the internet.

RollingCurl deals with both issues by maintaining a maximum number of simultaneous requests and "rolling" new requests into the queue as existing requests complete. When requests complete, and while other requests are still running, RollingCurl can run an anonymous function to process the fetched result. (You have the option to skip the function and instead process all requests once they are done, should you prefer.)

Installation (via composer)
---------------------------

[](#installation-via-composer)

[Get composer](http://getcomposer.org/doc/00-intro.md) and add this in your requires section of the composer.json:

```
{
    "require": {
        "chuyskywalker/rolling-curl": "*"
    }
}

```

and then

```
composer install

```

Usage
-----

[](#usage)

### Basic Example

[](#basic-example)

```
$rollingCurl = new \RollingCurl\RollingCurl();
$rollingCurl
    ->get('http://yahoo.com')
    ->get('http://google.com')
    ->get('http://hotmail.com')
    ->get('http://msn.com')
    ->get('http://reddit.com')
    ->setCallback(function(\RollingCurl\Request $request, \RollingCurl\RollingCurl $rollingCurl) {
        // parsing html with regex is evil (http://bit.ly/3x9sQX), but this is just a demo
        if (preg_match("#(.*)#i", $request->getResponseText(), $out)) {
            $title = $out[1];
        }
        else {
            $title = '[No Title Tag Found]';
        }
        echo "Fetch complete for (" . $request->getUrl() . ") $title " . PHP_EOL;
    })
    ->setSimultaneousLimit(3)
    ->execute();
```

### Fetch A Very Large Number Of Pages

[](#fetch-a-very-large-number-of-pages)

Let's scrape google for the first 500 links & titles for "curl"

```
$rollingCurl = new \RollingCurl\RollingCurl();
for ($i = 0; $i get('https://www.google.com/search?q=curl&start=' . $i);
}

$results = array();

$start = microtime(true);
echo "Fetching..." . PHP_EOL;
$rollingCurl
    ->setCallback(function(\RollingCurl\Request $request, \RollingCurl\RollingCurl $rollingCurl) use (&$results) {
        if (preg_match_all('#