PHPackages monperrus/crawler-user-agents - PHPackages - PHPackages [Skip to content](#main-content)[PHPackages](/)[Directory](/)[Categories](/categories)[Trending](/trending)[Leaderboard](/leaderboard)[Changelog](/changelog)[Analyze](/analyze)[Collections](/collections)[Log in](/login)[Sign up](/register) 1. [Directory](/) 2. / 3. [Parsing & Serialization](/categories/parsing) 4. / 5. monperrus/crawler-user-agents ActiveLibrary[Parsing & Serialization](/categories/parsing) monperrus/crawler-user-agents ============================= This repository contains a list of of HTTP user-agents used by robots, crawlers, and spiders as in single JSON file. v1.55.0(yesterday)1.4k6.3k286[10 issues](https://github.com/monperrus/crawler-user-agents/issues)[1 PRs](https://github.com/monperrus/crawler-user-agents/pulls)MITGoCI passing Since Apr 22Pushed yesterday38 watchersCompare [ Source](https://github.com/monperrus/crawler-user-agents)[ Packagist](https://packagist.org/packages/monperrus/crawler-user-agents)[ GitHub Sponsors](https://github.com/monperrus)[ RSS](/packages/monperrus-crawler-user-agents/feed)WikiDiscussions master Synced yesterday READMEChangelogDependenciesVersions (58)Used By (0) crawler-user-agents =================== [](#crawler-user-agents) This repository contains a list of of HTTP user-agents used by robots, crawlers, and spiders as in single JSON file. - NPM package: - Go package: - PyPi package: Each `pattern` is a regular expression. It should work out-of-the-box wih your favorite regex library. Sponsor ------- [](#sponsor) 💼 **Using crawler-user-agents in a commercial product?** This package is free to use, but it takes real time to maintain and expand. If it's providing value (and it probably is), please consider [sponsoring at the commercial tier](https://github.com/sponsors/monperrus?frequency=recurring). It keeps the project alive and actively maintained. Your company can afford it. 🙏 Install ------- [](#install) ### Direct download [](#direct-download) Download the [`crawler-user-agents.json` file](https://raw.githubusercontent.com/monperrus/crawler-user-agents/master/crawler-user-agents.json) from this repository directly. ### Javascript [](#javascript) crawler-user-agents is deployed on npmjs.com: To use it using npm or yarn: ``` npm install --save crawler-user-agents # OR yarn add crawler-user-agents ``` In Node.js, you can `require` the package to get an array of crawler user agents. ``` const crawlers = require('crawler-user-agents'); console.log(crawlers); ``` ### Python [](#python) Install with `pip install crawler-user-agents` Then: ``` import crawleruseragents if crawleruseragents.is_crawler("Googlebot/"): # do something ``` or: ``` import crawleruseragents indices = crawleruseragents.matching_crawlers("bingbot/2.0") print("crawlers' indices:", indices) print( "crawler's URL:", crawleruseragents.CRAWLER_USER_AGENTS_DATA[indices[0]]["url"] ) ``` Note that `matching_crawlers` is much slower than `is_crawler`, if the given User-Agent does indeed match any crawlers. ### Go [](#go) Go: use [this package](https://pkg.go.dev/github.com/monperrus/crawler-user-agents), it provides global variable `Crawlers` (it is synchronized with `crawler-user-agents.json`), functions `IsCrawler` and `MatchingCrawlers`. Example of Go program: ``` package main import ( "fmt" "github.com/monperrus/crawler-user-agents" ) func main() { userAgent := "Mozilla/5.0 (compatible; Discordbot/2.0; +https://discordapp.com)" isCrawler := agents.IsCrawler(userAgent) fmt.Println("isCrawler:", isCrawler) indices := agents.MatchingCrawlers(userAgent) fmt.Println("crawlers' indices:", indices) fmt.Println("crawler's URL:", agents.Crawlers[indices[0]].URL) } ``` Output: ``` isCrawler: true crawlers' indices: [237] crawler' URL: https://discordapp.com ``` Contributing ------------ [](#contributing) I do welcome additions contributed as pull requests. The pull requests should: - contain a single addition - specify a discriminant relevant syntactic fragment (for example "totobot" and not "Mozilla/5 totobot v20131212.alpha1") - contain the pattern (generic regular expression), the discovery date (year/month/day) and the official url of the robot - result in a valid JSON file (don't forget the comma between items) Example: ``` { "pattern": "rogerbot", "addition_date": "2014/02/28", "url": "http://moz.com/help/pro/what-is-rogerbot-", "instances" : ["rogerbot/2.3 example UA"], "tags": ["seo"] } ``` License ------- [](#license) The list is under a [MIT License](https://opensource.org/licenses/MIT). The versions prior to Nov 7, 2016 were under a [CC-SA](http://creativecommons.org/licenses/by-sa/3.0/) license. Related work ------------ [](#related-work) There are a few wrapper libraries that use this data to detect bots: - [Voight-Kampff](https://github.com/biola/Voight-Kampff) (Ruby) - [isbot](https://github.com/Hentioe/isbot) (Ruby) - [crawlers](https://github.com/Olical/crawlers) (Clojure) - [isBot](https://github.com/omrilotan/isbot) (Node.JS) Other systems for spotting robots, crawlers, and spiders that you may want to consider are: - [Crawler-Detect](https://github.com/JayBizzle/Crawler-Detect) (PHP) - [BrowserDetector](https://github.com/mimmi20/BrowserDetector) (PHP) - [browscap](https://github.com/browscap/browscap) (JSON files) ### Health Score 61 — FairBetter than 98% of packages Maintenance100 Actively maintained with recent releases Popularity50 Moderate usage in the ecosystem Community35 Small or concentrated contributor base Maturity51 Maturing project, gaining track record Bus Factor2 2 contributors hold 50%+ of commits How is this calculated?**Maintenance (25%)** — Last commit recency, latest release date, and issue-to-star ratio. Uses a 2-year decay window. **Popularity (30%)** — Total and monthly downloads, GitHub stars, and forks. Logarithmic scaling prevents top-heavy scores. **Community (15%)** — Contributors, dependents, forks, watchers, and maintainers. Measures real ecosystem engagement. **Maturity (30%)** — Project age, version count, PHP version support, and release stability. ### Release Activity Cadence Every ~10 days Recently: every ~3 days Total 44 Last Release 1d ago ### Community Maintainers ![](https://www.gravatar.com/avatar/7cff17040f814cfe1826b0fc67326734ffdb361f5f166cf17411e2e9bebeb67a?d=identicon)[monperrus](/maintainers/monperrus) --- Top Contributors [![monperrus](https://avatars.githubusercontent.com/u/803666?v=4)](https://github.com/monperrus "monperrus (155 commits)")[![Fale](https://avatars.githubusercontent.com/u/77888?v=4)](https://github.com/Fale "Fale (75 commits)")[![dependabot[bot]](https://avatars.githubusercontent.com/in/29110?v=4)](https://github.com/dependabot[bot] "dependabot[bot] (14 commits)")[![mirabellette](https://avatars.githubusercontent.com/u/34450971?v=4)](https://github.com/mirabellette "mirabellette (12 commits)")[![AlexWayfer](https://avatars.githubusercontent.com/u/6510020?v=4)](https://github.com/AlexWayfer "AlexWayfer (11 commits)")[![dan-blanchard](https://avatars.githubusercontent.com/u/976345?v=4)](https://github.com/dan-blanchard "dan-blanchard (9 commits)")[![dspinellis](https://avatars.githubusercontent.com/u/1212716?v=4)](https://github.com/dspinellis "dspinellis (8 commits)")[![tn3w](https://avatars.githubusercontent.com/u/141021574?v=4)](https://github.com/tn3w "tn3w (6 commits)")[![Boegie](https://avatars.githubusercontent.com/u/18569707?v=4)](https://github.com/Boegie "Boegie (6 commits)")[![fallax](https://avatars.githubusercontent.com/u/2036067?v=4)](https://github.com/fallax "fallax (5 commits)")[![synhershko](https://avatars.githubusercontent.com/u/212252?v=4)](https://github.com/synhershko "synhershko (4 commits)")[![eggpi](https://avatars.githubusercontent.com/u/489134?v=4)](https://github.com/eggpi "eggpi (4 commits)")[![erikdubbelboer](https://avatars.githubusercontent.com/u/522870?v=4)](https://github.com/erikdubbelboer "erikdubbelboer (4 commits)")[![JBYoshi](https://avatars.githubusercontent.com/u/12983479?v=4)](https://github.com/JBYoshi "JBYoshi (4 commits)")[![jpjacobpadilla](https://avatars.githubusercontent.com/u/98066312?v=4)](https://github.com/jpjacobpadilla "jpjacobpadilla (4 commits)")[![blaine-arcjet](https://avatars.githubusercontent.com/u/146491715?v=4)](https://github.com/blaine-arcjet "blaine-arcjet (4 commits)")[![samuelgiles](https://avatars.githubusercontent.com/u/2643026?v=4)](https://github.com/samuelgiles "samuelgiles (4 commits)")[![shimpeko](https://avatars.githubusercontent.com/u/137599?v=4)](https://github.com/shimpeko "shimpeko (4 commits)")[![haumacher](https://avatars.githubusercontent.com/u/5607145?v=4)](https://github.com/haumacher "haumacher (3 commits)")[![mattmarcum](https://avatars.githubusercontent.com/u/4745044?v=4)](https://github.com/mattmarcum "mattmarcum (3 commits)") --- Tags jsonlistinguser agentscrawlers ### Embed Badge ![Health badge](/badges/monperrus-crawler-user-agents/health.svg) ``` [![Health](https://phpackages.com/badges/monperrus-crawler-user-agents/health.svg)](https://phpackages.com/packages/monperrus-crawler-user-agents) ``` ### Alternatives [justinrainbow/json-schema A library to validate a json schema. 3.6k334.7M790](/packages/justinrainbow-json-schema)[mtdowling/jmespath.php Declaratively specify how to extract elements from a JSON document 2.0k504.8M167](/packages/mtdowling-jmespathphp)[jms/serializer Library for (de-)serializing data of any complexity; supports XML, and JSON. 2.3k141.9M929](/packages/jms-serializer)[jms/serializer-bundle Allows you to easily serialize, and deserialize data of any complexity 1.8k92.4M680](/packages/jms-serializer-bundle)[colinodell/json5 UTF-8 compatible JSON5 parser for PHP 30525.1M57](/packages/colinodell-json5)[clue/ndjson-react Streaming newline-delimited JSON (NDJSON) parser and encoder for ReactPHP. 15882.2M31](/packages/clue-ndjson-react) PHPackages © 2026 [Directory](/)[Categories](/categories)[Trending](/trending)[Changelog](/changelog)[Analyze](/analyze)