ScrapeHub is a curated collection of web scraping tools, crawlers, parsers and browser automation libraries, ranked by GitHub Stars. Designed for both humans and AI agents — parse the tables below to match any scraping task to the right tool.
Star counts verified at time of writing. Links point to GitHub unless noted.
All repos in one table sorted by GitHub star count. Use the Purpose column to filter by task need.
| # | Repo | ★ | Purpose | Lang | Description |
|---|---|---|---|---|---|
| 1 | yt-dlp/yt-dlp | ★180k+ | Media Downloader | Python | Active youtube-dl fork — more sites, faster updates |
| 2 | axios/axios | ★109k+ | HTTP Client | JavaScript | Promise-based HTTP client. Standard building block |
| 3 | puppeteer/puppeteer | ★95k+ | Browser Automation | JavaScript | Chrome DevTools Protocol — most popular browser automation library |
| 4 | microsoft/playwright | ★93k+ | Browser Automation | Multi-language | Cross-browser automation (Chromium, Firefox, WebKit) with auto-waiting |
| 5 | tesseract-ocr/tesseract | ★75k+ | OCR / Captcha | C++ | OCR engine for text-based captchas and image text extraction |
| 6 | scrapy/scrapy | ★63k+ | Crawler Framework | Python | Async framework for large-scale crawling with middleware & pipelines |
| 7 | soimort/you-get | ★57k+ | Media Downloader | Python | Media downloader for YouTube, Bilibili, and many other sites |
| 8 | cypress-io/cypress | ★51k+ | Browser Automation | JavaScript | E2E testing framework — useful for scraping SPAs |
| 9 | SeleniumHQ/selenium | ★34k+ | Browser Automation | Multi-language | Browser automation with W3C WebDriver standard |
| 10 | cheeriojs/cheerio | ★30k+ | HTML Parser | JavaScript | jQuery-like API for server-side HTML parsing. Fast, no browser |
| 11 | FingerprintJS/fingerprintjs | ★28k+ | Anti-Detect | JavaScript | Browser fingerprinting — understand what sites detect |
| 12 | nsqio/nsq | ★26k+ | Message Queue | Go | Real-time distributed messaging — used in scraping pipelines |
| 13 | apify/crawlee | ★25k+ | Crawler Framework | JavaScript | Full-featured scraping framework with queue, proxy & storage |
| 14 | gocolly/colly | ★25k+ | Crawler Framework | Go | Lightning-fast scraping framework with distributed support |
| 15 | jhao104/proxy_pool | ★23k+ | Proxy Tools | Python | Self-hosted proxy pool with automatic validation |
| 16 | chromium/chromium | ★18k+ | Headless Browser | C++ | Open-source browser powering most headless scraping |
| 17 | encode/httpx | ★15k+ | HTTP Client | Python | Modern async HTTP client for scraping pipelines |
| 18 | psf/requests-html | ★14k+ | HTML Parser | Python | HTML parsing with JS support (Chromium-backed). Simple API |
| 19 | PuerkitoBio/goquery | ★14k+ | HTML Parser | Go | jQuery-like DOM traversal for Go |
| 20 | sindresorhus/got | ★14k+ | HTTP Client | JavaScript | Human-friendly HTTP client with hooks, streams, caching |
| 21 | microsoft/playwright-python | ★12k+ | Browser Automation | Python | Official Python bindings for Playwright |
| 22 | Gallopsled/pwntools | ★12k+ | HTTP Utilities | Python | CTF framework — pwnlib.web has HTTP utilities for scraping |
| 23 | jsoup/jsoup | ★11k+ | HTML Parser | Java | Java HTML parser with jQuery-like API. Industry standard |
| 24 | chromedp/chromedp | ★11k+ | Browser Automation | Go | High-level Chrome DevTools Protocol client for Go |
| 25 | seanmonstar/reqwest | ★12k+ | HTTP Client | Rust | Ergonomic HTTP client for Rust |
| 26 | mozilla/readability | ★11k+ | Content Extraction | JavaScript | Firefox reader mode as a library — article content extraction |
| 27 | FriendsOfPHP/Goutte | ★9.2k+ | Crawler Framework | PHP | PHP web scraper with Symfony components |
| 28 | gotwarlost/istanbul | ★8.7k+ | Code Coverage | JavaScript | JS code coverage — useful in scraping QA pipelines |
| 29 | alirezamika/autoscraper | ★7.7k+ | Smart Scraper | Python | Learns scraping patterns from examples. No selectors needed |
| 30 | sparklemotion/nokogiri | ★6k+ | HTML Parser | Ruby | Ruby HTML/XML parser with CSS3 selectors |
| 31 | pdfminer/pdfminer.six | ★6k+ | PDF Extraction | Python | Python PDF text extraction |
| 32 | pdfplumber/pdfplumber | ★6k+ | PDF Extraction | Python | PDF parsing with table extraction and visual debugging |
| 33 | madmaze/pytesseract | ★6k+ | OCR / Captcha | Python | Python wrapper for Tesseract OCR |
| 34 | MechanicalSoup/MechanicalSoup | ★5k+ | Browser Automation | Python | Stateful browser simulator for form-heavy sites |
| 35 | yasserg/crawler4j | ★4.6k+ | Crawler Framework | Java | Multi-threaded web crawler framework in Java |
| 36 | sparklemotion/mechanize | ★4.4k+ | Browser Automation | Ruby | Stateful web scraping with form submission and cookies |
| 37 | scrapinghub/splash | ★4.2k+ | Headless Browser | Python | Lightweight scriptable headless browser with HTTP API |
| 38 | symfony/dom-crawler | ★4k+ | HTML Parser | PHP | DOM traversal and form handling for PHP |
| 39 | PHP-FFMpeg/PHP-FFMpeg | ★5k+ | Media Downloader | PHP | Media manipulation for video scraping workflows |
| 40 | pymupdf/PyMuPDF | ★5k+ | PDF Extraction | Python | Fast PDF text/image extraction |
| 41 | zzzprojects/html-agility-pack | ★2.8k+ | HTML Parser | .NET | HTML parser with XPath and LINQ for .NET |
| 42 | apache/nutch | ★3.3k+ | Crawler Framework | Java | Extensible scalable web crawler on Hadoop |
| 43 | rust-headless-chrome/rust-headless-chrome | ★2.9k+ | Browser Automation | Rust | Headless Chrome control via DevTools Protocol in Rust |
| 44 | lxml/lxml | ★3k+ | HTML Parser | Python | Fastest XML/HTML parser. libxml2 bindings |
| 45 | scrapy/parsel | ★3k+ | HTML Parser | Python | Standalone CSS/XPath selector from Scrapy core |
| 46 | microsoft/playwright-dotnet | ★3k+ | Browser Automation | .NET | Official .NET bindings for Playwright |
| 47 | watir/watir | ★3k+ | Browser Automation | Ruby | Ruby browser automation driver |
| 48 | internetarchive/heritrix3 | ★3.3k+ | Crawler Framework | Java | Internet Archive's web crawler for large-scale archiving |
| 49 | lorien/grab | ★2.5k+ | Crawler Framework | Python | Async framework with built-in proxy rotation and JS rendering |
| 50 | paquettg/php-html-parser | ★2.4k+ | HTML Parser | PHP | HTML DOM parser for PHP |
| 51 | sjdirect/abot | ★2.3k+ | Crawler Framework | .NET | C# web crawler with politeness and parallelization |
| 52 | rust-scraper/scraper | ★2.4k+ | HTML Parser | Rust | HTML parsing and querying with CSS selectors in Rust |
| 53 | tabula-java/tabula | ★2k+ | PDF Extraction | Java | Extract tables from PDFs |
| 54 | requireCool/stealth.min.js | ★1.1k+ | Anti-Detect | JavaScript | Puppeteer stealth plugin to avoid bot detection |
| 55 | apache/httpcomponents-client | ★1.5k+ | HTTP Client | Java | Low-level HTTP client for Java scraping pipelines |
| 56 | mozilla/geckoview | ★542 | Headless Browser | Java | Firefox engine for browser automation |
| 57 | s0md3v/Zen | ★595 | Anti-Detect | Python | Anti-bot detection evasion library |
| 58 | andybalholm/cascadia | ★754 | HTML Parser | Go | CSS selector library for Go (used by goquery) |
| 59 | antchfx/htmlquery | ★784 | HTML Parser | Go | XPath query engine for HTML documents |
| 60 | scrapy-plugins/scrapy-proxy-middleware | ★800 | Proxy Tools | Python | Rotating proxy middleware for Scrapy |
| 61 | apache/stormcrawler | ★988 | Crawler Framework | Java | Distributed web crawler on Apache Storm |
| 62 | antchfx/xmlquery | ★490 | HTML Parser | Go | XPath query engine for XML documents |
| 63 | html5lib/html5lib-python | ★1k+ | HTML Parser | Python | Spec-compliant HTML parser (browser-grade) |
| 64 | rushter/selectolax | ★1k+ | HTML Parser | Python | Binding for lexbor — very fast HTML parsing |
| 65 | flavorjones/chromedriver-helper | ★312 | Browser Automation | Ruby | Chromedriver management for Ruby |
| 66 | seleniumhq/selenium-dotnet | ★300 | Browser Automation | .NET | .NET bindings for Selenium WebDriver |
| 67 | 2captcha/2captcha-python | ★784 | OCR / Captcha | Python | API wrapper for 2Captcha service |
| 68 | facelessuser/soupsieve | ★267 | HTML Parser | Python | CSS selector engine used by BeautifulSoup4 |
| # | Repo | ★ | Language | Description |
|---|---|---|---|---|
| 1 | puppeteer/puppeteer | ★95k+ | JavaScript | Chrome DevTools Protocol — most popular browser automation library |
| 2 | microsoft/playwright | ★93k+ | Multi-language | Cross-browser (Chromium, Firefox, WebKit) with auto-waiting |
| 3 | cypress-io/cypress | ★51k+ | JavaScript | E2E testing framework — useful for scraping SPAs |
| 4 | SeleniumHQ/selenium | ★34k+ | Multi-language | Browser automation with W3C WebDriver standard |
| 5 | microsoft/playwright-python | ★12k+ | Python | Official Python bindings for Playwright |
| 6 | chromedp/chromedp | ★11k+ | Go | High-level Chrome DevTools Protocol client for Go |
| 7 | MechanicalSoup/MechanicalSoup | ★5k+ | Python | Stateful browser simulator for form-heavy sites |
| 8 | sparklemotion/mechanize | ★4.4k+ | Ruby | Stateful web scraping with form submission and cookies |
| 9 | rust-headless-chrome/rust-headless-chrome | ★2.9k+ | Rust | Headless Chrome control via DevTools Protocol in Rust |
| 10 | microsoft/playwright-dotnet | ★3k+ | .NET | Official .NET bindings for Playwright |
| 11 | watir/watir | ★3k+ | Ruby | Ruby browser automation driver |
| 12 | flavorjones/chromedriver-helper | ★312 | Ruby | Chromedriver management for Ruby |
| 13 | seleniumhq/selenium-dotnet | ★300 | .NET | .NET bindings for Selenium WebDriver |
| # | Repo | ★ | Language | Description |
|---|---|---|---|---|
| 1 | cheeriojs/cheerio | ★30k+ | JavaScript | jQuery-like API for server-side HTML parsing. Fast, no browser |
| 2 | psf/requests-html | ★14k+ | Python | HTML parsing with JS support (Chromium-backed). Simple API |
| 3 | PuerkitoBio/goquery | ★14k+ | Go | jQuery-like DOM traversal for Go |
| 4 | jsoup/jsoup | ★11k+ | Java | Java HTML parser with jQuery-like API. Industry standard |
| 5 | sparklemotion/nokogiri | ★6k+ | Ruby | Ruby HTML/XML parser with CSS3 selectors |
| 6 | symfony/dom-crawler | ★4k+ | PHP | DOM traversal and form handling for PHP |
| 7 | zzzprojects/html-agility-pack | ★2.8k+ | .NET | HTML parser with XPath and LINQ for .NET |
| 8 | lxml/lxml | ★3k+ | Python | Fastest XML/HTML parser. libxml2 bindings |
| 9 | scrapy/parsel | ★3k+ | Python | Standalone CSS/XPath selector from Scrapy core |
| 10 | paquettg/php-html-parser | ★2.4k+ | PHP | HTML DOM parser for PHP |
| 11 | rust-scraper/scraper | ★2.4k+ | Rust | HTML parsing and querying with CSS selectors in Rust |
| 12 | andybalholm/cascadia | ★754 | Go | CSS selector library for Go (used by goquery) |
| 13 | antchfx/htmlquery | ★784 | Go | XPath query engine for HTML documents |
| 14 | antchfx/xmlquery | ★490 | Go | XPath query engine for XML documents |
| 15 | html5lib/html5lib-python | ★1k+ | Python | Spec-compliant HTML parser (browser-grade) |
| 16 | rushter/selectolax | ★1k+ | Python | Binding for lexbor — very fast HTML parsing |
| 17 | facelessuser/soupsieve | ★267 | Python | CSS selector engine used by BeautifulSoup4 |
| # | Repo | ★ | Language | Description |
|---|---|---|---|---|
| 1 | scrapy/scrapy | ★63k+ | Python | Async framework for large-scale crawling with middleware & pipelines |
| 2 | apify/crawlee | ★25k+ | JavaScript | Full-featured scraping framework with queue, proxy & storage |
| 3 | gocolly/colly | ★25k+ | Go | Lightning-fast scraping framework with distributed support |
| 4 | FriendsOfPHP/Goutte | ★9.2k+ | PHP | PHP web scraper with Symfony components |
| 5 | yasserg/crawler4j | ★4.6k+ | Java | Multi-threaded web crawler framework in Java |
| 6 | apache/nutch | ★3.3k+ | Java | Extensible scalable web crawler on Hadoop |
| 7 | internetarchive/heritrix3 | ★3.3k+ | Java | Internet Archive's web crawler for large-scale archiving |
| 8 | lorien/grab | ★2.5k+ | Python | Async framework with built-in proxy rotation and JS rendering |
| 9 | sjdirect/abot | ★2.3k+ | .NET | C# web crawler with politeness and parallelization |
| 10 | apache/stormcrawler | ★988 | Java | Distributed web crawler on Apache Storm |
| # | Repo | ★ | Language | Description |
|---|---|---|---|---|
| 1 | axios/axios | ★109k+ | JavaScript | Promise-based HTTP client. Standard building block |
| 2 | encode/httpx | ★15k+ | Python | Modern async HTTP client for scraping pipelines |
| 3 | sindresorhus/got | ★14k+ | JavaScript | Human-friendly HTTP client with hooks, streams, caching |
| 4 | seanmonstar/reqwest | ★12k+ | Rust | Ergonomic HTTP client for Rust |
| 5 | apache/httpcomponents-client | ★1.5k+ | Java | Low-level HTTP client for Java scraping pipelines |
| # | Repo | ★ | Language | Description |
|---|---|---|---|---|
| 1 | yt-dlp/yt-dlp | ★180k+ | Python | Active youtube-dl fork — more sites, faster updates |
| 2 | soimort/you-get | ★57k+ | Python | Media downloader for YouTube, Bilibili, and many other sites |
| 3 | PHP-FFMpeg/PHP-FFMpeg | ★5k+ | PHP | Media manipulation for video scraping workflows |
| # | Repo | ★ | Language | Description |
|---|---|---|---|---|
| 1 | pdfminer/pdfminer.six | ★6k+ | Python | Python PDF text extraction |
| 2 | pdfplumber/pdfplumber | ★6k+ | Python | PDF parsing with table extraction and visual debugging |
| 3 | pymupdf/PyMuPDF | ★5k+ | Python | Fast PDF text/image extraction |
| 4 | tabula-java/tabula | ★2k+ | Java | Extract tables from PDFs |
| # | Repo | ★ | Language | Description |
|---|---|---|---|---|
| 1 | tesseract-ocr/tesseract | ★75k+ | C++ | OCR engine for text-based captchas and image text extraction |
| 2 | madmaze/pytesseract | ★6k+ | Python | Python wrapper for Tesseract OCR |
| 3 | 2captcha/2captcha-python | ★784 | Python | API wrapper for 2Captcha service |
| # | Repo | ★ | Language | Description |
|---|---|---|---|---|
| 1 | FingerprintJS/fingerprintjs | ★28k+ | JavaScript | Browser fingerprinting — understand what sites detect |
| 2 | requireCool/stealth.min.js | ★1.1k+ | JavaScript | Puppeteer stealth plugin to avoid bot detection |
| 3 | s0md3v/Zen | ★595 | Python | Anti-bot detection evasion library |
| # | Repo | ★ | Language | Description |
|---|---|---|---|---|
| 1 | jhao104/proxy_pool | ★23k+ | Python | Self-hosted proxy pool with automatic validation |
| 2 | scrapy-plugins/scrapy-proxy-middleware | ★800 | Python | Rotating proxy middleware for Scrapy |
| # | Repo | ★ | Language | Description |
|---|---|---|---|---|
| 1 | chromium/chromium | ★18k+ | C++ | Open-source browser powering most headless scraping |
| 2 | scrapinghub/splash | ★4.2k+ | Python | Lightweight scriptable headless browser with HTTP API |
| 3 | mozilla/geckoview | ★542 | Java | Firefox engine for browser automation |
| # | Repo | ★ | Language | Description |
|---|---|---|---|---|
| 1 | alirezamika/autoscraper | ★7.7k+ | Python | Learns scraping patterns from examples. No selectors needed |
| # | Repo | ★ | Language | Description |
|---|---|---|---|---|
| 1 | mozilla/readability | ★11k+ | JavaScript | Firefox reader mode as a library — article content extraction |
| # | Repo | ★ | Language | Description |
|---|---|---|---|---|
| 1 | nsqio/nsq | ★26k+ | Go | Real-time distributed messaging — used in scraping pipelines |
| Service | Best For | Description |
|---|---|---|
| ScrapingBee | Proxy + JS rendering | API for headless browser scraping with proxy rotation |
| ScraperAPI | CAPTCHA handling | Proxy + CAPTCHA solving as a service |
| Apify | Ready-made scrapers | Platform with thousands of pre-built scrapers |
| ScrapingFish | Proxy rotation | Proxy-rotating scraping API |
| Octoparse | No-code scraping | Visual scraper with point-and-click interface |
| ParseHub | No-code scraping | Desktop app for visual scraping |
Low-level parser libraries that power the higher-level tools above.
| Library | Bindings | Best For |
|---|---|---|
| lexbor | C, Python (selectolax) | Fastest HTML parsing |
| gumbo-parser | C, Python (html5lib) | HTML5 spec-compliant parsing from Google |
| tidy-html5 | C, many languages | HTML cleanup and repair |
| Resource | Type | Best For |
|---|---|---|
| Web Scraping with Python (O'Reilly) | Book | Comprehensive scraping fundamentals |
| Scrapy Docs | Documentation | Learning Scrapy step by step |
| Playwright Docs | Documentation | Browser automation patterns |
| Awesome Web Scraping | List | Community-curated resources |
| HTTP Bin | Tool | HTTP request debugging |
PRs welcome! Open an issue or submit a pull request.
Guidelines:
- Keep descriptions short and factual
- Prefer actively maintained repos (updated within 2 years)
- Include verified star counts and language tags
- If a star count is outdated, submit a correction
MIT — free to use, share, and remix.