StarRankExplorer
Back to rankings

shengqiangzhang/examples-of-web-crawlers

HTML

一些非常有趣的python爬虫例子,对新手比较友好,主要爬取淘宝、天猫、微信、微信读书、豆瓣、QQ等网站。(Some interesting examples of python crawlers that are friendly to beginners. )

crawlerspidertaobaotmallexamplepythonseleniumpyquerystockfundmultithreadingagent-pool
Star Growth
Stars
14.7k
Forks
3.8k
Weekly Growth
—
Issues
19
5k10k
Mar 2019Aug 2021Feb 2024Jul 2026
Related repositories
firecrawl/firecrawl

The API to search, scrape, and interact with the web at scale. 🔥

TypeScriptnpmGNU Affero General Public License v3.0aicrawler
firecrawl.dev
153.6k8.8k
D4Vinci/Scrapling

🕷️ An adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl!

PythonPyPIBSD 3-Clause "New" or "Revised" Licensecrawlercrawling
scrapling.readthedocs.io/en/latest/
70.4k7k
scrapy/scrapy

Scrapy, a fast high-level web crawling & scraping framework for Python.

PythonPyPIBSD 3-Clause "New" or "Revised" Licensepythonscraping
scrapy.org
63.3k11.8k
NaiboWang/EasySpider

A visual no-code/code-free web crawler/spider易采集：一个可视化浏览器自动化测试/数据采集/网页爬虫软件，可以无代码图形化的设计和执行爬虫任务。别名：ServiceWrapper面向Web应用的智能化服务封装系统。

JavaScriptnpmGNU Affero General Public License v3.0code-freecrawler
easyspider.net
44.3k5.4k
iawia002/lux

👾 Fast and simple video download library and CLI tool written in Go

GoGo ModulesMIT Licensedownloadergo
31.5k3.3k
mendableai/firecrawl

🔥 Turn entire websites into LLM-ready markdown or structured data. Scrape, crawl and extract with a single API.

TypeScriptnpmGNU Affero General Public License v3.0aicrawler
firecrawl.dev
29.5k2.5k
ScrapeGraphAI/Scrapegraph-ai

Python scraper based on AI

PythonPyPIMIT Licensescrapingscraping-python
scrapegraphai.com
28.5k2.8k
gocolly/colly

Elegant Scraper and Crawler Framework for Golang

GoGo ModulesApache License 2.0golangscraper
go-colly.org
25.4k1.9k
apify/crawlee

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

TypeScriptnpmApache License 2.0web-scrapingweb-crawling
crawlee.dev
24.8k1.6k
jhao104/proxy_pool

Python ProxyPool for web spider

PythonPyPIMIT Licensecrawlerproxy
jhao104.github.io/proxy_pool/
23.5k5.4k
Evil0ctal/Douyin_TikTok_Download_API

🚀「Douyin_TikTok_Download_API」是一个开箱即用的高性能异步抖音、快手、TikTok、Bilibili数据爬取工具，支持API调用，在线批量解析及下载。

PythonPyPIApache License 2.0pythonpywebio
douyin.wtf
18.9k2.7k
projectdiscovery/katana

A next-generation crawling and spidering framework.

GoGo ModulesMIT Licensecrawlerweb-spider
17.2k1.2k