数据系统
数据管道
978 个结果1yt-dlp/yt-dlpA feature-rich command-line audio/video downloaderPythoncliPyPIyoutube-dlpython+348 ★/日200.1k17.4k2ytdl-org/youtube-dlCommand-line program to download videos from YouTube.com and other video sitesPythoncliPyPIdownloadervideo+9 ★/日141.4k10.7k3D4Vinci/Scrapling🕷️ A Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_devPythonlibraryPyPIcrawlercrawling+164 ★/日89.1k9.2k4doocs/advanced-java😮 Core Interview Questions & Answers For Experienced Java(Backend) Developers | 互联网 Java 工程师进阶知识完全扫盲：涵盖高并发、分布式、高可用、微服务、海量数据处理等领域知识JavaMavenjavadistributed-systems79.2k19.1k5redis/redisFor developers, who are building real-time data-driven applications, Redis is the preferred, fastest, and most feature-rich cache, data structure server, and document and vector query engine.Cdatabasekey-value77k24.9k6NanmiCoder/MediaCrawler小红书笔记 | 评论爬虫、抖音视频 | 评论爬虫、快手视频 | 评论爬虫、B 站视频 ｜ 评论爬虫、微博帖子 ｜ 评论爬虫、百度贴吧帖子 ｜ 百度贴吧评论回复爬虫 | 知乎问答文章｜评论爬虫PythonPyPIcrawlerspider+128 ★/日67.6k12.9k7scrapy/scrapyScrapy, a fast high-level web crawling & scraping framework for Python.PythonlibraryPyPIpythonscraping+20 ★/日64.9k12k8pathwaycom/pathwayPython ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.PythonPyPIbatch-processingkafka62.1k1.7k9rclone/rclone"rsync for cloud storage" - Google Drive, S3, Dropbox, Backblaze B2, One Drive, Swift, Hubic, Wasabi, Google Cloud Storage, Azure Blob, Azure Files, Yandex FilesGocliGo Modulesgolanggo+15 ★/日60.5k5.6k10soimort/you-get:arrow_double_down: Dumb downloader that scrapes the webPythoncliPyPIdownloadervideo56.9k9.7k11apache/airflowApache Airflow - A platform to programmatically author, schedule, and monitor workflowsPythonPyPIairflowapache+15 ★/日47.3k18.1k12DIYgod/RSSHub🧡 Everything is RSSibleTypeScriptnpmrsstwitter+3 ★/日46.6k10.3k13DataTalksClub/data-engineering-zoomcampData Engineering Zoomcamp is a free 9-week course on building production-ready data pipelines. Join the course here 👇🏼Jupyter Notebooktutorialdata-engineeringkafka+31 ★/日46.3k9k14imputnet/cobaltbest way to save what you loveSvelteappdownloaderreddit+14 ★/日45.3k4k15NaiboWang/EasySpiderA visual no-code/code-free web crawler/spider易采集：一个可视化浏览器自动化测试/数据采集/网页爬虫软件，可以无代码图形化的设计和执行爬虫任务。别名：ServiceWrapper面向Web应用的智能化服务封装系统。JavaScriptappnpmcode-freecrawler+1 ★/日44.7k5.4k16DataExpert-io/data-engineer-handbookThis is a repo with links to everything you'd ever want to learn about data engineeringJupyter Notebookawesomeapachesparkawesome+10 ★/日44.5k9.4k17apache/sparkApache Spark - A unified analytics engine for large-scale data processingScalapythonscala+6 ★/日44.3k29.5k18google-ai-edge/mediapipeCross-platform, customizable ML solutions for live and streaming media.C++librarymediapipec-plus-plus+14 ★/日37.4k6.2k19restic/resticFast, secure, efficient backup programGocliGo Modulesgorestic+26 ★/日36.7k1.9k20apache/kafkaApache Kafka - A distributed event streaming platformJavaMavenscalakafka+1 ★/日34.1k15.6k21kestra-io/kestraEvent Driven Orchestration & Scheduling Platform for Mission Critical ApplicationsJavaMavenorchestrationdata-orchestration+30 ★/日29.9k3.5k22alibaba/canal阿里巴巴 MySQL binlog 增量订阅&消费组件JavaMavenmysqlbinlog29.7k7.6k23celery/celeryDistributed Task Queue (development branch)PythonlibraryPyPIpythontask-manager+5 ★/日29k5.2k24ArchiveBox/ArchiveBox🗃 Open source self-hosted web archiving. Takes URLs/browser history/bookmarks/Pocket/Pinboard/etc., saves HTML, JS, PDFs, media, and more...PythonPyPIpocketwget+7 ★/日28.8k1.6k