Heritrix is the Internet Archive's open-source, extensible, web-scale, archival-quality web crawler project.

从新浪财经、每经网、金融界、中国证券网、证券时报网上，爬取上市公司（个股）的历史新闻文本数据进行文本分析、提取特征集，然后利用SVM、随机森林等分类器进行训练，最后对实施抓取的新闻数据进行分类预测

HTTP API for Scrapy spiders

Open-source Enterprise Grade Search Engine Software

DotnetCrawler is a straightforward, lightweight web crawling/scrapying library for Entity Framework Core output based on dotnet core. This library designed like other strong crawler libraries like WebMagic and Scrapy but for enabling extandable your custom requirements. Medium link : https://medium.com/@mehmetozkaya/creating-custom-web-crawler-w…

This program provides efficient web scraping services for Tor and non-Tor sites. The program has both a CLI and REST API.

ralger makes it easy to scrape a website. Built on the shoulders of titans: rvest, xml2.

Raspagem de dados para iniciante usando Scrapy e outras libs básicas

An example using Selenium webdrivers for python and Scrapy framework to create a web scraper to crawl an ASP site

An extension for tracking your activities on myanimelist.net

Scrapyman数据接口服务。提供：淘宝、小红书、京东、抖音（电商）、抖音（视频）、快手、蒲公英、星图、拼多多、微信公众号、大众点评、哔哩哔哩、知乎、微博、贝壳、Bigo、Temu、Lazada、Shopee、SHEIN、百度指数、携程、Boss直聘、智联招聘、拉钩、今日头条、Facebook、Youtube、Instgram、Twitter。爬虫、采集、scrapy、接口、API。

ARGUS is an easy-to-use web scraping tool. The program is based on the Scrapy Python framework and is able to crawl a broad range of different websites. On the websites, ARGUS is able to perform tasks like scraping texts or collecting hyperlinks between websites. See: https://link.springer.com/article/10.1007/s11192-020-03726-9

A Web Crawler based on LLMs implemented with Ray and Huggingface. The embeddings are saved into a vector database for fast clustering and retrieval. Use it for your RAG.

Web-scraping script that writes the data of all players from FutHead and FutBin to a CSV file or a DB

News extraction and scraping. Article Parsing

Project on building a web crawler to collect the fundamentals of the stock and review their performance in one go

The Ultimate Guide to Sneaker Bot 🤖 Creation using JavaScript and NodeJS ☣️ . Learn how to get the most out of tools like the Chrome devTools, and JS Libraries like Puppeteer or Axios.

API definition, resources and reference implementation of URL Frontiers

Automates the process of repeatedly searching for a website via scraped proxy IP and search keywords

API to parse tibia.com content into python objects.