Collection of Java-based web crawlers

Prerequisites

• Maven 3
• JDK 21

How to build

mvn clean install

Common crawler functionality

• Your crawler should extend WebCrawler base crawler class
• DTO class which describes collected data should implement CrawlerData marker interface

Crawler for Orthodox torrent tracker pravtor.ru

Check PravtorRuWebCrawler for details

To make search - use run-search script in pravtor.ru-crawler folder.
Collected data will be placed into result.xls file in sandbox folder

Crawler for vacancies aggregator rabota.by (localized version of hh.ru in Belarus)

Check RabotaByWebCrawler for details

To make search - use run-search script in rabota.by-crawler folder.