InterroBot Web Crawler and Analyzer
Uncover your website's secrets with InterroBot - the web crawler for SEO pros, devs, and CMS admins. Find broken links, optimize content, and boost performance across Windows, Android, and macOS.
Web crawler
Crawl4AI Documentation
🚀🤖 Crawl4AI, Open-source LLM-Friendly Web Crawler & Scraper
unclecode/crawl4ai · GitHub
🔥🕷️ Crawl4AI: Open-source LLM Friendly Web Crawler & Scrapper - unclecode/crawl4ai
Firecrawl
Firecrawl crawls and converts any website into clean markdown.
cobalt.tools
save what you love. no ads, trackers, or other creepy bullshit.
AAndyProgram/SCrawler · GitHub
:rainbow_flag: Media downloader from any sites, including Twitter, Reddit, Instagram, YouTube, Pinterest, PornHub, XHamster, XVIDEOS, ThisVid etc. - AAndyProgram/SCrawler: Media downloader from any...
Knowledge Standards Foundation / EncycloCrawler · GitLab
GitLab.com
s0md3v/Photon · GitHub
Incredibly fast crawler designed for OSINT. Contribute to s0md3v/Photon development by creating an account on GitHub.
HTTP Archive
The HTTP Archive Tracks how the web is built by periodically crawl the top sites on the web and record detailed information about fetched resources, used web platform APIs and features, and execution traces of each page.
Common Crawl
StormCrawler
StormCrawler is collection of resources for building low-latency, scalable web crawlers on Apache Storm