Pixep/crowlet: Tiny sitemap crawler for cache warming, and website status monitoring

How Perplexity Crawls and Indexes Your Website

Robots.txt in the Age of AI: Which Bots to Allow (and Why It Matters) - tiptop SEO Agency

C.L.A.R.I.T.Y.: The AI Optimization Framework Your Brand Needs - tiptop SEO Agency

List of Top AI Search Crawlers + User Agents (April 2025) | Momentic

mvdbos/php-spider at morioh.com

PowerShell Snippet: Crawling a sitemap

SiteAnalyzer - Website SEO Crawler for scan and technical audit of the websites

spatie/crawler: An easy to use, powerful crawler implemented in PHP. Can execute Javascript.

Scrapy | A Fast and Powerful Scraping and Web Crawling Framework