
Crawler.sh
Local AEO and SEO spider with markdown content extraction for technical SEO.
Crawler.sh: Launched. Get it at crawler.sh.
crawler.sh is a local web crawler and SEO auditor built in Rust for developers who need clean Markdown from websites for RAG pipelines, fine-tuning corpora, or agent context. It extracts main article content with word count, author byline, language, and excerpt, and can export a whole site as a Markdown archive. A custom JavaScript engine renders React, Vue, and Next apps without headless Chrome, while robots.txt rules and adaptive per-host pacing are respected by default. It also runs automated SEO checks and generates Sitemap XML. A free tier crawls 50 pages per session; CLI Pro and Desktop Pro cost $99 per year.
Company details
Lifeline
▣ GraduatedEngagement
Get the week's earliest tools in your inbox.
One email every Sunday: new launches, staff picks and early-access deals.