Crawler.sh logo

Crawler.sh

Visit

Local AEO and SEO spider with markdown content extraction for technical SEO.

Crawler.sh: Launched. Get it at crawler.sh.

crawler.sh is a local web crawler and SEO auditor built in Rust for developers who need clean Markdown from websites for RAG pipelines, fine-tuning corpora, or agent context. It extracts main article content with word count, author byline, language, and excerpt, and can export a whole site as a Markdown archive. A custom JavaScript engine renders React, Vue, and Next apps without headless Chrome, while robots.txt rules and adaptive per-host pacing are respected by default. It also runs automated SEO checks and generates Sitemap XML. A free tier crawls 50 pages per session; CLI Pro and Desktop Pro cost $99 per year.

Company details

Lifeline

▣ Graduated
TRACKING SINCE
MAR 31, 2026
LAUNCHED
MAR 31, 2026

Engagement

8
0
Curated
The Sunday rundown

Get the week's earliest tools in your inbox.

One email every Sunday: new launches, staff picks and early-access deals.

Use arrow keys to browse