doc-scraper

Crawl any docs site into clean Markdown, then let your agent search it offline over MCP.

Install Use with Claude Code How detection works Quick start

doc-scraper crawling a docs site and answering search queries offline

Offline for agents

The MCP server answers from the stored corpus: search_docs (BM25 full-text) and read_page need no network.

Detects the docs framework

doc-scraper add <url> recognizes Docusaurus, Starlight, Sphinx, VitePress, MkDocs Material, mdBook and more, and validates the content selector before writing config.

Clean output

Markdown per page plus pages.jsonl, llms.txt, and llms-full.txt.

Stays fresh

Incremental re-crawls, watch mode, and crawl history with get_freshness and diff_crawl.