Sphinx

How doc-scraper recognizes it

doc-scraper has no generator-tag signature for Sphinx, so detection is structural. doc-scraper checks the theme first, then plain Sphinx:

ID	Theme	Signals	Content selector
sphinx-furo	Furo	DOM article#furo-main-content; assets furo.css, furo.js	article#furo-main-content
sphinx-book	Sphinx Book Theme	DOM .sbt-scroll-pixel-helper; assets sphinx-book-theme.js, sphinx-book-theme.css	article.bd-article, main.bd-main
sphinx-pydata	PyData Sphinx Theme	DOM article.bd-article; assets pydata-sphinx-theme.js, pydata-sphinx-theme.css	article.bd-article, main#main-content
sphinx-rtd	Read the Docs theme	DOM body.wy-body-for-nav, .rst-content; assets _static/	.rst-content div[itemprop='articleBody'], .rst-content, div[role='main']
sphinx	Classic and other themes	DOM div.body, .sphinxsidebar, div.document, a.headerlink; assets _static/documentation_options, _static/doctools, searchindex.js	div.body section[id], div.body, article.bd-article, div[role='main'], div.document

A structural match with no generator tag is reported as medium confidence, and the selector is still validated against the page before it is used.

Add a site

$ doc-scraper add -config config.yaml -dry-run https://www.sphinx-doc.org/en/master/usage/quickstart.html
Probing https://www.sphinx-doc.org/en/master/usage/quickstart.html ...
Detected: sphinx via dom, confidence medium
Corpus:   ~1 pages (sitemap)

Drafted entry:

  sphinx_doc_docs:
    start_urls:
      - https://www.sphinx-doc.org/en/master/usage/quickstart.html
    allowed_domain: www.sphinx-doc.org
    allowed_path_prefix: /en/master/
    content_selector: div.body section[id], div.body, article.bd-article, div[role='main'], div.document
    max_depth: 3

  # content_selector: sphinx detected via dom (medium confidence), validated on the fetched page
  # allowed_path_prefix: /en/master/ covers 1 of 1 sitemap URLs
  # max_depth: 3 from sitemap path depth under the prefix

Preview of the fetched page:
  13072 chars of markdown, 93% of page text, code blocks 10/10, 10 headings
  ...

Dry run: nothing written.

Output from doc-scraper 2.10.1 (built from main) against https://www.sphinx-doc.org/en/master/usage/quickstart.html, 2026-10-10. Exit code 2.

Crawl it

sites:
  sphinx_doc_docs:
    start_urls:
      - https://www.sphinx-doc.org/en/master/usage/quickstart.html
    allowed_domain: www.sphinx-doc.org
    allowed_path_prefix: /en/master/
    content_selector: div.body section[id], div.body, article.bd-article, div[role='main'], div.document
    max_depth: 3

doc-scraper crawl -site sphinx_doc_docs

Caveats

• A thin sitemap gives thin scope evidence. sphinx-doc.org’s sitemap is generated by Read the Docs and lists a single URL, /en/master/, so the drafted scope rests on one URL and max_depth: 3 is derived from it. Raise max_depth (or set 0 for unlimited) if the crawl stops short.
• /en/master/ is the development version of the docs. To crawl a release, start from that version’s URL instead.