Documentation Page Delivery

This page describes how the documentation section is served inside the web application, from markdown discovery to rendered HTML and PDF export.

Route Entry Point

The documentation experience is owned by docs.py. It registers the /docs routes, discovers markdown files from the documentation/ directory, builds canonical slugs, groups pages by numeric section, and renders either the full template or a partial JSON payload for client-side navigation.

The route flow is:

1. List and sort the markdown files in documentation/.
2. Build a deterministic filename -> slug map.
3. Resolve the active page from either the canonical slug or a legacy filename-style URL.
4. Convert the markdown to sanitized HTML.
5. Render docs.html for full page loads or return JSON when the docs UI fetches partial content.

Template and Client Navigation

docs.html is the shell for the documentation UI. It provides:

• The left sidebar with section grouping and active-page state.
• The main article container where rendered HTML is injected.
• Previous and next navigation controls.
• The PDF export modal.
• Client-side progressive navigation that fetches only the updated article content when users move between pages.

The inline script in docs.html keeps sidebar expansion state in localStorage, intercepts internal /docs/... links, requests partial payloads with X-Docs-Partial: 1, swaps the article HTML, and then re-runs Mermaid and MathJax rendering.

Markdown Enrichment Pipeline

The markdown conversion logic also lives in docs.py. Before Mistune renders the markdown, the content is enriched in several passes:

1. Plain-text code file mentions such as docs.py, repo_scanner.py, or scripts under matching_and_import_db/ are resolved to repository-relative paths and turned into GitHub blob links.
2. Mermaid code fences are processed separately so node labels stay valid Mermaid syntax while matching nodes receive click commands that open the corresponding GitHub file.
3. Relative links to other markdown documents are rewritten to canonical /docs/<slug> routes.
4. GitHub-style alerts are converted to styled HTML blocks.
5. Special placeholders are expanded. For example, the color palette on 5.7 is injected from the backend to the web doc renderer.
6. The final HTML is sanitized with Bleach before it is sent to the browser.

The Mermaid-specific isolation is important because Mermaid has its own DSL. The general text-linking pass must not inject standard markdown links inside node labels.

Repository File Resolution

repo_scanner.py performs a startup scan of the repository and builds a filename-to-path index for code files. The scanner ignores non-code folders such as documentation/, static/, templates/, and generated caches so the lookup table stays focused on implementation files.

This index supports two documentation features:

• Linking plain text mentions like backend/app.py or get_atlas_data.py directly to GitHub.
• Resolving Mermaid node labels such as SG[get_atlas_gtfs.py] into click SG "..." commands without forcing documentation authors to hardcode repository paths.

When a filename is ambiguous, the service only links it if the path can be resolved uniquely.

Dynamic Stats Placeholders

docs_stats.py provides the stats placeholder helpers used by the documentation renderer and the PDF builder.

It supports placeholders of the form <span class="dynamic-stat" data-stat-key="summary.match_rate_percent" title="Auto-updated from pipeline stats">89.5</span>, loads data/stats.json, formats values for human-readable output, and can inject either HTML-wrapped spans or plain text depending on the rendering target.

That split matters because:

• The web UI wants <span> wrappers for styling and future dynamic behavior.
• Mermaid fences need raw text values so the diagram syntax stays valid.
• Static PDF generation needs explicit HTML escaping.

The same helper module also converts GitHub alert syntax into reusable HTML fragments and exposes the canonical palette block used in the visual design documentation.

Partial Rendering Contract

The documentation page supports both full-page and partial-page responses from the same route.

• A normal request returns the full layout through docs.html.
• A request with X-Docs-Partial: 1 returns a JSON payload containing the rendered article HTML, the active slug, the active section, and previous or next navigation metadata.

This keeps the docs section feeling responsive without introducing a separate frontend framework.

PDF Export Integration

The documentation routes also expose async PDF generation endpoints. The modal in docs.html posts export parameters to /api/docs/generate_pdf_async, polls task progress, and downloads the generated file when the background export finishes.

The exported PDF uses the same markdown source and stats helpers as the web page, which keeps the on-site documentation and the downloadable document aligned.