A4.4 Running the quality checks

This page explains how to run, maintain and troubleshoot the quality checks of the application and the engine: the commands, the supported setups, CI, the module catalog and its baselines. Start in the application repository. To read the results, see A4.3 Code quality.

Choose a command

Command Purpose Requirements
make quality-fast-container Catalog, architecture, Python/JS/CSS lint, template/config syntax, vendor checks and focused quality/contract tests Docker Compose v2 and an engine checkout
make quality Full app, browser and engine suites; PostGIS publication and producer/consumer checks; coverage sanity; duplicated blocks; report and code map. CI runs this command too Docker Compose v2 and an engine checkout
make quality-fast Same fast checks using installed local tools Python 3.13, Node 22.x at least 22.13, requirements.txt and requirements-dev.txt
make quality-down Remove the dedicated quality Compose stack and its disposable data Docker Compose v2

A fast pass ends by naming make quality. It does not execute the complete browser or application suite, so it establishes neither coverage nor publication safety. Dependency audits and CodeQL are separate CI checks; make quality does not run those network services.

Even a full pass leaves some things unchecked. There is no type checking, no end-to-end, rendering or visual test in a real browser and no performance budget. Mutation testing is the engine's own check, on a sample of its result validation.

Supported local setup

Docker: the complete reproducible path

Keep a quality-enabled engine checkout beside the application:

workspace/
  stop_sync_osm_atlas/
  engine/
    quality/modules.yml

From stop_sync_osm_atlas/:

make quality-fast-container
make quality
make quality-down

For another engine location, set ENGINE_DIR, in .env or on the command line:

make quality ENGINE_DIR=/absolute/path/to/engine

Every tool finds the engine the same way: ENGINE_DIR if set (relative paths start at the application checkout), else engine/ inside the application, else ../engine. The Python tools share engine_dir() in backend/version.py; Docker Compose also reads ENGINE_DIR from .env.

Dockerfile.quality supplies Python 3.13, Node 22, PDF rendering libraries, app tools and an isolated engine virtual environment. compose.quality.yml bind-mounts app source and the engine checkout, keeps engine source read-only, and gives the runner its image's Node dependencies. The dedicated osm-atlas-quality project provisions PostGIS 16/3.4 with disposable in-memory database storage. It does not share production volumes or use the deployment's database credentials. Full publication tests can replace test data, so this separation is essential.

Both container commands build the image. Rebuild after requirement, lockfile or engine dependency changes; changing a bind-mounted source file does not update installed packages. Preflight rejects an installed Python package that does not satisfy the requirement files.

A full run clears quality/raw/ before collecting evidence, so a failed run cannot inherit an earlier passing result, then writes outputs back into the app checkout. Save any previous run you need to investigate before starting another full run. Stop editing either repository while the final run is in progress: source changes invalidate its fingerprints, including changes to documentation, configuration and tests.

Native development

Use Python 3.13 and Node 22.x, version 22.13 or later. The runner checks those versions; an arbitrary newer Node major is not the supported toolchain. Native PDF tests also need the platform's Pango libraries. Prefer the container when those libraries are unavailable.

python3.13 -m venv .venv
. .venv/bin/activate
python -m pip install -r requirements.txt -r requirements-dev.txt
npm ci
make quality-fast PYTHON=python

requirements-dev.txt holds the test tools and the exact analyser versions the quality runner uses. It includes Hypothesis and JSON Schema because ordinary pytest collection includes property and quality-schema tests. See A3.1 Dependencies and builds.

Do not install osm-public-transport-matcher into the app interpreter. Full verification checks this boundary explicitly. To run the full command natively, create a separate engine environment and supply a database dedicated to tests:

python3.13 -m venv ../engine/.venv
../engine/.venv/bin/python -m pip install '../engine[swiss,gtfs,quality]'
TEST_POSTGRES_URI=postgresql+psycopg://test:test@localhost:5432/transport_test \
  python scripts/run_quality.py full --engine-python ../engine/.venv/bin/python

The example assumes that disposable PostGIS database already exists. It is not a command for connecting to a production database. For engine-only work, use the engine's own quality guide, engine/quality/README.md.

CI and repository setup

The Required quality job of the Quality workflow (.github/workflows/tests.yml) runs on pull requests, pushes to main and manual dispatch. It checks out the engine commit pinned in engine.ref into engine/ and runs make quality, so CI builds the same image and disposable PostGIS database as a local run. The app suite runs without an engine installation; the engine suite uses the image's separate engine interpreter, which the app's engine import contract also runs as a subprocess.

Repository administrators must configure branch protection or rulesets requiring Required quality, Dependency security audit, and the applicable Security analysis checks for Python and JavaScript/TypeScript. Confirm the exact check names after the first run.

Committing workflow files does not configure repository protection or publish engine changes. Update engine.ref as part of a reviewed cross-repository contract change.

The required workflow always uploads the quality-<app SHA> artifact with the raw evidence, including failed-run outputs, and keeps it for 30 days. Inspect job output alongside the artifact: missing outputs may indicate that collection itself failed. On a push to main the job also publishes quality/evidence.json for the dashboard (How the evidence reaches the site). No workflow commits or pushes.

The Periodic quality workflow (.github/workflows/deep-quality.yml) runs production Python dependency auditing and an npm audit on pull requests, weekly and on manual invocation. The npm audit fails at high severity; the Python audit fails on its reported vulnerabilities. Its dependency-security-<app SHA> artifact preserves both audit reports. Dependency changes must update the corresponding requirement or lockfile and any vendored assets, then rerun behavioural tests.

CodeQL (.github/workflows/codeql.yml) analyses app Python and first-party browser JavaScript on pull requests, pushes to main and its weekly schedule. Vendor files are excluded. Findings are available through GitHub code scanning; a successful job is not itself a per-module security score. The report has no security column: read the audit and CodeQL results directly.

The standalone engine has its own production dependency audit and Python CodeQL workflow. Configure its required branch checks independently, as its quality guide describes. An app audit does not establish that the engine's separate dependency closure was scanned.

How the evidence reaches the site

The dashboard shows the evidence of the commit the site runs. Nothing is committed for it, and nobody runs a command before a deployment.

  1. On every push to main, the Required quality job uploads quality/evidence.json as the asset quality-<commit>.json of the GitHub release quality-data, also when a check failed. Anybody can download a release asset; a CI artifact needs a GitHub token, even from a public repository. The same step uploads the documentation's rendered Mermaid diagrams as diagrams-<commit>.tar.gz, for the PDF of a site without Node (A2.5). Of each kind, the job keeps the files of the newest 60 commits and of every commit a Git tag names, so a release tag deployed weeks later still finds its files.
  2. The application finds its own commit with app_revision() in backend/version.py: the APP_REVISION environment variable, else a REVISION file beside the code, else the .git folder of its checkout.
  3. On the first visit to the dashboard, backend/services/quality_report.py downloads the file of that commit, checks its shape and keeps it in .cache/quality/ and in memory. The file of a commit never changes, so GitHub is asked once for each deployed commit.
  4. When GitHub has no file for the commit, the dashboard says so and names the commit, and the application asks again after five minutes. This is the normal state while CI is still running, and for a commit that was never pushed to main.

A file quality/evidence.json in the checkout wins over the download. make quality and python scripts/quality.py map write it, Git ignores it, and the dashboard marks it as a local run. Delete it to see the CI evidence again.

The server therefore needs:

Requirement Why
The code is a Git clone, or the deployment writes the commit ID into a REVISION file or the APP_REVISION variable Otherwise the application does not know which evidence to ask for
Outgoing HTTPS to github.com and to the githubusercontent.com host that GitHub redirects downloads to To download the file, and the diagrams of the documentation PDF, once for each deployed commit
A writable .cache/ folder To keep the file across restarts; without it each worker downloads it once after starting

An image built from the dockerfile contains no .git: set APP_REVISION when starting it. The development Compose setup mounts the checkout, .git included.

Maintain the module catalog

File Who maintains it Meaning
quality/modules.yml Maintainers Module responsibility, criticality, owner, invariants, source/test paths, entry points, documentation and declared dependencies/contracts
quality/modules.schema.json Quality maintainers Validated catalog structure; the catalog uses JSON-compatible YAML
quality/exclusions.json Maintainers, with reasons Shared source exclusions and additional clone exclusions
quality/baseline.json Reviewed changes The accepted duplicated blocks
quality/report.schema.json Quality maintainers Structure of the generated report
quality/decisions.md Maintainers Why ownership, contracts, evidence and the baseline work this way

Every inventoried production source has exactly one ownership module. Source inventory combines tracked and new nonignored Git files, configured source roots and explicit files. It recognises Python, JS/MJS/CJS, CSS, templates, SQL, shell, YAML, JSON and TOML; explicit catalog entries include extensionless deployment files and requirement lists. Symlinks are not followed as source files.

When the patterns of several modules match a file, the pattern with the longest fixed start wins: a file named in full beats the pattern of its directory, and a deeper directory beats a shallower one. A directory therefore has a default owner, and the catalog names only the files that belong elsewhere:

Pattern Owner
backend/** app.query-api, the default for the backend
backend/jobs/**, backend/services/pipeline*.py app.pipeline-orchestration: a deeper directory and a narrower pattern
backend/blueprints/docs.py app.docs-delivery: a file named in full

A new file in such a directory needs no catalog change unless the default owner is wrong for it. A file in a directory no pattern covers, or matched equally well by two modules, fails the check.

Tests may support multiple modules and are associated through each module's test patterns. Contract nodes connect producer and consumer owners; a contract does not assign a second owner to their implementation files. Documentation paths are obligations, not an instruction to document every private helper.

The source exclusions cover generated evidence and documentation, the built CSS bundle, vendored/installed dependencies, runtime data, bytecode, the engine's mutation worktree, and tests/fixtures that are modelled as evidence. Migrations remain owned source, but their immutable steps are excluded from clone detection. Each repository can add explicit exclusions to its catalog; always explain the category being excluded.

Runtime coverage denominators are narrower than source ownership: app backend/**/*.py, engine src/**/*.py, and every first-party static/js/**/*.js. CSS, templates, deployment files and build tooling remain owned and checked, but are not assigned runtime coverage percentages.

When adding or moving code:

  1. Choose the existing responsibility owner, or add a module with a coherent responsibility and failure boundary.
  2. Add a source pattern only when the directory's default owner is wrong for the file. Update test patterns and required explanation/reference/operations documents. Set criticality with the rubric.
  3. Declare stable entry points, dependencies and cross-repository contracts. The dashboard flags code dependencies that the catalog does not declare.
  4. Run check, relevant focused tests and make quality-fast; finish with full evidence appropriate to the change.

Command-line reference

Run python scripts/quality.py --help for all options. Examples below assume the supported environment; each command finds the engine like every other tool, and --engine-dir PATH overrides it.

Command Result
python scripts/quality.py check Validates the catalog schema, single ownership, documentation/test references, entry points and enforced architecture boundaries
python scripts/quality.py coverage Validates current raw coverage against source fingerprints and all expected runtime files; does not execute tests
python scripts/quality.py debt Runs jscpd, writes the duplicated blocks to raw/debt.json and fails on one that the baseline does not accept
python scripts/quality.py report Reads the evidence of the last full run and writes the report and the code map to evidence.json; does not recollect tests
python scripts/quality.py map Reads the source of both repositories and writes the code map alone to evidence.json, in a few seconds

debt --write-baseline is an explicit baseline-maintenance operation, described in Duplicated blocks and the baseline. It must not be added to CI or to an automatic fix command.

The lower-level runner is python scripts/run_quality.py fast|full|syntax. fast and full accept --engine-dir; full also uses --engine-python (defaulting to ENGINE_PYTHON or /opt/engine-venv/bin/python). syntax parses Jinja templates and configured YAML/JSON without importing the app. Prefer the Make targets for ordinary use.

Evidence files and freshness

The full runner records in raw/checks.json both repositories' revisions and source fingerprints, the Python, Node and analyser versions, and each check's command, exit status and duration. It continues collecting independent checks after failures and returns nonzero if any required check failed. Report generation is useful even after a test failure; producing a report is not equivalent to passing the run. To look at a local run, open the dashboard of a local site: it shows quality/evidence.json when the file exists (How the evidence reaches the site).

Artifact, relative to quality/ Contents
evidence.json What the dashboard shows, in one file: report (app/engine revisions, generation time, completeness, tools, checks, each module's measurements and each measured file's coverage; null after map) and code_map (modules with their catalog entries, files, functions, links counted by kind, each module's names used from other modules, routes, blind spots, unused functions and duplicated blocks)
raw/checks.json Full-run manifest and check results
raw/fast-checks.json Fast-run manifest; does not certify earlier full-run evidence
raw/app-tests.xml, raw/engine-tests.xml Pytest JUnit test outcomes
raw/app-python-coverage.json, raw/app-python-coverage.xml App line/branch coverage in JSON and XML
raw/engine-python-coverage.json, raw/engine-python-coverage.xml Engine line/branch coverage in JSON and XML
raw/javascript-tests.json Jest suites, individual assertions and outcomes
raw/javascript-coverage.json, raw/lcov.info Browser production coverage: Istanbul statements, functions and branches keyed by source path, and LCOV for other coverage viewers
raw/jscpd/jscpd-report.json, raw/debt.json Raw clone-detector output and the normalised duplicated blocks
raw/python-audit.json, raw/npm-audit.json Separate dependency-audit workflow outputs, when copied/published with the evidence
raw/engine-python-audit.json Standalone engine production dependency audit, when copied/published with the evidence

Analyser formats retain their own details; the report is not a replacement for their raw reports. Coverage paths from another checkout are normalised for module association, and pytest class-based test names are attributed to their source test file. A shared test can count as evidence for multiple modules; module counts are not additive suite totals.

report refuses evidence that was not measured on the current source: the full-run manifest must record the same app and engine revisions and the same source fingerprints. Fingerprints hash tracked and new nonignored inputs while excluding generated quality/documentation outputs. Editing a document or test after collection therefore makes report fail; run make quality again. There is no stale state to show. Never hand-edit generated JSON.

Missing declared test files or skipped scenarios never become a clean module pass. The docs count measures required catalog slots, not the editorial quality or exhaustive correctness of those documents. Read the source documents when reviewing behaviour and invariants.

Coverage sanity

JavaScript tests use an instrumented browser-script loader. Coverage includes shared scripts and never-loaded first-party files, which correctly contribute zero hits. npm run test:coverage runs Jest and then scripts/check-js-coverage.cjs; that check requires every production JS file and actual executed statements and branches. Use A4.2 JavaScript tests when adding tests.

The full coverage sanity gate additionally requires all three app/engine/JS reports, matching source fingerprints, Python branch coverage, complete runtime file denominators, executed production statements in each report and valid count bounds. A green test process with broken instrumentation or missing files must fail this gate.

These are correctness checks on the evidence, not thresholds: no coverage percentage fails a run. The report keeps the line and branch coverage of each measured file beside the module totals, so the dashboard can colour a file by its own coverage.

Coverage measures execution, not correctness. A high percentage does not replace publication rollback, invalid-input, concurrency or contract scenarios. For a one-off look at backend coverage, run python -m pytest tests --cov=backend --cov-branch --cov-report=term-missing; it does not produce a supported quality report.

Duplicated blocks and the baseline

The clone gate uses jscpd at a minimum of 10 lines and 70 tokens. quality/baseline.json lists each accepted clone by its identity and its files; a clone that is not in it fails the gate. Unused code is not a gate: the code map marks functions that no production code names, and the dashboard lists them.

Baseline identities include content and participating files. Distinct physical occurrences are counted: adding another identical copy within an already baselined file pair still creates a new finding. Ordinary line movement does not change the content identity, but editing the code inside an accepted clone does: the edited clone is reported as new and its baseline entry matches nothing. Deleting one clone does not buy permission for a different new clone.

When the gate reports a new finding, read both copies in raw/debt.json and decide:

  • Refactor when both copies do the same job in one repository: move the shared code into one function and let the finding disappear.
  • Accept when the duplication is intentional, such as the bundle contract validated independently by the app and the engine, or when an accepted clone only changed content: re-record the baseline as below.

To re-record the baseline after removing or accepting findings, run:

python scripts/quality.py debt
python scripts/quality.py debt --write-baseline
git diff -- quality/baseline.json

Review the resulting JSON, confirm it reflects intended removals, and rerun the normal debt command. Do not refresh the baseline merely to make an unexpected failure disappear. Producer/consumer validation can intentionally duplicate a wire contract across independent repositories; sharing app/engine implementation imports would violate the boundary. Review the responsibility before extracting an abstraction.

Code map

python scripts/quality.py map parses the source of both repositories statically without running tests or needing a database, joining module ownership, files, functions, Python imports and calls, browser-script globals, template script order, requests to Flask routes, and declared contracts into quality/evidence.json (plus any findings from the latest debt run). See A4.3 Code quality for how the static graph, cleanup candidates, boundary checks and parser blind spots are defined.

Troubleshooting and review workflow

Symptom What to check
Unsupported Python/Node or missing package Use the supported versions with requirements.txt and requirements-dev.txt, or rebuild/run the container
Engine catalog missing Set ENGINE_DIR to a quality-enabled checkout and update engine.ref
App interpreter contains osm_public_transport_matcher Recreate an isolated app environment; use ENGINE_PYTHON for engine tests
PostGIS test missing/skipped Use make quality or provide a dedicated TEST_POSTGRES_URI; SQLite does not establish publication safety
Coverage all zero or missing JS files Use load-browser-script, keep production roots included in Jest, and run the coverage sanity command
Unexpected console/error or leaked timer Fix the cause or lifecycle cleanup; assert expected errors only within their test, as shown in the JS guide
New duplicated block Inspect raw/debt.json and both places before considering a reviewed baseline change
Report remains partial after passing tests The tests or the coverage of a module were not collected or are invalid; the module table shows which
report finds no evidence of the current source A file of either repository changed after collection; rerun make quality once inputs are stable
Dashboard has no results for its commit CI is still running or stopped before the report, the commit is not on main, or the server cannot reach GitHub (How the evidence reaches the site)
Dashboard does not know its commit The code is no Git clone: write the commit ID into a REVISION file or set APP_REVISION
Dashboard shows a local run Delete quality/evidence.json to see the CI evidence of the commit

Before requesting review, follow the checklist in Contributing.

Code

File Role
Makefile The quality* targets
scripts/run_quality.py Runs the fast and full check sequences and records their results
scripts/quality.py, scripts/quality_js_symbols.mjs Catalog check, coverage sanity, duplicated blocks, report and code map
.github/workflows/tests.yml Runs make quality and publishes the evidence of each commit of main
backend/version.py, backend/services/quality_report.py Find the running commit and load its evidence
quality/ The catalog, schemas, baseline and decisions
Dockerfile.quality, compose.quality.yml The reproducible quality environment

Related pages

Data update in progress
Elapsed: -- ETA: -- Phase: idle