A3.5 Pipeline progress

The Data page (/data/analytics) shows the progress of a pipeline run live, without the review application knowing how the engine works inside. The engine describes its own stages; the application merges them with its own into one versioned stage plan, and records the state of each stage under a stable ID. This page describes that model, the order of the phases, the rules at the engine boundary, the timestamps and how the browser displays them. A3.3 describes how runs are started, in which modes, and how a dead run is detected.

Data model

backend/services/pipeline_progress.py defines the eight top-level phases and the application's own stages for loading and publishing the database. The separately installed engine sends a plan of its stages, each with its own label and description. backend/services/pipeline_status.py validates and merges both plans into the persisted status:

{
  "progress_schema_version": 2,
  "stage_plan": [
    {
      "id": "database.validate",
      "parent_id": "database",
      "phase": "database",
      "label": "Validate the result bundle",
      "description": "Verify the bundle schema, checksums and entity references.",
      "order": 10,
      "progress_kind": "indeterminate"
    }
  ],
  "stage_states": {
    "database.validate": {
      "status": "complete",
      "started_at": "2026-09-21T10:00:00+00:00",
      "finished_at": "2026-09-21T10:00:02+00:00",
      "duration_seconds": 2.0
    }
  }
}

A stage is waiting, running, complete, reused, skipped or failed. A stage of kind determinate may also report processed and total; the UI never invents a percentage for indeterminate work.

Execution order

The phases follow the processing order. They describe the work a user can see rather than process boundaries. The engine performs the first six; the application's importer performs the last two.

# Phase ID Work
1 Check sources source_check Probe ATLAS and GTFS freshness and prepare the geographic boundary
2 Prepare source files source_files Download and filter or reuse ATLAS, download and extract or reuse GTFS, prepare timetable patterns, identities and routes, then download the current OSM extract
3 Load matching inputs matching_inputs Read ATLAS stops and cached timetable products, optionally prepare a supplied GTFS feed, then parse OSM stops, route relations and evidence
4 Stop matching stop_matching Group OSM stops, run the stop-matching rules and detect stop problems
5 Route matching route_matching Normalise and compare line families, itineraries, directions and stop sequences
6 Build result bundle bundle Record fingerprints and quality metrics, write and compress the result, write checksums and validate the bundle
7 Stage database database Validate the bundle independently, prepare application rows, load an isolated PostgreSQL schema and build indexes, while readers keep using the published snapshot
8 Publish dataset publish Switch the staged tables into public view in one transaction, generate statistics and update the publication metadata. This publishes data; it does not deploy the website

The engine's stages within phases 1 to 6 are listed in E5.3; the application's stages are listed below.

The stages within a phase run in the order shown. A stage served from cache is marked reused in place. match-import skips the source check, reuses the source files and then loads the inputs normally. import skips acquisition, input loading and matching, and reuses the existing bundle.

This is progress schema version 2. Live events of another version are rejected. When the version changes, update the engine, the scheduler and the web application together, between runs. The result bundle format does not depend on it.

Application stages

Parent Stable ID Label
Stage database database.validate Validate the result bundle
Stage database database.prepare Prepare application records
Stage database database.load Load and index the staged database
Publish dataset publish.swap Publish the new dataset
Publish dataset publish.analytics Generate analytics

The import functions receive an optional callback for stage events. The events carry no progress_schema_version: the runner's callback in backend/jobs/job_runner.py stamps the current version before recording them, so the importer does not depend on the progress schema. Reporting is best effort: an exception in the callback is logged, and can neither abort the work nor hide the work's own exception. publish.swap finishes only after its transaction has committed, and the cleanup after it depends on the committed result even if reporting fails.

A failure to generate statistics is reported, not hidden. The importer then raises PublishedDatasetError with the committed manifest. The runner keeps the publication, records the data-update timestamp and finishes with dataset_published: true and non-empty warnings. The page shows "Dataset published with warnings" and does not claim that the statistics were generated. Such a run does not advance last_success_at.

Input warnings that the engine records in the bundle's metadata.warnings, such as a missing ATLAS column, come first in the same warnings list and lead to the same message.

Engine boundary

The runner copies every line the engine prints, standard error included, to the scheduler log, and reads the lines as JSON. A version-2 progress event is validated completely before a pure transition function turns it into one stored update. JSON without a progress version, such as the engine's log lines about cache decisions, is diagnostic output and never changes what the UI shows. An invalid event is logged and ignored: it neither changes the plan nor stops the reading of the engine's output.

The engine can add a stage without any change to the application, because the plan carries its label, description, parent and order. A conflicting definition for an existing ID is rejected during a run, and an unknown schema version is rejected rather than guessed.

Cache and run-mode outcomes stay explicit. When a whole phase is marked reused or skipped before the engine's plan arrives, its stages inherit that outcome.

Engine events may describe only stages below the engine's own phases, never a top-level phase or the database and publication work. Numbers must be finite and non-negative, counts must be integers, and a supplied stage_spec.id must equal stage_id. Definitions cannot conflict, and a plan cannot repeat an ID.

Lifecycle and persistence

pipeline_progress.apply_progress_event(current, payload, now) computes stage transitions without any I/O. project_stage_states derives the state of each phase from its stages and from the runner's phase. The browser uses these phase states instead of assuming that every earlier phase succeeded. A phase's duration is the sum of its stages' work. Preparing source files finishes before loading matching inputs begins; reuse and skip decisions take no work time.

The runner binds events to its run ID. The store checks that ID and the active-run state again under its write lock. A finished run ignores late updates; a completed or failed stage ignores repeated events and delayed heartbeats. An explicit start may replace an earlier reuse or skip decision, but completed or failed work is never silently restarted.

When a run fails, running stages are marked failed and stages that never started are marked skipped with a reason; finished work keeps its state. finish_failure() stores the error of the first failed stage that reported one as last_error, prefixed with the stage label, the stage ID as failed_stage and the active phase as failed_phase. The runner's own error, The matching engine exited with code N, is used only when no stage reported a cause; the engine command and its exit code stay in the scheduler log. A successful exit with unresolved stages produces warnings instead of invented completion. A new run resets the plan, the stages, the publication flag and the warnings. A lifecycle event that carries its own definition can still add a stage that the initial plan missed, under the same validation rules.

Timestamps

The status store records every stage event with a UTC ISO 8601 timestamp; the browser shows it in the viewer's time zone. Reused and skipped stages are recorded as events with zero duration, so every stage has a time without pretending that cached work took time. phase_history records each phase the run has left, with the same fields; a cache decision looks like this:

{
  "phase": "source_files",
  "started_at": "2026-09-21T14:18:23.100000+00:00",
  "finished_at": "2026-09-21T14:18:23.100000+00:00",
  "duration_seconds": 0.0,
  "status": "reused"
}

The other timestamps of the status record mean different things:

Field Meaning
started_at Start of the whole run; never changes during the run
phase_started_at Start of the active phase; cleared when the run ends
phase_history[].started_at / finished_at Measured boundaries of completed, failed, reused or skipped phases
updated_at Last write of the run state; neither a phase boundary nor proof of liveness
finished_at End of the run, successful or failed
last_success_at End of the most recent successful run without warnings
last_pipeline_data_import_ended_at Publication time used by the statistics snapshot
schedule_updated_at / next_run_at Scheduler metadata, independent of any run
Lease acquired_at / heartbeat_at / expires_at Lease ownership and health (A3.3); never used as stage times

Browser rendering

Polling and the navbar

Every page polls /api/system/pipeline_status: every 10 s when idle, every 1.5 s during a run. A failed or timed-out request keeps the last status until the next poll. The navbar is rendered with the current state on the server, so a run in progress is visible before the first poll.

Every status read also derives data_warning, which the navbar shows as an amber dot before Data updated, with the text as its tooltip:

Condition data_warning
The latest run failed before publishing Latest update failed; showing data from <time>, or …; no data is published yet
The publication is older than twice schedule_interval_hours No update for more than <hours> hours; showing data from <time>
Otherwise, or before the scheduler has recorded its interval None; the dot is hidden

The value is derived, never stored, so it appears as the data ages, without a writer.

The pipeline card

The pipeline card on the Data page does not depend on the published statistics. It stays visible during the first run of a new deployment, when stats.json does not exist; the notice about missing statistics appears below the card until statistics exist.

When a stage reported the failure of a run, the card shows the first line of its error, and below it the stage, its phase and when it failed; the full error is the message's hover text. Otherwise the card shows last_error.

Timeline and stage cards

static/js/pages/data-analytics.js renders both the phase timeline and the per-phase stage cards from stage_plan without page reloads:

Phase state Time shown Duration
Running phase_started_at Browser clock minus the phase start, refreshed every second
Completed or failed The start of its first stage, or the history entry's started_at for a phase without stages The sum of its stages' durations, or finished_at - started_at for a phase without stages, calculated by the backend
Reused When the cache decision arrived 0; the UI says Reused instead of 0s
Skipped When the mode decision was recorded 0; the UI says Skipped instead of 0s
Waiting None None

Completed phases weight their bar width by measured duration while the active phase grows by one unit per elapsed second (falling back to template proportions when no durations exist). The timeline displays elapsed time rather than estimated remaining time: processed, total and eta_seconds appear in status text only and never set a stage's duration.

Stage card summaries report N/M finished (counting complete, reused and skipped stages) while each stage row keeps its individual status. To preserve keyboard focus, text selection and XSS safety across 1.5 s polls, the script updates existing DOM nodes in place via textContent rather than innerHTML, inserts late-arriving stages in order position, and treats all stage IDs as opaque strings rather than CSS selectors.

Code

File Role
backend/services/pipeline_progress.py The phases, the application's stages, apply_progress_event and project_stage_states
backend/services/pipeline_status.py Records progress events in the stored status, finish_failure() and data_warning
backend/jobs/job_runner.py Reads the engine's output and stamps the importer's stage events
static/js/shared/pipeline-status.js Polls the status and updates the navbar
static/js/pages/data-analytics.js The timeline and the stage cards on the Data page
tests/test_pipeline_progress.py Pure transition, state-service, malformed-event, cache-inheritance and ownership tests
tests/test_engine_import_contract.py Replays real engine CLI runs through the application's progress consumer and publishes each bundle into PostGIS

Related pages

Data update in progress
Elapsed: -- ETA: -- Phase: idle