A1.2 Statistics and timestamps

The website shows statistics and dates about the published dataset: match rates, problem counts, when the sources were downloaded and when the data was last updated. This page explains where each value comes from. In short, the engine records facts about its run in the bundle, and the review application derives its analytics files from the bundle and its own database after each publication.

What the bundle provides

Every bundle's manifest carries, in metadata, a summary of counts, the complete dataset profile, the capabilities and a fingerprint of every input file. The Swiss integration adds its acquisition record: which sources were downloaded, when, and with which validators. The quality_metrics extension holds the engine's matching-quality summaries; the optional gtfs_atlas_stats and atlas_filtering extensions hold the Swiss GTFS identity and ATLAS filter statistics. The engine writes all timestamps in UTC.

What the application derives

After the switch, the publication step (publish.analytics) writes these files under data/:

File Written by Holds
data_meta.json record_bundle_metadata() in backend/services/data_meta.py, and the job runner The active run_id, the bundle's capabilities, the source acquisition times, the time the data was last updated, the result schema version
stats.json export_stats_after_import() in backend/importing/importer.py The analytics of the published snapshot, described below
gtfs_atlas_stats.json The importer A copy of the GTFS identity statistics. The application does not read it; in Docker the engine's workspace is the application directory, and the engine reads this file when its source_metadata.json lacks the statistics

stats.json combines values the engine wrote to the bundle with counts the application queries from its own database; it never recomputes spatial matching. backend/services/stats_export.py builds stats.json and the statistics PDF. Its recount_from_database() counts the summary, the problems and, with backend/services/dataset_stats.py, the per-source sections from the published tables only; the import and scripts/regenerate_stats.py both call it, so they publish the same numbers. The matches per rule and the unmatched OSM analysis come from the bundle, so regenerate_stats.py keeps those of the last import. The documentation pages read stats.json through {{stat:…}} placeholders, such as the key statistics of the start page.

dataset_publication.manifest in the database records the published run within the switch transaction, so it stays correct even if writing these files fails afterwards. In that case the run ends as "published with warnings" and the files can be regenerated.

The sections of stats.json

/data/analytics and the statistics PDF show these sections in this order.

Section Keys Source
Stop matching summary, matching_rules, propagation, quality_metrics stops_matched, osm_nodes, osm_stops and osm_stop_members for summary; the bundle's matches and the engine's quality metrics
Unmatched unmatched_analysis stops_matched for ATLAS; the bundle's unmatched records for OSM
Route matching route_links line_families, itineraries and their matches
Problems problems problems and stops_matched
ATLAS stops atlas_stops, atlas_filtering atlas_stops; the engine's ATLAS filter counts
Swiss GTFS gtfs, gtfs_atlas gtfs_stop_identity_resolution and the GTFS side of the route tables; the engine's GTFS identity statistics
OSM stops osm_stops; its identifiers block counts SLOIDs in OSM: elements with ref:sloid or railway:ref:sloid, those linked by it (sloid_identifier) and the platforms they reach, and matched elements without one osm_nodes, osm_stops, osm_stop_members, stops_matched
OSM routes osm_routes osm_route_relations and the OSM side of the route tables

Each figure names what it counts:

  • An OSM element is one active node or way tagged as a stop. Elements with a lifecycle prefix, such as disused:, are counted apart.
  • An OSM stop unit is one single element, or one pair or two-sided stop grouped as one physical stop (E2). It counts as matched when one of its elements is matched. An unmatched two-sided stop middle between two matched sides counts as matched.
  • A line is a GTFS route, or an OSM route master or group of relations (E3.2). An itinerary is one stop pattern: one per distinct GTFS stop sequence, one per OSM route relation.
  • A matched pair is one link between an ATLAS platform and an OSM element; one platform can have several. An effectively matched two-sided stop middle also counts as one, as in the map's summary.
  • An ATLAS platform or OSM element is isolated when it is unmatched and nothing from the other source lies within 50 m.

A line can have several links, because the timetable splits one line into several GTFS lines and OSM can hold one line twice. The page therefore counts the links, the linked lines of each side and the lines with several links apart; the coverage on each side is the share of its lines that are linked. Itineraries are linked one to one inside each link. OSM route figures count public transport relations only (E1.3).

Time zones

The engine and the pipeline status store use UTC (ISO 8601), and the browser converts status times to the viewer's local time. The Swiss presentation formats data-update and source dates in Europe/Zurich time with backend/services/time_utils.py; the presentation time zone never affects matching.

Code

File Role
backend/services/data_meta.py record_bundle_metadata(): writes data_meta.json
backend/importing/importer.py export_stats_after_import(): writes stats.json after the switch
backend/services/stats_export.py The matching sections of stats.json and the statistics PDF
backend/services/dataset_stats.py The per-source sections, read from the published tables
scripts/regenerate_stats.py Recounts the database sections of stats.json without the bundle
backend/services/time_utils.py Europe/Zurich times for the presentation

Related pages

Data update in progress
Elapsed: -- ETA: -- Phase: idle