The benchmark harness of TripleTrail, its results and the dashboard that shows them.
TripleTrail is measured against PostGIS + pgRouting, QLever, Apache Jena (Fuseki +
jena-geosparql), GraphDB, Oxigraph and maplib, on spatial and routing workloads (benchmark v2,
Geographica 2, OSM at scale) and on the standard RDF suites (BSBM, Sparqloscope, Trainmarks,
updates). Every result comes from a run of the Benchmarks
workflow on a Hetzner server rented for that run: the run's page has its logs, its manifest
(bench/v2/results/runs/<run>.json) names the TripleTrail commit, the harness commit and the
machine, and the raw results are next to it in bench/v2/results/.
| Where | What |
|---|---|
bench/v2/ |
The harness: bench.sh (one entry point), run_all.sh and run.py (benchmark v2), the engines' settings (engines/), the queries per engine (queries/), the data generators (data/), the profiles (profiles/) |
bench/v2/suites/ |
BSBM, Sparqloscope, Trainmarks, updates, scale, OLTP |
bench/v2/geographica/ |
Geographica 2 |
bench/v2/cloud/ |
How a run rents, uses and deletes a server |
bench/rml/ |
RML lifting against Morph-KGC |
crates/tripletrail-bench/ |
Micro-benchmarks in Rust on a generated ERA-shaped network and on real RINF files (cargo run --release -p tripletrail-bench, --bin store, --bin cost, --bin rinf, --bin access), built against TripleTrail in core/ |
bench/v2/results/ |
Results, run manifests, and the files the dashboard reads |
site/ |
The dashboard (GitHub Pages) and summary.py, the headline numbers as JSON |
The results start empty: the first complete run measures the first public release.
| Dataset | Triples | Content |
|---|---|---|
small (synthetic RINF) |
805,538 | 24 × 24 grid of lines, 20 elements between crossings: 26,304 linear elements, 30,864 net relations, 1,201 tracks, 24,000 signals, 4,176 operational points, 48 national lines, 9 regions |
large (synthetic RINF) |
10,539,890 | 88 × 88 grid: 344,256 linear elements, 406,032 net relations, 15,665 tracks, 313,280 signals, 54,736 operational points, 25 regions |
osm_norway |
1,458,029 | From norway-rail-landuse-admin.osm.pbf (OSM as of 2026-11-05): 13,108 railway=rail ways, 1,123 stations, 275,718 landuse areas, 390 administrative areas (levels 2-7) |
geographica |
2,065,148 | The Geographica 2 real-world datasets used by the micro and macro workloads: CORINE land cover, DBpedia, GAG, GeoNames, hotspots, LinkedGeoData, each in its own named graph |
Synthetic RINF (gen_rinf_synth.py --h 24 --v 24 --k 20 --regions 3 --seed 42 for small,
--h 88 --v 88 --k 20 --regions 5 for large). It uses the ERA vocabulary as published:
era:LinearElement with geo:hasGeometry/geo:asWKT (CRS84) and era:length,
era:NetRelation with era:elementA/elementB, era:isOnOriginOfElementA/B and
era:navigability (Both / None), era:Track with era:wheelSetGauge (1435 or 1520 mm),
era:contactLineSystem (overhead or not electrified) and era:maximumPermittedSpeed,
era:OperationalPoint and era:Signal with POINT geometries, era:lineReference /
era:kilometer, and topological coordinates (era:netReference / era:hasTopoCoordinate /
era:onLinearElement / era:offsetFromOrigin). Lines cross on a grid. Every crossing has four
switch-connected link elements, one per turn, so routing is edge-based (the ERA micro profile:
graph vertices are element ends). Two differences from real RINF data matter for the results:
the geometry is a regular grid with short, straight elements (real track is longer and
curved, so per-geometry costs are lower here), and the regions are 9 or 25 jittered
rectangles (real admin units have many more vertices). The generator is deterministic, so the
same arguments give the same file on every machine.
OSM Norway. bench/v2/data/osm_extract.py builds the dataset from the pbf with osmium
(tags-filter + export, which assembles multipolygons). It writes N-Triples with osm2rdf's
IRIs (https://www.openstreetmap.org/{node,way,relation}/<id>) and geo:hasGeometry/geo:asWKT,
and loads the same WKT strings into PostGIS, so every engine sees identical geometries.
Geographica 2. bench/v2/geographica/run_geographica.py prepare unpacks the six
real-world tarballs into one N-Quads file, each dataset in the named graph the query lists use
(http://geographica.di.uoa.gr/dataset/{clc,dbpedia,gag,geonames,hotspots,lgd}).
- One engine at a time.
run_all.shstops every engine before it starts the next, and callssync. Nothing else ran while an engine was timed. - Warm and cold. Each query family starts with a restart of the engine. The first run of each query is the cold run: for the first query of a family the engine's caches are empty. The OS page cache stays warm, because dropping it needs root on the host. Then 2 untimed warm-up runs, then N timed warm runs. Each run has its own parameters (bounding box, point, route pair...), drawn from a seed per query and run index, so run i uses the same parameters on every engine. The tables give the warm p50 / p95 and the cold run.
- N.
small: 30 runs for selections, KNN and LR, 1,000 route pairs, 200 constrained pairs, 50 driving-distance / KSP / mixed runs, 10 cost matrices, 5 joins and aggregates.large: the same counts × 0.2 (WARM_SCALE=0.2, so 200 route pairs and one warm run for the joins). - Timing. Wall clock at the client, including the protocol (HTTP + SPARQL JSON results for the RDF engines, libpq for PostGIS) and fetching all rows. Timeout 300 s (600 s for joins and aggregates). QLever's result cache is cleared before every run.
- Query forms. Each engine runs its own fastest correct form of the same question, in
bench/v2/queries/<engine>/. TripleTrail and GraphDB use standard GeoSPARQLFILTERfunctions (GraphDB uses its indexedgeo:sfWithinproperty form for the box and polygon selections). Jena usesspatial:withinBox,spatial:nearbyandspatial:nearbyGeomfrom its spatial index; aFILTERwas faster than its property functions for the region join. QLever usesgeof:metricDistance, itsspatialSearchservice (S2 for KNN, libspatialjoin for the distance joins). PostGIS usesST_Intersects/ST_DWithinongeography,<->KNN and pgRouting functions on amicro_edgestable built from the same N-Triples. - Spatial indexes. TripleTrail builds its R-tree on the first spatial query (this is in the cold
times). PostGIS has GiST indexes on
geomand ongeom::geography, thenVACUUM ANALYZE. GraphDB has its GeoSPARQL plugin enabled after loading (the time is in "index"). Jena builds its spatial index file at first start. QLever builds with--vocabulary-type on-disk-compressed-geo-split. - Memory. Each engine gets the same fixed budget, as the only process running:
PostgreSQL
shared_buffers=4GB,work_mem=256MB,effective_cache_size=10GB,max_parallel_workers_per_gather=4,random_page_cost=1.1,jit=off(engines/postgis.conf). GraphDB and Jena-Xms8g -Xmx8g. QLever-m 8G -c 2G. TripleTrail default settings. - Routing graph. TripleTrail builds named networks from the ERA micro profile edge query
(
queries/tripletrail/setup_routing.ru;bench:microcadds gauge and electrification columns for the constrained route). PostGIS gets the same edges, computed in Python bydata/export_postgis.py. Vertices are element ends (E#fwd,E#bwd), cost is the element length in metres, and route pairs are drawn from all vertices, so some pairs have no path.
The workflow is the way results get here (see bench/v2/cloud/README.md).
To run something on your own machine:
git clone https://github.com/Matdata-eu/tripletrail-bench && cd tripletrail-bench
git clone https://github.com/Matdata-eu/tripletrail core
(cd core && cargo build --release -p tripletrail-server) # core/target/release/tripletrail
export BENCH_WORK=$PWD/bench/v2/work # stores and data; gitignored
mkdir -p $BENCH_WORK/jena && curl -o $BENCH_WORK/jena/jena-fuseki-geosparql-6.2.0.jar \
https://repo1.maven.org/maven2/org/apache/jena/jena-fuseki-geosparql/6.2.0/jena-fuseki-geosparql-6.2.0.jar
bench/v2/bench.sh v2 small # benchmark v2 on the small synthetic set, all engines
bench/v2/bench.sh bsbm 2785 tripletrail qlever # one suite, some engines
bench/v2/bench.sh all full # everything of profiles/full.txt (a 64 GB machine)Requirements: Linux, Docker (QLever, GraphDB, Oxigraph and the suites' tools run in
containers), Java 21 for Jena, Python 3 with requests, pyyaml and psycopg2, osmium-tool
for OSM, bc and GNU time, and either a local PostgreSQL 16 with PostGIS and pgRouting or the
compose service (docker compose -f bench/v2/docker-compose.yml up -d postgis). Environment
variables: BENCH_WORK, WARM_SCALE, ONLY (queries or families, e.g. ONLY=6,mixed_*),
SKIP_LOAD=1, DROP_AFTER=1, TRIPLETRAIL_BIN, JENA_JAR, JENA_HEAP, GRAPHDB_IMAGE,
GRAPHDB_HEAP, QLEVER_IMAGE, QLEVER_MEMORY, QLEVER_CACHE, PG_DSN_BASE, PG_CONTAINER,
GEOGRAPHICA_TIMEOUT.
The harness is under the Apache License 2.0 (LICENSE). The suites' own tools and
data keep their licences (listed in bench/v2/suites/README.md);
nothing under a copyleft licence is copied into this repository, it is fetched at run time.