Performance

What a governed call costs

{{ lede }}

Size a deploymentSee the numbers
Read the story behind these numbers → MySQL, a database per microservice → Benchmarks by shape, from one worker to the run store →
{{ segNote }}
01 · The numbers

One worker, on PostgreSQL

0.43.0, one gunicorn worker as in the Docker image, a saved query by primary key on 10 million rows, 32 callers at once.

852-1,012 calls/sPer worker (one gunicorn process with 8 threads), a primary-key lookup, 32 callers. 1,012 with the metadata cache on.
1.00 msOne call on its own, at p50, with the metadata cache. 1.27 ms without it.
1.60 msOne call with PostgreSQL 0.5 ms away, with the metadata cache: 4× faster than 0.42's 6.58 ms.
~5%What column masking and a table allow-list cost a call: 809 against 852 calls a second, 964 against 1,012 cached.
0.42 against 0.43One worker, 32 callers, calls a second; median of 5 rounds, min-max in brackets.
Call0.420.430.43, cached
Lookup, plain key786 (751-829)852 (827-892)1,012 (991-1,043)
Lookup, masking and table allow-list738 (713-771)809 (803-856)964 (956-992)
100-row page with a join682 (661-694)725 (710-758)835 (833-873)
Write793 (764-836)850 (832-887)1,002 (999-1,046)

PostgreSQL answers this lookup in 0.03 ms. Everything else is the gateway.
01 · The numbers

One MySQL server, six services

Per-user queries joining across six microservice databases on one MySQL server, one gunicorn worker, 32 callers at once.

+1.0-1.3 msWhat the gateway adds to a call, with the metadata cache, on per-user queries MySQL answers in 0.03-0.42 ms.
662-875 calls/sPer worker, 32 callers, across three per-user queries joining up to three databases. 727-875 with the metadata cache.
+52%What 0.43 serves over 0.42 with MySQL 0.5 ms away: one round trip fewer, and no more stalls.
+21%One worker with gunicorn's access log and the request lines off, where a proxy logs requests already.
Per query, one worker, 32 callersCalls a second.
Query (databases it joins)MySQL directly0.420.430.43, metadata cache
One customer by id (crm)75,420647743875
Order-Status (crm, orders)25,063573662727
Stock-Level (org, orders, inventory)5,343630751833

One MySQL server, a database per microservice, one read-only connection. Joins across databases cost no more than joins within one.

Closed in 0.43. Table allow-lists on MySQL match database.table: a key granted crm's customer can't read billing.customer. When upgrading, a bare grant now means the connection's own database; keys that read another database's tables need those grants written out. 0.43's breaking changes →

The full MySQL page: threads, busy users, the store across a network, keeping services apart →

Latency under load, scale-out and where the time goes on MySQL: see the full MySQL page →
02 · Under load

Plan at 75%.

p50 stays flat right up to capacity. p99 holds at about 4-5 ms to 75% load, then climbs: about 16 ms at 90%. At capacity a queue builds.

0.430.43, cachedOffered load, as a share of capacity
051015202525%50%75%90%100%p99 msoff scaleplan here4.64.315.913.3

At 100%, a queue builds: p99 reaches 108 ms (141 ms cached), off this scale.

Measured open-loop, at fixed request rates, with latency corrected for coordinated omission - so saturation isn't hidden by slow answers delaying the next request.

03 · Scale-out

Add processes, not threads.

0.420.42, cachedCalls a second, 64 callers
1 worker
734
869
2 workers
1,196
1,441
4 workers
1,716
2,112
8 workers
2,215
2,727

Measured on 0.42; 0.43 didn't change how it scales. On MySQL, 0.43's scale-out on a shared PostgreSQL store is on the MySQL page.

  • Throughput grows with processes. On this machine eight workers shared four physical cores, so it grows less than linearly; containers on their own cores scale better.
  • The metadata cache halves PostgreSQL's work: at 8 workers, 1.65 cores without it and 0.92 with it. Most of that load is the gateway's own store, not your queries.
  • Threads. For fast queries, 2 threads per worker beat the default 8 by 13% (1,012 against 896 calls a second, on 0.43): extra threads compete for Python's interpreter lock. Keep 8 or more when calls wait - slow queries, exports, long-lived event streams. Set them with QUERYAPIGATE_THREADS.
04 · Where the time goes

Round trips, then Python.

Round trips a call waits forthe gateway's store your database
0.4273 to the gateway's store, 4 to your database
0.4343 + 1
0.43, cached10 + 1

Counted by timing calls at increasing network delay. With the cache, a call makes one round trip: the query, sent as one message that also sets its access mode and time limit. Across a 0.5 ms network, a call took 6.58 ms in 0.42, 4.39 ms in 0.43, and 1.60 ms with the cache.

Where the CPU goes0.43, without the cache
The gateway's store: the key, the query and the connection
22.9%
Logging
21.2%
gunicorn, sockets, HTTP
18.7%
Flask and Werkzeug
10.8%
Your query: psycopg2 and the pool
7.6%
Request handling
5.6%
Everything else
13.2%
The next cost: logging
  • With the metadata cache on, logging is a quarter of a call's CPU (25.2%), the largest single part.
  • Where a proxy or load balancer already logs requests, turn both off: QUERYAPIGATE_ACCESS_LOG=off and QUERYAPIGATE_LOG_REQUESTS=off were worth 21% on MySQL. Errors, warnings and slow queries are still logged.
05 · Rust data plane

Python decides, Rust serves.

A small Rust process in front of the gateway serves saved-query reads itself, from plans the Python server computes, and forwards everything else unchanged. Every security decision stays in Python; every run it serves is still recorded there.

Experimental: in the image and the Helm chart since 0.47.0, off unless started (ADR 0030). Measured, and decided go (ADR 0028).

  1. 1
    Python answers the first call, as it always does, and adds a plan to a successful response: the statement to send, the masking to apply, and digests of the key, query and connection rows it was made from.
  2. 2
    The data plane serves the calls after it. It re-reads those three rows each call and serves only while they still match. A revoked key or an edited query changes a digest, and the call goes to Python.
  3. 3
    Python records every run, reported in batches: run history, the event log and /metrics, as if it had served them.
7.3×The calls a second per gateway core of Python with the metadata cache: 6,734 against 925, every run recorded.
0.11 msOne call on its own, at p50, through the data plane. 1.01 ms through Python with the cache.
3.2×Per core without the metadata cache, on plain and governed lookups: the ADR's bar was 3×.
+0.35 msWhat a call the data plane forwards (a write, say) pays for the extra hop, at p50.
7.9×Calls a second per gateway core, each side at its best on the same 8 cores: 23,619 against 2,976.
3.8 msp99 at 64 callers through the data plane, against 93 ms for Python's 8 workers.
3.2-4.4×Per core without the metadata cache, across the customer lookup, Order-Status and Stock-Level.
100%Of runs recorded in every MySQL cell, with 2 Python workers behind the data plane.
16,403 calls/sFrom one pod, as the image ships it in 0.47.0: the data plane held to 2 CPUs, one Python worker, PostgreSQL with the metadata cache, 64 callers, for 10 minutes. 9,842,067 runs, every one recorded; p99 5.5 ms. The image's data plane serves at 0.95-0.99× of the from-source build measured above. The image measurement →
Each side at its best, on the same 8 cores64 callers, calls a second, median of 3 rounds.
Python + cache, 8 workersData plane + cache, 2 Python workers behind it
MySQL customer lookup2,976 at p99 93 ms23,619 at p99 3.8 ms
MySQL Order-Status, governed key2,252 at p99 116 ms13,863 at p99 6.4 ms
Gateway cores used7.37.3 and 5.8

Every run recorded in both. Python's eight workers can't use their cores: 405 calls a second per core, against 769 for one worker on its own.

Where it doesn't help
  • writes, which it forwards: 0.8× per core, and 0.35 ms more at p50;
  • a 100-row page on PostgreSQL without the metadata cache: 2.8× per core, where PostgreSQL itself is the limit;
  • the first call for each query and key, which Python answers;
  • anything the prototype leaves to Python: administrators, signed-in users, other formats, streaming, rate-limited keys.
What it needs
  • Two Python workers behind it. One Python process records about 17,000 runs a second; the data plane serves more. With two, every run was recorded up to about 24,000 calls a second on the benchmark laptop.
  • Disk for the event log. Every run is kept for 24 hours by default. At tens of thousands of calls a second that's more than a SQLite store should hold: use the run store below, or shorten QUERYAPIGATE_EVENTS_RETENTION_HOURS.
Full report →Try it →The story →The design and the decision →
06 · Run store

Every run, recorded.

Experimental, since 0.45.0. Measured and decided go, security-reviewed, and measured again at 250 million runs (ADR 0029). The data plane writing its own runs needs the data plane, in the image and the Helm chart since 0.47.0.

An optional ClickHouse store for run history, the event log and usage. The data plane writes every run it serves straight to it, in batches, with an insert-only user. Keys, queries and the audit log stay where they are.

100%Of runs recorded at the 64-caller peak, where 0.44's store kept 52-54%. And at 20,000 calls a second for 30 minutes: 35,999,897 of 35,999,897.
13 BA run, compressed, in the event log and in history each: 6.9 GB for 250 million runs, from 134 GB of raw entries.
−11%Gateway cores at 20,000 calls a second: Python's share fell from 0.49 cores to 0.13. One Python worker behind the data plane is enough again.
0Events missing or repeated for clients that hang up and resume at random: 35 million events over 7,515 reconnects.
Reads at 250 million runsp95 of 20 sequential reads through Python.
Readp95ADR 0029's bar
GET /api/v1/history, the newest 10012.5 ms100 ms
One key's newest 10073.6 ms100 ms
The Console's versions and run counts37.3 ms100 ms
A 24-hour usage report, exact66.7 ms500 ms
One key's replay of the whole event log 34,893 events6.2 s10 s

Usage reports count every run, whatever the history sample rate keeps: whole minutes from per-minute rollups, the edges from the event log.

  1. 1
    One insert per batch. Each batch of runs is one insert into events, up to 100,000 runs. ClickHouse fills history and the rollups from it through materialized views, and drops whole partitions as they age out.
  2. 2
    Event ids that keep their order. Each writer leases a block of ids for every millisecond, so the data plane and Python never mint the same one. Ids stay below 2^53, so JavaScript clients hold them exactly.
  3. 3
    Delivered once settled. queryapigate events delivers an event once it's a second old, so a client resuming from any id it was given misses nothing. A rare late insert is counted in /metrics.
What changes when it's on
  • history is kept by time, 30 days unless set, not by a count per version;
  • live events arrive about a second later;
  • a backup of the metadata store no longer holds the runs: back up ClickHouse with its own tools;
  • ClickHouse 24.8 or later, over https unless it's on the same machine.
Limits
  • measured on one laptop, with a single ClickHouse server: replicated clusters are an open question;
  • 250 million runs is a 30-minute soak's real runs, copied through the same views;
  • one queryapigate events process delivers about 6,500-10,000 events a second per stream: run several behind a load balancer for more.
Keep runs in ClickHouse →The story →The measurements →The design and the decision →
07 · Sizing

How many workers?

A worker runs out of CPU or of threads to wait with, whichever comes first. Put in your own load and this works out which, and how many workers to run.

sizing · workers
Workers needed {{ workers }} including one spare
Per worker{{ perWorker }} calls a secondlimited by {{ limit }}
CPU allows {{ cpuCap }}Waiting allows {{ waitCap }}

{{ hint }}

{{ caveat }}

The rule: a worker serves at most 670 fast calls a second on CPU (780 with the metadata cache), which is 75% of measured capacity. Waiting allows threads ÷ (query time + round trips × round-trip time), with 4 round trips a call in 0.43, or 1 with the cache. Take the smaller, divide your load by it, and add one worker you can afford to lose.

With the Rust data plane in front, size the data plane's CPU from its section above, and keep at least two Python workers behind it.

08 · How it's measured

Measured, not promised.

  • Every part on its own cores: the gateway, PostgreSQL and the load generator, on a 20-thread laptop on AC power with a performance profile. A /health check before and after every block shows whether the machine changed speed. One session, 2026-10-26 22:00 to 2026-10-27 00:28; the /health check stayed between 1,978 and 2,323 requests a second.
  • The builds take turns, in an order that rotates each round, 5 rounds per comparison.
  • A difference counts only when the min-max ranges don't overlap. By chance, that happens under 1% of the time.
  • Open-loop load at fixed rates for latency; closed-loop runs for throughput.
  • Round trips counted twice: from the statement log, and by timing calls at increasing network delay. The two agree.
  • Source or image matters for Python. Most rows were measured from source, on the laptop's Python 3.12.3. The Docker image's Python, from python:3.12-slim, links against a shared libpython and is about 30% slower per CPU for every call Python handles (BACKLOG #104). The image's Rust data plane serves at 0.95-0.99× the from-source build. So data-plane rows carry over to the image, and Python-only rows overstate what the container does today. Which build each number came from →
What it doesn't cover
  • one machine, not separate hosts;
  • no TLS, load balancer or Redis in the path;
  • a simulated network;
  • uniform random keys;
  • small results only;
  • no soak or failure tests.
Full method and every table →Raw results →
09 · Reproduce

Run it on your own machine

It starts its own PostgreSQL, pins each part to its own cores, compares two releases side by side, and cleans up after itself. About 2½ hours.

benchmarks · oltp
$ pip install -e ".[server,postgres]" py-spy
$ python benchmarks/oltp.py --out /tmp/qag-oltp --base v0.42.0 setup
$ python benchmarks/oltp.py --out /tmp/qag-oltp --base v0.42.0 env pgbench b1 b2 b3 b5 b7 b9 b10 b11 env
$ python benchmarks/oltp_report.py /tmp/qag-oltp/results.jsonl
Read the full write-upDeploy it