Every night, only what changed, and nothing lost when a night fails

Your scheduler starts the run. QueryAPIGate takes a lease so two runs never overlap, asks the database for what changed since the last file that landed, and writes it under the one prefix the destination allows. The watermark moves only after the file is written, so a failed night is sent again the next.

1run at a time, on any instance

0rows lost when a night fails

≤ 300 MBto write 6 M rows to S3, measured

Who starts it

Front door

The governed core: one run

Reads from · writes to

↗ Your scheduler 0 2 * * * cronCronJobAirflow 2026-10-11 02:00

Someone runs it by hand POST …/orders_daily/runs

↗

Exports

Stable queryapigate exports run orders_daily POST /api/v1/exports/{name}/runs

the published version of one saved query, to one destination

The governed coreexport orders_daily↗

1 · Lease

One run at a time, on any instance. A crashed run's lease expires.

409 export_running

2 · Watermark

Binds :since to the last updated_at delivered.

since 10-13 23:57

3 · Stream

The published version, once, through a temporary file. The export's mask applies.

4 · Write

Only under the destination's prefix. Then, and only then, the watermark moves.

write failed

5 · Record

History: object, rows, watermark from → to.

export_failing

↗

PostgreSQL

shop WHERE updated_at > :since 4,214 rows changed

↗ Acme's bucket s3://acme-inbound/orders/

Writes are locked to this prefix.

Four nightsthe watermark moves only when a file lands

run 184 · 2026-10-11 02:002,310 rows · watermark → 10-10 23:58

run 185 · 2026-10-12 02:001,904 rows · a second run refused

run 186 · 2026-10-13 02:00write failed · watermark stays

run 187 · 2026-10-14 02:004,214 rows · nights 3 and 4 · watermark → 10-13 23:57

watermark

Stablecompatibility promise Expmay change in a minor release Test a destination first: one small probe object, written under its prefix.

Who starts itYour cron, CronJob or AirflowQueryAPIGate doesn't keep a clock; your scheduler runs the export.

Front door StableExportsOne command or one API call runs the published version of a saved query.

The governed coreLease → watermark → stream → write → recordOne run at a time. Only what changed since the last file that landed. Written under the destination's prefix, and only then does the watermark move.

Reads fromYour databaseWHERE updated_at > :since

Writes toTheir bucketAn s3://, gs:// or r2:// prefix; nothing can be written anywhere else.

NightsA failed night is sent againThe watermark stays where it was, and the next run delivers both nights' rows: at least once, never lost.

The same governed core as the full architecture.

whocron 02:00 doorqueryapigate exports run since = last watermark ✓one run at a time ✓locked to the prefix ✓ datas3://acme-inbound/orders/…parquet

For pipelines and partners' buckets

Deliver a daily file, incrementally, to someone else's bucket

Every night a partner needs yesterday's changes as Parquet in their S3 bucket. Writing that job means credentials, retries, a watermark and a place to see when it failed. A saved export is all of those.

✓A destination is an s3://, gs:// or r2:// prefix with credentials stored like a connection password. Every write is locked to that prefix.

✓Incremental: the largest updated_at delivered last time is bound to :since next run. The watermark moves only after the file is written.

✓One run at a time per export; a crashed run cannot hold the lease forever. The export_failing alert fires while the newest run failed.

✓Scheduling stays with what you have: cron, a Kubernetes CronJob, Airflow.

Guide 44: Deliver a daily file to a partner's bucket →Guide 30: Schedule an export →

Scheduled deliveries

# the export: query, destination, format, path, watermark column
curl -X POST https://data.example.com/api/v1/exports \
  -H 'X-API-Key: <admin>' -H 'Content-Type: application/json' -d '{
  "name": "orders_daily",
  "query": "orders_changed_since",
  "destination": "acme-bucket",
  "format": "parquet",
  "path": "orders/{date}/orders_{run}.parquet",
  "incremental": {"column": "updated_at", "parameter": "since"}
}'

# 02:00 every day, from cron or a CronJob
queryapigate exports run orders_daily
# → s3://acme-inbound/orders/2026-10-11/orders_184.parquet · 2,310 rows