Service and system names below are generic (the trigger service, the data service, the publishing service) rather than internal names, out of respect for client confidentiality. The architecture, mechanisms, and numbers are real.
Why this couldn’t run inline
Publishing a dataset to a public catalog meant generating several export files — JSON, XML, CSV — each carrying up to 300,000 records, plus rendering and validating the metadata that describes them. That’s not a sub-second operation. Running it inside the request that triggers it means the caller sits there until it’s done, and every proxy, load balancer, and ingress timeout between the caller and the service gives up long before a multi-hundred-thousand-record export finishes generating:
The fix: acknowledge the request immediately (202 Accepted), and do the actual work
asynchronously, on a background worker:
Saying that sentence is easy. The hard part is everything it implies: what happens when a worker crashes mid-job? What if the same publication gets triggered twice at once? What if a recurring job was due while the whole system was down? An async job system that can’t answer those isn’t reliable — it’s just a queue nobody’s accountable for.
A four-state pipeline, not one big step
Every publication moves through the same four states, in order:
GET_DATA calls out to the data service that actually generates the export files — the slow step, since that generation happens on its side. PERSIST_DATA renders the catalog metadata against a public open-data standard (DCAT-AP), runs it through schema validation (SHACL) for the higher quality tier, and writes the result atomically into the triple store (RDF4J) — clearing out any prior version of the same entry in the same transaction that inserts the new one, so a republish never leaves two partial copies coexisting. PUBLISHED is retained indefinitely as the current state of that version, because a status endpoint and delete-by-version both need to read it long after publication finished.
Every republish of the same object gets a new version number; old published versions aren’t deleted when a new one lands — only a version whose attempt failed gets cleaned up.
The core guarantee: a record can’t hang forever
The whole resilience model reduces to one invariant: a record is always exactly one of three things — actively being processed, recovered because its worker died, or permanently failed after too many attempts. There’s no fourth state where a record just sits there, silently stuck, blocking anything that depends on it (including a uniqueness constraint described below).
This is implemented with a lease: every claimed record gets a locked_until timestamp,
set in the same atomic database update that claims it — using SELECT ... FOR UPDATE SKIP LOCKED so multiple worker replicas can claim different records concurrently without ever
racing for the same one. The lease isn’t a work deadline; it’s a liveness promise — “this
record won’t be touched by anyone else until at least this time.” Long-running phases refresh
it with a heartbeat partway through instead of waiting for a single lease to cover the whole
phase.
A separate recovery job periodically resets any non-terminal record whose lease has expired
back to REQUESTED, incrementing an attempts counter. Once attempts exceed a fixed cap (3),
the record stops being recovered — it’s moved to a permanent failure table instead, so a
genuinely broken record can’t loop forever burning worker time.
Preventing duplicate work
The same publication can be triggered twice near-simultaneously — a manual request and a
scheduled run landing at the same moment, for example. Rather than trying to prevent the race,
a partial unique index on the object’s identifier (scoped to non-published, i.e. active,
records) makes the database reject the second attempt outright. The caller that loses the race
gets a normal 202 Accepted back — the constraint violation is caught and treated as “already
in progress,” not surfaced as an error. Already-published versions of the same object don’t
count against this constraint — only one active attempt is restricted at a time.
Cleanup after a failed attempt
When an attempt fails, the failure is recorded — including what needs to be cleaned up — and the in-flight record is removed in the same transaction, so nothing stays half-alive on the pipeline. A separate, low-frequency job works through that failure log and asks the data service to delete whatever files it had generated for that attempt. That call is idempotent and retried if the data service is temporarily unreachable — safe to retry indefinitely, unsafe to give up on silently.
Recurring publications and catch-up
Some publications repeat on a schedule (daily/weekly/monthly) rather than firing once. A schedule table tracks each object’s next due time, and a throttled job checks it periodically, processing a bounded batch per run rather than firing every due object at once — which matters specifically after downtime, when a batch of overdue jobs could otherwise all fire in the same instant.
Catch-up is deliberately “once,” not “once per missed interval”: if a daily job was due for three days during an outage, it publishes a single current snapshot on recovery, and the next due time is advanced forward (from the originally scheduled time, not from now, to avoid drift) until it lands in the future. Nobody consuming this data wants three stale, back-dated snapshots — they want the current one, back on schedule from that point on.
What this bought
- A job that used to have to complete inside a single web request now runs entirely asynchronously, acknowledged immediately, with a status endpoint to check progress.
- A worker crash mid-job no longer leaves a publication permanently stuck — it’s recovered automatically, and a record that keeps failing gets flagged instead of retried forever.
- Two near-simultaneous triggers for the same publication can’t produce two competing attempts — the database rejects the second one outright, transparently to the caller.
- A failed attempt cleans up its own generated files instead of leaving orphaned data behind.
- A recurring publication that missed its window during downtime catches up once, not once per missed interval.