Cache Purging

Janus actively purges the Cloudflare (CF) cache whenever content or routing changes, so the CDN never serves stale pages longer than the propagation delay of a purge call.

Purging is implemented once, as a reusable microbatch topology (topologies/cache_purging.clj), and instantiated per module. It runs two independent flows against CF’s purge_cache API on the module’s zone:

Flow Trigger CF purge mode

By cache tag

An entity update becomes externally visible

tags — removes every cached response tagged with the entity’s URI

By URL prefix

A redirect functionally changes

prefixes — removes every cached URL under host/path

Purging by cache tag is the primary mechanism: it is what keeps every page that includes an updated entity fresh, regardless of the URL it was served under. Prefix purging exists because a redirect is not an entity — nothing is tagged with it — so the affected URLs must be purged by address. Redirect handling itself is documented in Redirects.

Purge by cache tag

The tag: global entity URIs

Every entity id has a global URI representation produced by janus.entities.api/entity-id→uri with the module’s :system-id:

janusg:{system-id}:{entity-tag}:{id}      ; string ids
janusg:{system-id}:{entity-tag}:{id}:i    ; integer ids

janusg:chm/prod:ld/document:5001030:i
janusg:sve/dev:ld.data-record/tenant:4:i

The :system-id (e.g. chm/prod, sve/dev) namespaces the tags so several janus systems can share one CF zone without purging each other’s entries.

Flow

  1. The denormalizing topology appends every updated entity summary to *updated-entities.

  2. The routing topology parks the updated ids in a visible-guarantee PState and re-emits them to the *visible-entity-ids depot one microbatch later — guaranteeing that all PState writes for the update (e.g. $$path→page) are committed and externally visible before any purge fires. Purging can therefore never race ahead of the data it purges for.

  3. The cache-purging topology converts each id to its janusg: URI and calls the CF purge API with {:tags […​]}.

Ignored tags

Derived entities that are never cached by the CDN (dynamic-list machinery, derived lists) are filtered out via the :ignored-tags config set. The match is on the entity-id tag (e.g. :sve.dynl/query-result), not :janus/tag.

Requirement: Cache-Tag response headers

Purging by tag only removes responses that CF cached with these tags: the delivery serving the pages must emit a Cache-Tag response header listing the janusg: URI of every entity included in the response. This is a CF Enterprise feature and a delivery-side responsibility — janus can purge correctly and still have no effect if the delivery does not tag its responses.

Purge by URL prefix

Every functional redirect change emits a [path, origin] tuple to the module’s redirect-purge depot (see Redirects). The purge topology expands the tuple into one prefix per affected host using the :origin→host config map:

[path origin]  →  "host/path"        ; one entry, scheme-less
[path :*]      →  one prefix per configured host

Prefixes are scheme-less (host/path) because the CF prefix purge rejects https://. Which hosts are configured differs per module — see Module setup.

Batching and fault tolerance

Both flows aggregate per task and partition the items by :page-size (100 — the CF API maximum), issuing one API call per chunk.

A successful purge is logged with its item count. A failed one is logged and throws, which fails the microbatch; since both depot sources use {:retry-mode :all-after}, the batch is retried until it succeeds. Purges are therefore at-least-once — purging the same tag or prefix twice is harmless.

The CF client is a per-worker managed resource. The real client (cloudflare.public-api/client) authenticates with a bearer token read from AWS Secrets Manager (janus/cf-purge-token). A logging-client drop-in exists for modules whose CF zone/credentials are not settled yet: it logs every would-be purge call and answers success.

Configuration

The topology reads its config from the module’s :cache-purge key:

Key Example Description

:active

true

Master switch. false skips all purging (records are still consumed).

:page-size

100

Items per CF API call. CF allows at most 100; 0 also disables purging.

:cf-client

—

Per-worker CF client resource (real or logging).

:zone-id

"9a7d…"

The CF zone the purge calls target.

:system-id

"chm/prod"

Namespaces the entity URIs used as cache tags.

:ignored-tags

#{:chm.page/derived-list}

Entity-id tags excluded from tag purging.

:origin→host

{:tagblatt.ch "chm-sgt.prod…"}

Maps redirect origins to the hosts CF caches (prefix purging).

:active and :page-size (like the rest of the block) are runtime-overridable through the control topology’s config overrides, so purging can be toggled on a live module without a redeploy.

Module setup

CHM SVE

Client

Real CF client (token from janus/cf-purge-token)

logging-client — logs would-be purges; real client pending zone/token confirmation with devops

Hosts purged (prefixes)

Paywall-worker upstream hosts (chm-{code}-site.forward-publishing.net) — CF caches by the host the worker fetches from, not the public domain

Public hosts (www.schwaebische.de, www.nordkurier.de, …) — no paywall worker; Varnish reverse-proxies delivery directly

System ids

chm/dev, chm/stg, chm/prod

sve/dev (only Local/Dev modules exist so far)

Ignored tags

:chm.page/derived-list, :chm.dynl/*, :chm.external-feed/derived-list

:sve.dynl/query, :sve.dynl/query-result, :sve.dynl/author, :sve.list/author

Wiring it into a module

;; ids whose updates are guaranteed visible (fed by the routing topology)
(declare-depot setup *visible-entity-ids (hash-by ent/id))

;; [path origin] tuples emitted on redirect changes
(declare-depot setup *redirect-purge-paths (hash-by first))

(redirect-topology setup topologies
                   ;; ...
                   :purge-depot *redirect-purge-paths)

(cache-purging-topology setup
                        topologies
                        :name             "cache-purging"
                        :entity-depot     *visible-entity-ids
                        :url-depot        *redirect-purge-paths
                        :config           (:cache-purge config)
                        :config-overrides $$config-overrides2)

See modules/chm.clj and modules/sve.clj for the two live instantiations.