Depot dumps and loads

How to dump a foreign depot to a folder and load (upload) a dump into a depot.

Public API: janus.rama.api.ds (not re-exported from janus.rama.api).

(require '[janus.rama.api.ds :as ds])

You need a foreign depot handle from the cluster manager, for example:

(require '[com.rpl.rama :refer [open-cluster-manager-internal
                                foreign-depot]])

(def cluster
  (open-cluster-manager-internal {"conductor.host" "…"}))

(def my-depot
  (foreign-depot cluster "your.module.ns/ModuleName" "*depot-name"))

In Janus dev REPLs, dev.sve.conn / dev.chm.conn already re-export the Dataset helpers and provide (depot env *name) style access.

What you get on disk

A dump is a directory (not a single file):

/path/to/dump/
  manifest.edn           (1)
  partitions/
    0.part
    1.part
    …
1 Metadata: format version, encoding, compression, counts, optional source.

Choose a profile when writing:

Profile When to use On-disk shape

:archive (default)

Large / production dumps, transfer between environments

Nippy frames + Brotli per partition (compact, not human-readable)

:fixture

Small seeds and golden data checked into git

Plain EDN lines, no compression (diffable; records must be EDN-printable)

Dump a depot to a folder

Full depot

(ds/write-depot! my-depot "/var/tmp/my-depot-dump")
;; default :profile :archive

This freezes each partition’s current offset range, streams records out with foreign-depot-read, and writes them under /var/tmp/my-depot-dump. Returns the manifest map (counts, encoding, …).

Trailing window only

Useful for large depots when you only need recent data:

(ds/write-depot! my-depot
                 "/var/tmp/my-depot-tail"
                 :offset-back 10_000
                 :profile :archive)

:offset-back is applied per partition (same idea as depot-count / historical samples): up to that many records from the end of each task.

Git-friendly fixture

(ds/write-depot! my-depot
                 "test/fixtures/my-depot-seed"
                 :profile :fixture
                 :offset-back 500
                 :max-bytes 5_000_000)   ; optional size guard

Tips for fixtures:

  • Prefer a small :offset-back or a deliberately small depot.

  • Omit timestamps: do not pass :created-at? true if you want stable git diffs.

  • Fail if the path already exists, unless you pass :overwrite? true.

Overwrite an existing dump directory

(ds/write-depot! my-depot
                 "/var/tmp/my-depot-dump"
                 :profile :archive
                 :overwrite? true)

Writes go to a staging directory first; a failed write leaves any previous dump at the destination intact until commit.

Equivalent two-step form

write-depot! is sugar for “extract Dataset, then write”:

(ds/write! (ds/depot-dataset my-depot 1000)
           "/var/tmp/my-depot-tail"
           :profile :archive)

Load a dump into a depot

Open the directory as a Dataset, then append every record into the target depot:

(def dump (ds/open "/var/tmp/my-depot-dump"))

(ds/info dump)
;; => {:format :janus.rama.ds/v1
;;     :encoding …
;;     :num-partitions …
;;     :counts […]
;;     :total-count …
;;     …}

(ds/into-depot! target-depot dump)
;; default :ack-level :append-ack

One-liner:

(ds/into-depot! target-depot (ds/open "/var/tmp/my-depot-dump"))

Ack level

(ds/into-depot! target-depot dump :ack-level :append-ack)  ; default — durable append
(ds/into-depot! target-depot dump :ack-level :ack)         ; wait for colocated stream topologies
(ds/into-depot! target-depot dump :ack-level nil)          ; fire-and-forget (faster, less safe)

Use :append-ack for normal restores. Use :ack only if you need stream topologies to finish each record before continuing.

Important: placement is re-partitioned

into-depot! calls foreign-append! for each record. The target depot’s partitioner decides which task receives the record. Partition files on disk reflect the source layout only — they are not pinned to the same partition index on load.

That is correct for seeding and cross-environment replay. It is not a byte-identical copy of depot partition logs.

Inspect before loading

(ds/read-manifest "/var/tmp/my-depot-dump")

(def dump (ds/open "/var/tmp/my-depot-dump"))
(ds/num-partitions dump)
(take 5 (ds/partition-seq dump 0))   ; prefer small peeks; close streams if needed
(count (ds/ds-seq dump))             ; full realize — avoid on huge dumps

For large dumps, prefer into-depot! (streaming reduce) over realizing ds-seq into memory.

End-to-end example (copy between environments)

(require '[janus.rama.api.ds :as ds]
         '[com.rpl.rama :refer [open-cluster-manager-internal foreign-depot]])

(def stg
  (open-cluster-manager-internal {"conductor.host" "rama.staging.example"}))
(def prod
  (open-cluster-manager-internal {"conductor.host" "rama.prod.example"}))

(def module "io.example/MyModule")
(def src  (foreign-depot stg  module "*entities"))
(def dest (foreign-depot prod module "*entities"))

;; 1. Dump from staging (archive for size)
(ds/write-depot! src "/var/tmp/entities-stg" :profile :archive)

;; 2. Copy /var/tmp/entities-stg to a machine that can reach prod if needed

;; 3. Load into prod
(ds/into-depot! dest (ds/open "/var/tmp/entities-stg"))

Choosing options at a glance

Goal Suggested call

Full production dump for transfer

(write-depot! depot path) or :profile :archive

Small window for debugging

(write-depot! depot path :offset-back N :profile :archive)

Fixture committed to the repo

(write-depot! depot path :profile :fixture :offset-back N :max-bytes …)

Load dump into another depot

(into-depot! target (open path))

Faster load, accept less durability

(into-depot! target (open path) :ack-level nil)

See also

  • janus/rama — managed objects and Dataset API table

  • Component docs (architecture + ADRs): components/rama/doc/ (architecture/datasets.adoc, decisions.adoc)