Skip to main content

Archiving

DataRetentionTtl (see Cluster Configuration) purges finalized jobs, terminated recurring schedules, and logs once they age past the TTL. By default that purge is a hard delete. If you want that data kept longer term — for audit, compliance, or historical reporting — without bloating the cluster that actually processes jobs, point the purge at a dedicated archive cluster instead.

Enabling archiving​

Pass a targetArchivedClusterId as the second argument to DataRetentionTtl on the source cluster:

config.DataRetentionTtl(TimeSpan.FromDays(30), targetArchivedClusterId: "payroll-archive");

Once a job or recurring schedule ages past the TTL, it's moved to the target cluster instead of being deleted outright.

What gets archived​

Archiving a job carries its full execution history along with it, not just the job row itself:

  • The job's own row (same Id, same final status).
  • Every JobExecution record for that job — one per attempt, with start time, outcome, and any error message.
  • Its structured logs, but only the JobExecution-category ones (the ones tied to a specific attempt, e.g. a failure or a timeout). Every other log category (Job, Bucket, Cluster, AgentWorker, RecurringSchedule, Api) is always hard-deleted once it ages past DataRetentionTtl, regardless of whether archiving is configured — those are operational noise, not something worth preserving long-term.

This means you can look up a job on the archive cluster after the fact and still see exactly what happened on each attempt, not just its final outcome.

Setting up the archive cluster​

The target must be a dedicated cluster, configured with:

  • Mode(ClusterMode.Archived)
  • Only Coordinator-mode workers (or Full, which behaves identically here — no agent connection needed either way)
  • No agent connections at all
// Source cluster — processes jobs normally
builder.Services.AddJobMasterCluster("payroll-cluster", config =>
{
    config.UsePostgresForMaster(masterConnectionString);
    config.DataRetentionTtl(TimeSpan.FromDays(30), targetArchivedClusterId: "payroll-archive");
    config.AddAgentConnectionConfig("agent-1").UsePostgresForAgent(agentConnectionString);
    config.AddWorker();
});

// Archive cluster — receives finalized data only, never processes anything itself
builder.Services.AddJobMasterCluster("payroll-archive", config =>
{
    config.UsePostgresForMaster(archiveConnectionString);
    config.Mode(ClusterMode.Archived);
    config.AddWorker(); // Coordinator mode only — no AddAgentConnectionConfig
});

All of this is enforced at startup:

  • The archive cluster can't have buckets, agent connections, or any worker mode other than Coordinator/Full.
  • targetArchivedClusterId can't point at the cluster itself.
  • It must resolve to another configured cluster, and that cluster must actually be Mode(ClusterMode.Archived).
  • An archive cluster refuses to start if it already contains any job or recurring schedule in a non-final status — it may only ever hold finalized data, arriving exclusively through the archive path.

Cascading tiers​

An archive cluster can set its own targetArchivedClusterId, pointing at a further archive cluster — so retention can cascade through multiple tiers, e.g. a 30-day hot archive that itself archives into a 365-day cold archive.

If the archive cluster is unreachable​

If the target can't be reached when a purge runs, the data is deleted directly instead of being lost in limbo — a Critical log is written so this doesn't go unnoticed, since data is being deleted rather than archived.

See: Cluster Configuration · Migrating a Cluster