Skip to main content

Benchmark Methodology

How the JobMaster Benchmark Results page is set up, so the numbers on that page can be read (and challenged) with full context.

Concurrency settings​

JobMaster's per-worker execution concurrency comes from its bucket model: each bucket gets a fixed base capacity per priority level, multiplied by ParallelismFactor. Every JobMaster worker in this round runs 2 Medium-priority buckets at ParallelismFactor 2, across 20 worker containers, so every engine tested has the same total in-flight job capacity throughout the entire test.

Scope: same server for master and agent connections​

Every config in this comparison points the master connection and every agent connection at the same database server (for example, one PostgreSQL server for both). JobMaster also supports distributing the transport/execution side onto its own separate server entirely -- for example, a RavenDB master paired with NATS as the agent-side transport, running on its own dedicated hardware. That's a real, supported topology, not a workaround, but splitting work across two or more servers instead of one isn't a fair apples-to-apples comparison against everything else on this page running single-server. See the RavenDB + NATS page for that variant, documented in its own section for exactly that reason.

Resource limits​

Every container, database and worker alike, has an explicit CPU/memory cap, not the host's full uncapped resources, so every framework and database engine competes for a fixed budget instead of however much the host happens to have free:

  • Worker container: 2 CPU / 4 GB, every config.

  • Database container: 6 CPU / 16 GB for MySQL and PostgreSQL. RavenDB's database container ran at 3 CPU / 6 GB instead -- half the CPU and a third of the memory of the other two. This isn't a benchmark choice: RavenDB's free Community Edition caps a node at 3 CPU cores, so 3 CPU is the most RavenDB can use here without a commercial license. It's called out with a footnote wherever RavenDB appears on the comparison page and its own detail page, since it means RavenDB's numbers aren't a strict apples-to-apples resource comparison against the other two this round.

info

Database CPU% in the detail pages can read above 100 -- the cap is measured in whole CPUs (600% in single-core-percentage terms for a 6-CPU cap, 300% for RavenDB's 3-CPU cap), not a single core.

Test hardware​

Every config in this comparison runs on its own dedicated AWS EC2 instance, not a shared host, so no config's numbers are affected by another config's load:

  • RavenDB: m6id.12xlarge (48 vCPU / 192 GB RAM).

  • PostgreSQL, MySQL: m6id.16xlarge (64 vCPU / 256 GB RAM).

Both sizes use m6id (Ice Lake CPUs + local NVMe SSD instance-store, mounted as Docker's own data root instead of the default EBS-backed root volume) for faster container image builds and database I/O.

Burst-mode test shape​

Each run fires a fixed number of jobs (50,000) all at once, split across all worker containers in parallel: 60 concurrent POST /schedule-now calls firing simultaneously, rather than a steady arrival rate. This isolates two distinct things instead of conflating them into one number:

  • Scheduling throughput: how fast the system accepts and durably persists the whole burst, measured by the slowest of the concurrent schedule calls to return (all of them start at once, so the phase ends when the last one finishes).

  • Execution throughput: how fast, once accepted, jobs are actually picked up and completed by workers, measured from run start to the last real completion, a separate and usually much longer phase than scheduling itself.

Scope: SQL Server not included​

This round also tested JobMaster and Hangfire against SQL Server, but those results aren't published on the comparison page. SQL Server's license terms require Microsoft's prior written approval before publishing benchmark results, which wasn't obtained before this round -- see the note on JobMaster Benchmark Results for details.

Repetition​

Every result on this page and its detail pages is the average of 3 repeated runs, not a single run -- single-run comparisons for this kind of test carry more noise than a single number suggests. No result on this site is reported without repetition.