Skip to content

Benchmark architecture

This page describes how the suite is built: folder layout, who the measurements are for, what is timed, and where configuration lives.

If you need… Go here
Statistics (warmup, outliers, confidence intervals) Analysis methodology
Payload shapes and size knobs Test data
Thread vs async vs families Queue categories
Tests and how to read a result Benchmark design
How to add another language Adding a language
How to add one library Adding a queue

One pipeline, four kinds of reader

Everyone uses the same measurement contract and the same analysis path.

Reader Primary question
Student or researcher Are the rankings inside one language trustworthy?
Library author Can I drop in my queue? Did it get better or worse?
System builder What fits our payload sizes and runtime?
Maintainer Can I add a language without rewriting analysis?

Repository layout

Path Role
config/benchmark_config.yaml Run modes, statistics defaults, language list, CSV schema
schemas/ Shared payload catalog
logs/<language>/ Timestamped result CSVs (gitignored)
analysis/ Python package that implements analyze-benchmarks
python/, c-sharp/, rust/, c/, javascript/ One benchmark runner per language
docs/ MkDocs site
scripts/run-all-benchmarks.sh Orchestrates multi-language runs

What we measure

A fair timing experiment separates preparation (once, untimed) from the loop (many times, timed).

  1. Prepare (not timed)
    Allocate payload bytes, construct the queue, spawn workers. Do not move items here. See Timing honesty.

  2. Timed loop (for each repetition i):

  3. enqueue every item → record enqueue time (TimeEnq)
  4. dequeue every item → record dequeue time (TimeDeq)
  5. wall-clock of the pair → handoff (TimeHandoff)

  6. Check fidelity (every item arrived, in order). Failures go to an errors file; a broken handoff is never a speed win.

  7. Warmup
    Index 0 is written to the CSV. Analysis drops it from averages.

CSV ABI

Runners write queue names. The parser still accepts leftover serializer-benchmark columns so historical logs keep loading.

Column Queue meaning
LibraryName / LibraryVersion Implementation + installed version
TimeEnq Enqueue ns
TimeDeq Dequeue ns
TimeHandoff Handoff ns
Pattern bytes = 1P1C, 4p4c = 4P4C (stream is leftover 2P2C, not I/O)
CpuTimeNs Process CPU time (spin vs block)

Patterns

  • 1P1C (CSV bytes): one producer, one consumer. Every library must implement this.
  • 4P4C (CSV 4p4c): four producers, four consumers. Libraries that cannot run that cell skip it rather than fake it.

The word stream is a leftover ABI name for 2P2C. The dashboard and docs say 1P1C / 4P4C. See Benchmark design.