Skip to content

Benchmarks

This section explains how this project measures serializers, not the theory of formats themselves. On the site it lives under the Benchmarks tab. Live interactive results live under the Dashboard tab.

If Learn / Serialization 101 answers “what is serialization?”, Benchmarks answers “how do we time it fairly, and what do the numbers mean?”

You do not need to be a performance engineer to read these pages. They are written for introductory computer science students and for anyone who wants a clear picture of the suite before diving into tables and plots.


Learning goals

After reading this section you should be able to:

  1. Say what a benchmark is measuring here (serialize and deserialize time, size, and correctness).
  2. Find the page for the five data sets, for modes, for statistics, for metrics, or for published numbers.
  3. Compare serializers within one language and one standard, on the same data set and data type, instead of treating absolute times as a global ranking.
  4. Open a language Overview (what libraries we measure, roster and caveats) under the Languages tab.
  5. Open the Dashboard for measured numbers and interactive comparisons.

For the ideas behind formats, start with Serialization 101 under Learn.


Data sets

Every published number is for one data set and one data type inside it. A library is measured only on the sets it can carry. There are five sets.

Data set Data types Shapes
Suite message, document, telemetry, strings, event Suite
Columnar table, table_project, nested_table, signal Columnar
Graph graph Graph
Array grid, grid_window Array
Tensor tensor, tensor_view, prompt, shard, request Tensor

Generator rules and the run-config table are on Data sets. The shapes are on each set’s page. The word itself is defined on Terminology.


The big picture in one paragraph

Each programming language has a small program called a benchmark runner. The benchmark runner builds sample data, runs each serializer many times, checks that the data comes back correctly, and writes a spreadsheet-like file (CSV). A shared analysis program then cleans those runs, computes summaries, and produces the tables and charts you see on the site. Everyone shares the same data shapes and the same analysis rules so comparisons stay fair.

  benchmark runner (C#, Python, Rust, …)
       │  timed serialize + deserialize
       ▼
  logs/<language>/*.csv
       │
       ▼
  analyze-benchmarks
       │
       └──► reports/stats_<lang>_latest.json   (Dashboard sync input)

Contribute

Task Guide
Add one library to an existing language Adding a serializer
Add a new language tree Adding a language

Terminology

Data set, data type, runner, and the prohibited words are defined on Terminology. Fixture means data type. Harness means benchmark runner.


How to read this section

The left sidebar is the reading order. Each page assumes you have read the ones above it.

  1. Read Terminology for the words, including data set and data type.
  2. Read Data sets for the shared rules, then the page for one set: Suite, Columnar, Graph, Array, or Tensor. The Array page also says which languages are measured on that set.
  3. Read Families for the teaching map, and for how a family differs from a data set.
  4. Read Architecture for how the repository times a library.
  5. Read Timing honesty for what prepare may do and what serialize must do.
  6. Read Modes for bytes versus stream, and for smoke through research.
  7. Read Methodology for warmup, filters, and uncertainty.
  8. Open Metrics when a column name is opaque.
  9. Read Claims and replication before you publish a claim.

Contributor checklists come after that path: Adding a serializer and Adding a language.

Page What you will learn Start here if you want…
Terminology Data set, data type, runner, and the words this site does not use A term is unclear, or an older page says “fixture”
Data sets The five sets, the shared knobs, and links to each set’s page What we serialize
Suite, Columnar, Graph, Array, Tensor The types in one set, and what each one stresses. Array also lists who is measured. One data set
Families Teaching map: JSON, schemaless binary, schema-driven, language-native, columnar Which standard a product question points at
Architecture How the repository is organized and how timing works The measurement design
Timing honesty What belongs in prepare vs the timed encode/decode “Is this row measuring a real encode?”
Modes I/O modes (bytes/stream) and run modes (smoke…research) Why the Dashboard Mode filter has two paths; which run preset to use
Methodology Warmup, outliers, confidence intervals, effect sizes, exploratory ranks How CSVs become published numbers
Metrics Names and meanings of every reported measurement “What does this column mean?”
Claims and replication L1 / L2 / L3: what you may claim from one run vs many Honest generalization language
Adding a serializer Drop-in checklist for one library Author path
Adding a language Checklist to plug in a new language benchmark runner Extending the suite
Dashboard Live measured numbers (Overview, Details, Compare) The published L1 artifact

Theory tracks: 101 · 201 · 301 · 401. Live charts: Dashboard.


Languages in this suite

Each language has a hand-written Overview (roster, caveats, how to read this runner). Measured numbers live on the Dashboard.

Language Serializers (registered) Overview Dashboard
C 21 Overview Dashboard
C# 56 Overview Dashboard
C++ 40 ‡ Overview Dashboard
Fortran 9 Overview Dashboard
Go 35 Overview Dashboard
Java 30 Overview Dashboard
JavaScript 34 † Overview Dashboard
Kotlin 34 Overview Dashboard
Mojo 21 Overview Dashboard
PHP 18 Overview Dashboard
Python 37 Overview Dashboard
Rust 33 Overview Dashboard
Swift 23 Overview Dashboard
Zig 17 Overview Dashboard

† In JavaScript, simdjson is optional (it needs a native addon). If that addon fails to build, the rest of the run still continues without it.

‡ C++ registers 40 on this machine when ARROW_ROOT points at Arrow 25.0.1. Without that prefix the five Arrow rows are skipped and sbe still registers (35 here; Boost.Serialization is not installed). Fory is included in both counts.

Log language ids (the Language column in CSVs) use short names such as csharp, python, rust, c, javascript, go, java, kotlin, php, cpp, swift, zig, mojo, and fortran. Documentation folders sometimes differ (for example docs/c-sharp/ for C#).

To refresh published numbers after a local run, pack Dashboard data with python3 dashboard/scripts/sync-data.py. Regeneration and claim levels: Claims and replication.


One rule for fair comparison

Prefer:

Same language + same standard + same data set + same data type

A chart that mixes standards is a number wearing a lab coat. It has not earned a ranking. Absolute times across languages (for example “Python vs C++”) mix runtimes, garbage collectors, and allocators. Those numbers can still be informative as a rough direction, but they are not a precise ranking. Details live under methodology limitations.