Skip to content

Serialization 101

Welcome to Serialization 101. This course is a starting point for anyone who wants to understand data serialization. You do not need prior experience with distributed systems, data lakes, or binary formats. First-year students, data scientists, and working engineers can all begin here.

Jump
This track Three lenses below · then 201 mechanisms
See numbers Dashboard · language Overview for roster
How we measure Benchmarks

By the end of this theory track you should be able to:

  1. Explain what serialization is and why computer programs need it.
  2. Read the history of data formats as answers to real problems, not as a list of product names.
  3. Choose a format for a specific kind of work by using the right lens. One lens is data work. Another lens is services and systems.
  4. Connect the ideas in these pages to measured libraries in this multi-language benchmark suite.

Theory alone does not tell you what to ship in production. Use this course to build vocabulary and judgment. Then check real numbers on the Dashboard. Language Overview pages list the roster and caveats. For how those numbers are produced, see Benchmarks.

What is serialization?

Serialization is the process of turning an in-memory data structure into a linear sequence of bytes. Those bytes can be stored on disk, held in a cache, or sent across a network. Deserialization is the reverse process. It rebuilds a usable structure from those bytes. The rebuilt structure may live in another process, on another machine, or in another programming language.

Why is this necessary? Inside a running program, data is often a web of pointers and types. A record may point to an array. That array may point to strings. Those strings may point to characters. Networks and disks do not understand that web. They only store and transmit bytes in order. Every serialization format is therefore a contract between a writer and a reader. The contract says how the web of meaning is flattened into bytes. It also says how those bytes are rebuilt later.

Serialization as a contract: in memory, on the wire, rebuilt Serialization as a contract: in memory, on the wire, rebuilt

Three lenses

The same family of formats appears under three perspectives on purpose. Each document answers a different question:

Lens Primary question Best if you care about…
Historical Why do these formats exist? Eras, people, constraints, and major shifts in thinking
Data science What should I use for data and machine-learning work? Lakes, pipelines, notebooks, models, and columnar input/output
Engineering What should I ship in services and systems? APIs, remote procedure calls (RPC), performance, security, and long-term change

Suggested order for a first pass

  1. Skim the shared trade-offs below. This takes about ten minutes.
  2. Read the historical perspective once for the big picture.
  3. Deep-dive the lens that matches your work. Choose either data science or engineering.
  4. Open Serialization categories. Also open a language Overview (roster) and the Dashboard (numbers) for libraries you might actually use.
  5. When you need mechanisms (how formats work under the hood), work through the Serialization 201 track:
    1. Memory layout
    2. Encode/decode cost
    3. Self-describing vs schema
    4. Schema evolution
    5. Dynamic vs IDL binary
    6. Zero-copy
    7. Compression vs format

You can reverse steps 2 and 3 if you already have a concrete problem. For example, you might need Parquet for analytics, or an internal service format. Jump to a single 201 article when you already know the question you want answered.

When the mechanisms feel solid and you need production judgment under several constraints at once, continue to Serialization 301.

Core trade-offs

These axes appear in every lens. Learn the names here. The perspective documents fill in the details.

Text versus binary

Text (JSON, XML, YAML, and similar) Binary (MessagePack, Protocol Buffers, Parquet, and similar)
Strength Humans can read it. It is easier to debug and log. Compact. Often much faster to encode and decode.
Cost Larger payloads. Parsing character by character is slower. Opaque without tools. Harder to inspect by hand.

In other words, text formats trade size and speed for readability. Binary formats trade readability for density and often for speed.

Schema versus schemaless

A schema is a written description of the shape of the data. It says which fields exist, what types they have, and how they may change over time.

Schemaless (JSON, MessagePack, and similar) Schema-driven (Protocol Buffers, Avro, FlatBuffers, and similar)
Strength Flexible. You can ship data without an interface-description step. Compact on the wire. Supports code generation. Clearer evolution rules when you invest in process.
Cost Validation and compatibility are your job. Up-front schema design and tooling.

Row-oriented versus columnar

Row (JSON objects, Protocol Buffers messages, Avro records) Columnar (Parquet, ORC, Arrow tables)
Strength Natural for whole records. Fits APIs, RPC, and online transaction-style access. Scan a few columns over huge tables with far less input/output.
Cost Poor for wide analytical queries. Wrong default when you mostly fetch one document by id.

Think of a spreadsheet. A row-oriented format stores one complete row after another. A columnar format stores all values of column A together, then all values of column B, and so on. Analytics queries that touch only a few columns benefit from the columnar layout.

Self-describing versus schema-dependent

  • Self-describing (to varying degrees): Field names or type tags travel with the data. JSON, MessagePack, and CBOR are examples. These formats are easier to inspect. They also carry more metadata on the wire.
  • Schema-dependent: The wire data is nearly meaningless without a shared schema. Classic Protocol Buffers and raw Avro work this way. These formats are smaller and faster when both ends already agree on the contract.

Portable versus language-native

  • Portable: Designed for multi-language interchange. JSON, Protocol Buffers, MessagePack, and similar formats fit here.
  • Language-native: Tied to one runtime. Examples include pickle and Java serialization. These are convenient inside a tight trust boundary. They are dangerous or unusable across languages. They are also unsafe on untrusted inputs.

A trust boundary is any place where data leaves a fully controlled environment and may be influenced by someone else. A public network request is one example.

Lab notebooks (Python / Colab)

Hands-on companions for two of the lenses:

Notebook Article
Data science lab Data science perspective
Engineering mini lab Engineering perspective

Install and layout notes live in the notebooks README.

Scope and honesty

  • This theory track is a map, not an encyclopedia of every library.
  • Performance claims in prose are illustrative. Prefer the Dashboard for numbers on this benchmark runner and hardware.
  • “Best format” always means best under your constraints. Those constraints include team skills, trust boundaries, retention needs, latency budgets, and multi-language requirements.