Data Science Perspective
Chapter 1 — A Chronicle of Frozen Data: The Complete History of Serialization
"The purpose of abstracting is not to be vague, but to create a new semantic level in which one can be absolutely precise." — Edsger W. Dijkstra
Preface: What Serialization Actually Is
Before we travel back in time, we need a precise definition to anchor the journey. Serialization is the process of translating a data structure or object state into a format that can be stored or transmitted — and later reconstructed — possibly in a different process, on a different machine, or by a program written in a completely different language. The reverse operation, reconstructing the original structure from the flat sequence of bytes, is called deserialization (or sometimes unmarshalling or decoding).
The word comes from the Latin serialis, meaning "arranged in a row" or "in sequence." This etymology is exact: a running program's memory is a rich, interconnected web of objects, pointers, and heap allocations. Serialization collapses that web into a flat, one-dimensional stream of bytes — something that can travel down a wire, be written to disk, or sit in a cache. The art of serialization is deciding how to perform that collapse, and the history of computing is largely a history of arguing about the best answer.
This chapter tells that story chronologically. You will meet the scientists, engineers, and researchers who shaped the field, the real problems they faced, the trade-offs they chose, and the new problems those choices created for the next generation. By the end, you will understand not just what each format does, but why it exists — which is the only understanding that actually helps you choose between them in practice.
Part I — The Physical Age: When Data Was Made of Holes (1950s–1960s)
It is 1952. A programmer at the IBM Thomas J. Watson Research Center in Yorktown Heights, New York, is working on an IBM 701 — one of the first commercially produced scientific computers. She is not thinking about serialization. She is thinking about punched cards.
The punched card, invented by Herman Hollerith in 1890 for the United States Census Bureau, is the world's first practical serialization format in computing history. Data is encoded as patterns of holes in a rigid 80-column card using a system called Hollerith code (later evolved into EBCDIC on IBM mainframes). A deck of cards is the serialized data. There is no separate encoding step; the physical arrangement of holes in the card is the binary representation. To read the data back, you feed the deck through a card reader that detects which holes are punched and maps them to characters or numerical values.
The first great lesson of serialization history is embedded right here, in this room full of magnetic card readers: every serialization format is born from a physical constraint. The punched card format was constrained by the mechanics of Hollerith's original tabulating machine, the size of a standard card (7⅜ × 3¼ inches), and the maximum number of columns a card reader could handle reliably. Data had to fit in 80 columns, period. That constraint was so deeply wired into the culture of computing that early video terminals defaulted to 80-character line widths — a legacy that persists in your terminal emulator today, more than seven decades after the constraint that created it disappeared.
As computers became more capable through the 1950s, programs needed to persist data between runs and share it between programs. The solution was immediate and obvious: write the raw bytes of in-memory data structures directly to magnetic tape or disk. John Backus and his team at IBM published the FORTRAN language specification in 1957, and FORTRAN's binary I/O was a direct window into memory:
! FORTRAN binary I/O (circa 1957): writing a fixed-size integer array to tape.
! Unit 1 is a magnetic tape drive. The WRITE statement dumps raw memory bytes.
! There is no encoding, no metadata, no schema — just the bits.
!
! A program on an IBM 704 (36-bit words, big-endian-ish) writes this tape.
! A UNIVAC 1103 tries to read it and gets garbage — different word size,
! different sign representation, different byte order.
INTEGER IDATA(100)
WRITE (1) IDATA
READ (1) IDATA
This approach was brutally fast and brutally simple. It was also brutally fragile: the bytes written by one machine were meaningless on any other machine. There was no format — just whatever the CPU's memory layout happened to be.
Grace Hopper, who designed COBOL (the Common Business-Oriented Language, standardized in 1960 after her earlier work on FLOW-MATIC), took a fundamentally different approach for business computing. COBOL's DATA DIVISION defined records with explicit field widths and types declared in a human-readable schema:
* COBOL DATA DIVISION (circa 1960): an explicit, schema-driven record layout.
* The schema fully describes the byte layout. Any program that knows this
* schema can read the data — on any machine, in any year.
* This is the world's first schema-defined serialization system in wide use.
01 EMPLOYEE-RECORD.
05 EMP-ID PIC 9(6). * 6-digit packed numeric
05 EMP-NAME PIC X(30). * 30-character alphanumeric string
05 EMP-SALARY PIC 9(8)V99. * Implied decimal: 8 digits + 2 cents
05 EMP-DEPT PIC X(4). * 4-character department code
This is the fixed-width record format, and it is one of the most durable ideas in the history of data engineering. The schema (the COBOL DATA DIVISION) fully and completely describes the byte layout of every field. Any program that knows the schema can read the data deterministically. Fixed-width records are still running today in banking mainframes, government systems, and payment processors, faithfully encoding billions of transactions that were designed before the internet existed.
The 1950s established three principles that would echo through every subsequent era.
First, a serialization format is always a contract between a writer and a reader, and that contract must account for the hardware it runs on. Second, explicit schemas dramatically improve interoperability, even crude ones like COBOL's fixed-width definitions. Third, and most important: the simpler the format, the harder it is to extend without breaking existing readers. Adding a new field to a COBOL fixed-width record means every existing program that reads that record must be updated or it will misinterpret every byte that follows the new field. This last principle is so fundamental that we will see it drive the design of every format in this chapter.
Part II — The Network Awakens: Byte-Order Wars and the First Standards (1970s–1980s)
By the mid-1970s, computing had fragmented into a zoo of incompatible architectures. Digital Equipment Corporation's PDP-11 stored multi-byte integers in little-endian order (least significant byte first). IBM's System/370 mainframes used big-endian order (most significant byte first). Motorola's 68000 — which would power the Apple Macintosh and most Unix workstations of the 1980s — was big-endian. Intel's 8086, destined to dominate personal computing for decades, was little-endian.
This became a crisis when ARPANET — the precursor to the internet — connected these machines together. Two computers could now talk, but a 32-bit integer sent by a PDP-11 would be decoded as a completely different number by an IBM mainframe. The integer 1 in little-endian is the bytes 01 00 00 00. Read in big-endian order, that's 16,777,216. The machines were speaking different dialects of the same language.
The intellectual shot across the bow came from Danny Cohen, a computer scientist at the Information Sciences Institute at the University of Southern California, in a now-legendary 1980 paper titled "On Holy Wars and a Plea for Peace" [1]. Cohen, borrowing from Jonathan Swift's Gulliver's Travels — where the kingdoms of Lilliput and Blefuscu waged war over which end of a boiled egg to crack first — named the two camps "big-endians" and "little-endians." His paper argued that the choice of byte order was arbitrary but that everyone needed to agree on one for network transmission. The argument won, and network byte order (big-endian) was enshrined in the standards that followed.
The second challenge of the 1970s-80s was floating-point numbers. Different vendors used different floating-point formats with different precisions, different exponent biases, and different representations for infinity and NaN. The IEEE 754 standard for floating-point arithmetic, published in 1985 after years of work led by William Kahan at Berkeley (who won the Turing Award partly for this work), finally gave the world a common representation. Every modern CPU, programming language, and serialization format that handles floats is built on IEEE 754 — it is another invisible foundation beneath everything we use.
XDR: The Pragmatist's Solution (1987)
Sun Microsystems was building the Network File System (NFS), which allowed Unix machines to share disk storage over a network. Their engineers needed a standard way to encode procedure arguments and return values for network transmission. Their answer was XDR (External Data Representation), published as RFC 1014 in 1987 [2] and later updated in RFC 4506 [3].
XDR is a binary format built around a clear philosophy: everything is aligned to 4-byte boundaries, all integers are signed 32-bit big-endian by default, strings and arrays include an explicit 4-byte length prefix, and padding zeroes bring each element up to the next 4-byte alignment. Here is what the core encoding looks like — the logic that NFS clients and servers were running on millions of workstations by the late 1980s:
/*
* XDR integer encoding (conceptual implementation, based on RFC 1014).
* XDR mandates big-endian (network byte order) regardless of host architecture.
* A Sun workstation (SPARC, big-endian) writes these bytes natively.
* A DEC VAX (little-endian) must byte-swap before writing — the XDR library
* handled this automatically, giving programmers a hardware-neutral API.
*/
#include <stdint.h>
void xdr_encode_int32(uint8_t *buf, int32_t value) {
/* Most-significant byte first — always, on every machine. */
buf[0] = (uint8_t)((value >> 24) & 0xFF);
buf[1] = (uint8_t)((value >> 16) & 0xFF);
buf[2] = (uint8_t)((value >> 8) & 0xFF);
buf[3] = (uint8_t)((value ) & 0xFF);
}
/* Strings use a 4-byte length prefix, followed by the bytes,
* followed by zero-padding to the next 4-byte boundary.
* This length-prefix approach is faster and safer than C's null terminator:
* you know exactly how many bytes to read before you start reading. */
void xdr_encode_string(uint8_t *buf, const char *s, uint32_t len) {
xdr_encode_int32(buf, len); /* 4-byte length */
memcpy(buf + 4, s, len); /* string bytes */
memset(buf + 4 + len, 0, (4 - len%4)%4); /* padding zeros */
}
The length-prefix pattern — write the count first, then the data — is so obviously better than null-terminated strings for a network protocol that you will see it in virtually every binary format invented after XDR. It is one of those ideas that, once you understand it, you cannot unsee.
XDR's fatal weakness was the same as every format in this era: it was schema-dependent without being self-describing. If you received an XDR byte stream, you could not know what was in it without the schema — an .x file written in XDR's own Interface Definition Language. The format and the schema were always separate documents, and keeping them synchronized was the developer's problem.
ASN.1: The Standards Committee's Answer (1984)
Meanwhile, the international telecommunications community — the CCITT (now ITU-T) and ISO — was solving the same problem from a different direction. Their answer was ASN.1 (Abstract Syntax Notation One), standardized as ISO 8824 in 1984 [4].
ASN.1 is simultaneously a notation language for defining data structures and a family of encoding rules that specify how those structures become bytes. The most important encoding rule, BER (Basic Encoding Rules) and its strict subset DER (Distinguished Encoding Rules), use a TLV (Type-Length-Value) structure: every piece of data is preceded by a tag byte that identifies its type, followed by a length field, followed by the value bytes. This makes ASN.1-encoded data partially self-describing — you can traverse a BER stream without the schema, even if you don't know the semantic meaning of what you find.
# Using pyasn1 (https://github.com/pyasn1/pyasn1) to illustrate
# the TLV structure that underlies X.509 certificates, SNMP, and LDAP.
from pyasn1.type import univ, namedtype
from pyasn1.codec.der import encoder, decoder
class Person(univ.Sequence):
"""An ASN.1 SEQUENCE — roughly equivalent to a struct or record."""
componentType = namedtype.NamedTypes(
namedtype.NamedType('name', univ.OctetString()),
namedtype.NamedType('age', univ.Integer())
)
person = Person()
person['name'] = b'Alice'
person['age'] = 30
encoded = encoder.encode(person)
# Wire format (hex): 30 0c <- SEQUENCE tag (0x30), 12 bytes long
# 04 05 <- OCTET STRING tag (0x04), 5 bytes long
# 41 6c 69 63 65 <- "Alice"
# 02 01 <- INTEGER tag (0x02), 1 byte long
# 1e <- 30 in decimal
# The TLV structure lets a reader skip unknown tag types gracefully.
# This "ignore unknown fields" property is exactly what allows ASN.1
# protocols to be extended without breaking existing parsers.
decoded, _ = decoder.decode(encoded, asn1Spec=Person())
assert str(decoded['name']) == 'Alice'
ASN.1's TLV forward-compatibility trick — unknown tags are ignorable — was visionary. It would be reinvented, consciously or not, in Protocol Buffers (as field numbers), in CBOR (as major types), and in dozens of proprietary protocols over the following four decades. It is the solution to the extension problem that the COBOL fixed-width record could never provide.
ASN.1 remains deeply embedded in modern infrastructure. Every time your browser establishes an HTTPS connection, it parses an X.509 certificate encoded in DER. Every SNMP packet, LDAP query, and SSH certificate uses ASN.1. It is one of the most durable serialization formats ever created.
The Formal Insight: Remote Procedure Calls (1984)
The 1980s also produced a pivotal academic contribution that formalized an idea that XDR and ASN.1 were groping toward. Andrew Birrell and Bruce Nelson at Xerox PARC published "Implementing Remote Procedure Calls" in 1984 [5], making the case that network communication should feel like a local function call — arguments go in, results come out, and the serialization should be generated automatically from an interface description rather than written by hand. Their RPC system used a stub compiler that took an interface description and generated the encode/decode code for both the client and the server.
This idea — define the interface once, generate the serialization code — is the DNA of every modern RPC framework. You will see it in CORBA's IDL compiler, in Protocol Buffers' protoc, in Apache Thrift's compiler, and in gRPC today. It was the right answer in 1984, and it is still the right answer in 2025.
The OSI model, formalized by ISO in the early 1980s, enshrined this insight architecturally: Layer 6, the Presentation Layer, is explicitly about serialization and encoding — translating between the application's internal representation and the network's canonical format. The fact that a separate layer in the canonical network model exists purely for serialization tells you how fundamental the problem is.
Part III — The Object-Oriented Revolution and Its Serialization Problem (Late 1980s–Mid 1990s)
The late 1980s brought a revolution in how programmers thought about programs: object-oriented programming. Objects had state (fields), behavior (methods), and — crucially — they pointed to other objects, forming graphs with cycles and shared references. This created a serialization problem that flat records, XDR streams, and ASN.1 structures had never been designed for.
CORBA and the IDL Approach (1991)
The Object Management Group's CORBA (Common Object Request Broker Architecture), released in 1991, was the enterprise industry's attempt to solve distributed objects. CORBA used a language-neutral Interface Definition Language (IDL), and a compiler generated stub code in C++, Java, Python, COBOL, or any other supported language. The wire format was IIOP (Internet Inter-ORB Protocol), which encoded data in CDR (Common Data Representation) — a binary encoding close in spirit to XDR.
The CORBA approach was intellectually coherent and technically powerful. It was also famously, almost comically, complex to configure and deploy. Configuring an Object Request Broker often took more effort than writing the actual application. When the web arrived and demanded lightweight, loosely coupled communication, CORBA's weight made it unsuitable. The ORB market declined quickly after 2000. But CORBA's IDL concept — describe the interface in a neutral language, generate code in many languages — proved immortal, reborn in every subsequent RPC system.
Java Serialization: Magic with Hidden Costs (1995)
When James Gosling and the Java team at Sun Microsystems released Java in 1995, they built serialization directly into the language runtime. Any class that implemented java.io.Serializable could be automatically serialized using reflection — no schema file, no code generator, no boilerplate. The JVM would discover the class's fields at runtime and write them to a stream.
import java.io.*;
// Java's built-in serialization: implement one marker interface,
// and the JVM handles everything automatically via reflection.
public class User implements Serializable {
// serialVersionUID acts as a version stamp for the class.
// If the class changes and this UID doesn't match what was serialized,
// deserialization throws InvalidClassException — a brittle versioning mechanism.
private static final long serialVersionUID = 1L;
private String name;
private int age;
public User(String name, int age) {
this.name = name;
this.age = age;
}
}
// Serializing to a file:
try (ObjectOutputStream oos =
new ObjectOutputStream(new FileOutputStream("user.bin"))) {
oos.writeObject(new User("Alice", 30));
// The stream starts with a magic header (0xACED 0x0005),
// then writes the fully-qualified class name, the serialVersionUID,
// and then each field's type and value. All of this overhead means
// even a simple two-field object produces dozens of bytes of metadata.
}
// Deserializing from a file:
try (ObjectInputStream ois =
new ObjectInputStream(new FileInputStream("user.bin"))) {
User user = (User) ois.readObject();
}
Java serialization was remarkably ergonomic for 1995. But it embedded three problems that would haunt the Java ecosystem for the next three decades.
The language lock-in problem. The format was Java-only. A Python service could not deserialize a Java-serialized object. This tightly coupled the data format to the implementation language — a design choice that becomes increasingly painful as systems grow polyglot.
The brittleness problem. Adding a new field to User, or changing an existing field's type, would cause old serialized data to fail deserialization if the serialVersionUID changed. Versioning distributed data formats is one of the genuinely hard problems in distributed systems, and Java serialization's approach — a manual version stamp that developers must update and interpret correctly — is an easy one to get wrong.
The security disaster. Java deserialization executes code paths inside the JVM as it reconstructs objects. A maliciously crafted byte stream can invoke arbitrary constructors and methods during deserialization, leading to remote code execution with no other vulnerability required. Security researcher Alvaro Muñoz and Chris Frohoff documented devastating gadget chain attacks in 2015 that affected virtually every major Java application server. Java's own documentation now explicitly recommends against using built-in serialization for any data that crosses a trust boundary. The Apache Commons Collections library's deserialization gadget chain became one of the most widespread critical vulnerabilities in enterprise computing history.
Python's Pickle: A Powerful, Sharp Knife (1995)
Python, created by Guido van Rossum and first released in 1991, developed the pickle module around 1995 as part of the Python 1.4 release. Pickle takes an even more ambitious approach than Java serialization: it can serialize almost any Python object — functions, classes, lambdas, closures, NumPy arrays — by emitting a sequence of opcodes for a simple stack-based virtual machine. The VM then replays those opcodes during deserialization to reconstruct the object.
import pickle
# pickle can serialize almost anything in Python.
# For production ML code, this is extremely convenient —
# model weights, preprocessing pipelines, and hyperparameter objects
# can all be saved and loaded in one line.
data = {
"model_name": "LogisticRegression",
"weights": [0.23, -0.71, 1.42, 0.09],
"metadata": {"trained_at": "2024-01-15", "accuracy": 0.94}
}
# protocol=5 (Python 3.8+) is the fastest; it supports out-of-band buffers
# for zero-copy transfer of large binary objects like NumPy arrays.
serialized = pickle.dumps(data, protocol=5)
# Deserializing: this line executes arbitrary Python code.
# NEVER call pickle.loads() on data from an untrusted source.
# An attacker can craft a pickle payload that runs any command on your server.
restored = pickle.loads(serialized)
assert restored == data
# The cloudpickle library (https://github.com/cloudpipe/cloudpickle)
# extends pickle to handle even more Python objects, including
# locally-defined functions and lambdas, making it the standard for
# serializing PySpark and Dask distributed computation tasks.
Pickle is still the workhorse of Python's data science and ML ecosystem. PyTorch's model checkpointing uses pickle. scikit-learn's joblib uses a pickle-compatible format. Ray and Dask use cloudpickle for shipping computation closures across cluster nodes. But like Java serialization, it inherits the same trio of weaknesses: Python-only, brittle under schema evolution, and deeply insecure with untrusted input.
By the mid-1990s, the serialization landscape was a fracture map. Low-level systems used XDR or ASN.1. Mainframes used COBOL fixed-width records. Java programs used Java serialization. Python programs used pickle. C programs used hand-rolled binary formats. None of these worlds communicated with each other without custom translation code. Something had to change.
Part IV — The XML Decade: A Universal Language (1996–2005)
The World Wide Web — invented by Tim Berners-Lee at CERN and popularized after the release of Mosaic in 1993 — changed the problem entirely. Suddenly, millions of computers on different operating systems, written in different languages, by developers who had never met, needed to exchange data. The internet needed a serialization format that was universal: human-readable, self-describing, extensible, and language-neutral.
The answer the industry converged on was XML (Extensible Markup Language). The XML working group at the W3C was chaired by Jon Bosak of Sun Microsystems; Tim Bray and Jean Paoli were the lead editors of the specification, published as a W3C Recommendation in February 1998 [6]. XML was derived from SGML (Standard Generalized Markup Language, ISO 8879:1986) but radically simplified for web use. James Clark, one of the most prolific XML tool authors, built the reference parser expat and the XSLT processor XT, which bootstrapped the entire XML tooling ecosystem.
XML is a text-based, hierarchical, self-describing format. Every piece of data is wrapped in named tags; the structure is a tree. Every XML document declares its character encoding. Any text editor can open it. Any developer can read it without special tools.
<?xml version="1.0" encoding="UTF-8"?>
<!-- XML is verbose: every field name appears twice — in opening and closing tags.
But that verbosity is also its strength: the data is completely self-describing.
A human reading this bytes stream understands what it means. -->
<user>
<name>Alice</name>
<age>30</age>
<scores>
<score>95</score>
<score>87</score>
<score>92</score>
</scores>
<metadata>
<active>true</active>
<role>admin</role>
</metadata>
</user>
XML's ecosystem of standards was impressive: XSD (XML Schema Definition) for schema validation, XSLT for data transformation, XPath for querying, XQuery for complex queries, and XML Namespaces for extensibility without naming conflicts. These standards made XML attractive for enterprise integration, document formats (DocBook, OOXML), and configuration files.
SOAP: XML on the Wire (1998–2003)
The enterprise software industry immediately saw XML as the basis for a universal RPC layer. SOAP (Simple Object Access Protocol) was designed by Dave Winer, Don Box, and a team at Microsoft, first published in 1998 [7]. SOAP encoded RPC calls as XML, wrapped in an envelope/header/body structure, transmitted over HTTP. Any language on any platform that could speak HTTP and parse XML could participate.
<!-- A SOAP request: an XML-encoded remote procedure call.
The actual payload — userId=12345 — is buried under four layers
of XML structure. This overhead ratio worsened as services grew. -->
<soapenv:Envelope
xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"
xmlns:usr="http://example.com/user">
<soapenv:Header/>
<soapenv:Body>
<usr:GetUser>
<usr:userId>12345</usr:userId>
</usr:GetUser>
</soapenv:Body>
</soapenv:Envelope>
Microsoft built SOAP into .NET's System.Web.Services. IBM built it into WebSphere. The WS-* family of standards grew to encompass security (WS-Security), reliability (WS-ReliableMessaging), transactions (WS-AtomicTransaction), and dozens of others. By the early 2000s, SOAP-based web services were the canonical way for enterprise systems to communicate, and the Web Services Description Language (WSDL) was the schema system that glued it all together.
But the cracks were showing. A simple "get a user by ID" request required dozens of lines of XML boilerplate. Parsing XML was computationally expensive — a DOM parser had to load the entire document into memory as a tree before any data could be accessed. SAX (Simple API for XML) parsers streamed the document but were painful to program correctly. The WS-* specification family grew so complex that it spawned an entire industry of Enterprise Service Buses and application servers just to manage the complexity.
The XML decade proved a clear and painful theorem: human-readability and self-description carry real costs. XML documents are verbose — the tag overhead often dwarfs the actual data. Parsing XML is CPU-intensive. The specification ecosystem became so Byzantine that it required specialists just to understand it. Something simpler was long overdue.
Part V — JSON and the RESTful Rebellion (2001–2010)
It was around 2001, and Douglas Crockford — a veteran programmer working at State Software in California — was staring at JavaScript's object literal syntax and recognizing something important. JavaScript objects, written as {"name": "Alice", "age": 30}, were already a perfectly useful data format. They were human-readable, hierarchical, and parseable by every web browser's built-in JavaScript interpreter, with no additional libraries required.
Crockford coined the name JSON (JavaScript Object Notation), registered the domain json.org, and began advocating for it as a lightweight alternative to XML for web data exchange. He later recalled that he didn't invent JSON so much as discover it — the format was already latent in the JavaScript language specification, and he simply pointed at it, named it, and wrote a one-page grammar.
JSON was documented in RFC 4627 in 2006 [8], later updated in RFC 7159 [9] and RFC 8259 [10], and standardized as ECMA-404. Its type system is deliberately minimal: null, boolean, number, string, array, and object (a map with string keys). That minimalism is both its greatest strength and its most important limitation — a limitation that would drive several subsequent innovations.
import json
data = {
"name": "Alice",
"age": 30,
"scores": [95, 87, 92],
"active": True
}
# Encoding: Python's True becomes JSON's true (lowercase).
# Python's None becomes JSON's null.
# Python's int and float both become JSON's number — with no distinction.
json_str = json.dumps(data)
# Result: '{"name": "Alice", "age": 30, "scores": [95, 87, 92], "active": true}'
# Decoding: JSON's number always becomes Python float or int based on
# presence of a decimal point — a subtle source of type-mismatch bugs.
restored = json.loads(json_str)
# The type system gaps become painful in practice:
# - No date type: encode as ISO 8601 string? Unix timestamp integer? Milliseconds?
# Every team makes their own choice. APIs become incompatible.
# - No binary type: must base64-encode raw bytes, adding 33% overhead.
# - No integer/float distinction: JSON's "30" and "30.0" are both valid
# representations of the same number. Receivers must guess which type was intended.
# - No schema: the receiver cannot know if "age" will be present or what range it has.
JSON's adoption was supercharged by Roy Fielding's influential 2000 doctoral dissertation, which articulated REST (Representational State Transfer) as an architectural style for web APIs [11]. REST APIs returned JSON over plain HTTP, and every browser could consume it with a single JSON.parse() call. By 2010, JSON had largely displaced XML for new APIs. By 2015, it was the undisputed lingua franca of the web.
The JSON library ecosystem began to proliferate as performance concerns emerged. Python's standard library json module is written in pure Python with a C accelerator, but for high-throughput services it became a bottleneck. ujson (UltraJSON) was released around 2012 by ESN Social Software, wrapping a fast C implementation. orjson (2019, by the developer known as ijl) rewrote the serializer in Rust, claiming 2–3× speedups over ujson. rapidjson (a Python wrapper around Tencent's C++ RapidJSON library) offered yet another performance profile. The benchmark wars between these libraries became a minor industry of their own — and a significant source of conflicting claims that motivate the benchmarking work described in the next chapter of this course.
JSON's type system gap was not merely a cosmetic complaint. For a startup building its first REST API, it is barely noticeable. For a team building a financial system where a timestamp's timezone matters, or a scientific system where floating-point precision matters, or a binary protocol where raw bytes need to be transferred efficiently, the gaps become active hazards. The industry's answer to these gaps came in two waves: a binary renaissance, and a validation renaissance. Let's examine the first.
Part VI — The Binary Renaissance: Speed, Schemas, and Efficiency Return (2007–2013)
It is 2004, and Google's internal systems are making millions of remote procedure calls per second. Google engineers had been encoding and decoding ad-hoc text formats — custom pipe-delimited strings, manually assembled data blobs — for years. The maintenance burden was significant: changing a data format required updating every service that read or wrote it, and there was no tooling to help. Jeff Dean and Sanjay Ghemawat — the same pair who designed GFS and MapReduce, and who would later win the Turing Award in 2023 — were among the engineers who felt this pain most acutely.
Protocol Buffers: The IDL Idea, Modernized (2001 internally, 2008 open-sourced)
The solution Google developed internally, starting around 2001, was Protocol Buffers (protobuf). The system was designed by several engineers, with Kenton Varda doing substantial architecture work and leading the open-source release in 2008 [12]. Protocol Buffers revived the IDL approach of CORBA and XDR but applied it with modern engineering discipline: a clean schema language, a code generator that targeted multiple languages, and a binary encoding optimized for both compactness and parse speed.
You define your data in a .proto file, and the protoc compiler generates code in Python, Java, C++, Go, Kotlin, Ruby, or any other supported language:
// user.proto — the schema for a User message.
// This file is the single source of truth. Run:
// protoc --python_out=. user.proto
// to generate user_pb2.py with a fully-featured Python class.
syntax = "proto3";
message User {
string name = 1; // Field number 1 identifies "name" on the wire.
int32 age = 2; // Field number 2. Field names do NOT appear in
repeated int32 scores = 3; // the binary encoding — only field numbers do.
} // This is both a compactness win and the key to
// forward compatibility: unknown field numbers
// are simply ignored by older parsers.
The generated Python code:
# After: protoc --python_out=. user.proto
from user_pb2 import User
user = User(name="Alice", age=30, scores=[95, 87, 92])
# SerializeToString produces compact binary — no field names, no delimiters,
# just field-number-tagged values packed as tightly as possible.
data = user.SerializeToString()
# Round-trip:
restored = User()
restored.ParseFromString(data)
assert restored.name == "Alice"
The key innovation in protobuf's binary encoding is the varint (variable-length integer). Instead of always using 4 or 8 bytes for every integer, varints use as many bytes as the value requires — 7 bits of data per byte, with the most significant bit serving as a continuation flag. The following is adapted from the Python protobuf implementation (python/google/protobuf/internal/encoder.py):
def encode_varint(value: int) -> bytes:
"""
Encode a non-negative integer as a Protocol Buffers varint.
Each output byte contributes 7 bits of data. The most significant bit
(the continuation bit) signals whether more bytes follow:
0 = this is the last byte
1 = more bytes to come
Examples:
encode_varint(1) -> b'\\x01' (1 byte: 0_0000001)
encode_varint(128) -> b'\\x80\\x01' (2 bytes: 1_0000000, 0_0000001)
encode_varint(300) -> b'\\xac\\x02' (2 bytes: 1_0101100, 0_0000010)
The practical impact: a field whose value is 0–127 takes ONE byte.
For the vast majority of real-world integers (small IDs, ages, status codes),
this is a dramatic improvement over a fixed 4-byte encoding.
"""
result = []
while value > 0x7F:
result.append((value & 0x7F) | 0x80) # 7 bits + continuation flag
value >>= 7
result.append(value) # Last byte, no continuation flag
return bytes(result)
Protocol Buffers also solved the schema evolution problem cleanly. Field numbers never change, and unknown field numbers are ignored. If you add a new field (say, field number 4, email) to the User message, old parsers simply skip it. If you remove a field, old serialized data that contains it is still parseable — the new parser just ignores the removed field's bytes. The only constraint is that once you assign a field number, you must never reuse it for a different field — doing so would corrupt data silently. Field numbers can be marked reserved in the schema to prevent accidental reuse.
This was a profound improvement over Java serialization's brittle serialVersionUID and COBOL's rigid fixed-width layouts. For the first time, a binary format with machine-generated serialization code was also safely evolvable.
Facebook's Thrift: The Competing Vision (2007)
Facebook solved the same problem independently. In April 2007, Mark Slee, Aditya Agarwal, and Marc Kwiatkowski published a white paper describing Thrift [13], a system that combined an IDL, a multi-language code generator, and a pluggable protocol layer. Thrift's key differentiator was flexibility: it supported multiple binary protocols (Binary, Compact, and JSON) over multiple transports (sockets, HTTP, in-memory pipes). This made it more flexible than Protocol Buffers for teams with heterogeneous infrastructure. Facebook open-sourced it in April 2007 and later donated it to the Apache Software Foundation, where it remains widely used today in large polyglot service meshes.
MessagePack: JSON's Binary Twin (2008)
Not every team wanted a schema compiler. Many developers wanted something that felt like JSON — the same flexible data model, no schema required, easy ad-hoc use — but in a compact binary encoding. Sadayuki Furuhashi, a Japanese software engineer, designed MessagePack to fill this gap, releasing the first specification and reference implementation around 2008.
MessagePack maps JSON's six types directly to binary type codes, encodes small integers in a single byte (a positive integer 0–127 is just itself, with no overhead), and uses length prefixes for strings, arrays, and maps.
# MessagePack in Python (https://github.com/msgpack/msgpack-python)
import msgpack
import json
data = {
"name": "Alice",
"age": 30,
"scores": [95, 87, 92],
"active": True
}
packed = msgpack.packb(data, use_bin_type=True)
json_bytes = json.dumps(data).encode('utf-8')
print(f"JSON size: {len(json_bytes)} bytes") # ~57 bytes
print(f"MessagePack size: {len(packed)} bytes") # ~37 bytes (~35% smaller)
# MessagePack is typically 20–40% smaller than JSON for typical payloads,
# and 3–10× faster to encode/decode in benchmarks, because there is no
# string-to-number conversion, no delimiter scanning, and no escape processing.
restored = msgpack.unpackb(packed, raw=False)
assert restored == data
MessagePack's "JSON but faster and smaller" positioning made it enormously popular for caching layers (Redis supports it natively via modules), message queues, and inter-service communication where developers want the flexibility of a schemaless format without JSON's parse overhead.
BSON: MongoDB's Document Format (2009)
The MongoDB team developed BSON (Binary JSON) in 2009 to serve simultaneously as a wire format and a storage format for their document database. BSON extends JSON's type system with practically important additions: an explicit Date type (64-bit milliseconds since Unix epoch, avoiding the timestamp string ambiguity that plagues JSON APIs), a Binary type for raw bytes, and MongoDB-specific types like ObjectId. Crucially, BSON stores the total document length at the very beginning, allowing a database engine to skip entire documents without parsing them — a critical property for range scans and index lookups.
By the early 2010s, the binary-versus-text choice had crystallized into a clear, if not always correctly applied, heuristic: if you control both ends of the communication, are performance-sensitive, and can afford a build step, use a binary format with a schema (protobuf, Thrift). If you need maximum flexibility and human-readability, use JSON. If you want a middle ground without a schema, use MessagePack.
Part VII — Big Data, Schema Evolution, and the Columnar Revolution (2009–2015)
In 2003 and 2004, Google published two papers that would reshape distributed computing. "The Google File System" [14] described how Google stored petabytes of data across thousands of commodity servers. "MapReduce: Simplified Data Processing on Large Clusters" [15] described how Google processed that data at scale. Doug Cutting, who had been building an open-source web crawler at Yahoo! called Nutch, used these papers as blueprints to create Apache Hadoop — making large-scale distributed computing available to anyone who could afford a cluster of commodity servers.
Hadoop introduced a new class of serialization requirements. In a distributed batch job, data is written once, stored for months or years, and read by many different programs running at different times — some of which may be newer or older versions of each other. Schema evolution — the ability to add, remove, or rename fields over time without breaking readers of old data — became a first-class engineering concern, not an afterthought.
Apache Avro: Schemas Without Field Numbers (2009)
Doug Cutting designed Apache Avro in 2009 as Hadoop's native serialization solution. Avro made a choice that distinguished it sharply from Protocol Buffers and Thrift: the wire format contains only values — no field names, no field tags, no type codes. The schema must accompany the data, either embedded in the file header or stored in an external schema registry. This produces the most compact possible binary encoding (no overhead per field beyond the value itself), but it means schema management is a first-class operational concern.
Avro's schema is defined in plain JSON — no new IDL to learn:
# Apache Avro in Python (https://github.com/apache/avro/tree/master/lang/py)
import json
import avro.schema
from avro.datafile import DataFileWriter, DataFileReader
from avro.io import DatumWriter, DatumReader
# The schema is JSON. Adding a field with a "default" value enables
# backward compatibility: new readers can process old data (the default fills in).
# Forward compatibility: old readers skip fields they don't know.
schema_json = json.dumps({
"type": "record",
"name": "User",
"fields": [
{"name": "name", "type": "string"},
{"name": "age", "type": "int"},
{"name": "scores", "type": {"type": "array", "items": "int"}},
# New field with a default: old data lacking this field gets True automatically.
{"name": "active", "type": "boolean", "default": True}
]
})
schema = avro.schema.parse(schema_json)
# Writing: the schema is embedded in the Avro container file header.
with open("users.avro", "wb") as f:
writer = DataFileWriter(f, DatumWriter(), schema)
writer.append({"name": "Alice", "age": 30, "scores": [95, 87, 92], "active": True})
writer.close()
# Reading: the schema is recovered from the file header automatically.
# Avro's "schema resolution" lets the reader's schema differ from the writer's:
# fields present in writer but not in reader are skipped;
# fields present in reader but not in writer are filled with defaults.
with open("users.avro", "rb") as f:
for record in DataFileReader(f, DatumReader()):
print(record)
Avro's schema resolution mechanism — where reader and writer schemas can differ, with explicit compatibility rules and default values — became the gold standard for schema evolution in event-driven systems. Apache Kafka adopted Avro (with the Confluent Schema Registry) as its recommended serialization format, and it dominates event-driven architectures where data persists for years.
The Dremel Paper: Nested Data, Columnar Storage (2010)
In 2010, a team of Google engineers — Sergey Melnik, Andrey Gubarev, Jing Jing Long, Geoffrey Romer, Shiva Shivakumar, Matt Tolton, and Theo Vassilakis — published "Dremel: Interactive Analysis of Web-Scale Datasets" [16], describing Google's system for querying petabytes of nested data interactively. This paper introduced an observation that seems obvious in retrospect but required real ingenuity to act on: when analysts run queries, they typically access only a few columns from tables with hundreds of columns.
If data is stored row-by-row (as in every format we have examined so far), reading one column requires loading every row from disk. If the same data is stored column-by-column, reading one column requires loading only that column's data. For analytical workloads on wide tables, this difference can be two orders of magnitude in I/O.
The challenge Dremel solved was representing nested records (objects with sub-objects and repeated fields) in a flat columnar layout without losing structural information. Their solution used two small integers per value — a repetition level (which repeated field in the path repeated to produce this value?) and a definition level (how many optional fields in the path are actually defined?) — that together fully encode the nesting structure with no redundancy. This is a beautiful piece of algorithmic design.
Apache Parquet: Open-Source Dremel (2013)
Julien Le Dem (then at Twitter) and Nong Li (then at Cloudera) implemented the Dremel paper's ideas as Apache Parquet, announcing the project in 2013. Parquet is a columnar storage format for analytical systems. It combines the Dremel encoding with column-level compression (where similar values in the same column compress dramatically better than mixed values in rows), predicate pushdown (reading only the pages that could contain rows satisfying a filter, without decoding the rest), and dictionary encoding (representing repeated values as small integer codes).
# Apache Parquet with PyArrow (https://github.com/apache/arrow/tree/main/python)
import pyarrow as pa
import pyarrow.parquet as pq
# Create a typed, columnar table.
table = pa.table({
"name": pa.array(["Alice", "Bob", "Carol", "Dave"], type=pa.string()),
"age": pa.array([30, 25, 35, 28], type=pa.int32()),
"dept": pa.array(["Eng", "Eng", "Sales", "Eng"], type=pa.string()),
"salary": pa.array([120000, 95000, 110000, 105000], type=pa.int64()),
})
# Write as Parquet with Snappy compression (lz4 and zstd are also common).
pq.write_table(table, "employees.parquet", compression="snappy")
# Read back only the columns needed for a specific query.
# Only "name" and "salary" data is read from disk — "dept" and "age" are skipped.
# For a table with 500 columns and a 2-column query, this is ~250× less I/O.
result = pq.read_table("employees.parquet", columns=["name", "salary"])
# Parquet is now the standard format for data lakes (S3, GCS),
# Spark pipelines, and analytical databases like DuckDB and Athena.
The Parquet ecosystem — combined with cloud object stores — fundamentally changed how companies store and analyze data. The "data lake" architecture, where raw data is stored as Parquet files in S3 or GCS and queried by engines like Spark, Athena, or DuckDB, is the dominant large-scale analytical architecture of the 2020s.
Martin Kleppmann's Designing Data-Intensive Applications (2017) [20] provides the most thorough treatment of these formats in the context of real systems, and is essential reading alongside this chapter.
CBOR: Binary JSON for the Internet of Things (2013)
Not everyone was working at Google or Cloudera scale. The Internet of Things community needed a binary format that was small (for constrained microcontrollers with kilobytes of RAM), self-describing (no schema required, like JSON), and semantically compatible with JSON's data model. Carsten Bormann of the University of Bremen and Paul Hoffman designed CBOR (Concise Binary Object Representation), published as RFC 7049 in 2013 [17] and updated in RFC 8949 in 2020 [18].
CBOR uses the same TLV structure that ASN.1's BER pioneered in 1984, but maps it directly onto JSON's type system. A one-byte integer fits in one byte. A short string adds one byte of overhead. The format is completely self-describing — you can parse any CBOR stream without a schema. CBOR is now an IETF standard deployed in COSE (the cryptographic envelope used in WebAuthn hardware security keys), CTAP2 (the protocol used when your YubiKey talks to your browser), and dozens of IoT protocols.
FlatBuffers: Zero-Copy Deserialization (2014)
Wouter van Oortmerssen, a programming language researcher at Google, designed FlatBuffers in 2014 for Google's game development toolchain (specifically for Android games). FlatBuffers addressed a limitation that Protocol Buffers had not: even after decoding, protobuf data lives in separate heap-allocated objects that the garbage collector must track. FlatBuffers encodes data in a memory layout that matches how it will be accessed — so deserialization is essentially pointer arithmetic, not object construction.
# FlatBuffers in Python (https://github.com/google/flatbuffers)
# After generating Python code from a schema with the flatc compiler:
import flatbuffers
from MyGame.Sample import Monster # Generated code
builder = flatbuffers.Builder(256)
# Strings and vectors must be created before the table that contains them.
name = builder.CreateString("Orc")
Monster.MonsterStart(builder)
Monster.MonsterAddName(builder, name)
Monster.MonsterAddHp(builder, 300)
monster_offset = Monster.MonsterEnd(builder)
builder.Finish(monster_offset)
buf = bytes(builder.Output())
# Reading requires zero allocation and zero copying.
# We are reading directly from the buffer — no parsing, no heap objects.
monster = Monster.Monster.GetRootAsMonster(buf, 0)
assert monster.Name().decode('utf-8') == "Orc"
assert monster.Hp() == 300
FlatBuffers is used in Google's internal tooling, in several major game engines, and in systems where even a microsecond of deserialization latency is unacceptable. Cap'n Proto, created by Kenton Varda (who had previously led Protocol Buffers at Google) in 2013, takes the same zero-copy idea even further — Cap'n Proto messages can be used directly from the wire representation with no transformation step at all, and the format supports streaming and capabilities (a form of object reference) that FlatBuffers does not.
Part VIII — The Validation Renaissance: Schema as Code (2015–Present)
By 2015, the web API landscape had settled into a rough equilibrium. JSON was the lingua franca of public REST APIs. Binary formats (protobuf, MessagePack) dominated internal service communication at large companies. Columnar formats (Parquet) were standard in data engineering pipelines. Yet a persistent, grinding problem irritated every developer working with JSON APIs: how do you know the data you receive is what you expect?
Runtime type errors from malformed API responses — a field that should be an integer arriving as null, a required field missing entirely, a date string in an unexpected format — were a significant source of production bugs. Teams were writing defensive validation code by hand, which was verbose, inconsistent, and rarely comprehensive. The industry needed a better answer.
JSON Schema: Contracts for JSON (2013–Present)
JSON Schema, first proposed around 2009 and maturing through successive drafts in the 2010s, addressed the validation gap by defining a JSON-based language for describing and validating JSON data. A JSON Schema document is itself valid JSON, and it specifies types, required fields, value constraints, string patterns, array lengths, and more:
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"type": "object",
"properties": {
"name": {
"type": "string",
"minLength": 1,
"maxLength": 200
},
"age": {
"type": "integer",
"minimum": 0,
"maximum": 150
},
"scores": {
"type": "array",
"items": { "type": "integer", "minimum": 0, "maximum": 100 }
}
},
"required": ["name", "age"],
"additionalProperties": false
}
JSON Schema powers OpenAPI (formerly Swagger), the dominant standard for documenting REST APIs. The OpenAPI ecosystem includes code generators that produce validated client SDKs, server stubs, and API documentation from a single schema document — a modern echo of the CORBA IDL compiler from Part III, but for JSON over HTTP. This convergence is not coincidental: the code-generation approach is simply the right answer to the schema problem, and it keeps being rediscovered.
Pydantic: The Python Type System as Schema Language (2017)
The Python community's most influential answer to the validation problem was Pydantic, created by Samuel Colvin in 2017. Pydantic made a key insight: Python's type hints — introduced in Python 3.5 via PEP 484 — are already a schema language. Instead of writing a separate JSON Schema document, why not use the type annotations you were going to write anyway?
# Pydantic (https://github.com/pydantic/pydantic)
from pydantic import BaseModel, Field, ValidationError
from typing import List, Optional
from datetime import datetime
class Score(BaseModel):
value: int = Field(ge=0, le=100, description="Score from 0 to 100")
subject: str
class User(BaseModel):
name: str
age: int = Field(ge=0, le=150)
scores: List[Score] = []
active: bool = True
created_at: Optional[datetime] = None # datetime is natively supported
# Validation happens at construction time — invalid data raises immediately.
try:
bad_user = User(name="Bob", age=200) # age exceeds 150
except ValidationError as e:
print(e)
# 1 validation error for User
# age: Input should be less than or equal to 150 [type=less_than_equal, ...]
# Valid construction:
user = User(name="Alice", age=30, scores=[Score(value=95, subject="Math")])
# Serialize to JSON with type mappings handled automatically:
print(user.model_dump_json())
# '{"name":"Alice","age":30,"scores":[{"value":95,"subject":"Math"}],...}'
# Deserialize from JSON with full validation:
restored = User.model_validate_json('{"name":"Bob","age":25}')
Pydantic v2, released in 2023, rewrote the core validation engine in Rust via the pydantic-core library, achieving roughly a 5–50× speedup over Pydantic v1 while maintaining API compatibility. It is now one of the most downloaded packages on PyPI, forming the validation foundation of FastAPI (one of the fastest-growing Python web frameworks), and is embedded in tools across the Python ecosystem. Colvin's insight — that Python's type annotations are already a schema language, and should be treated as such — proved profoundly right.
msgspec: When Speed and Correctness Must Both Win (2021)
For use cases where even Pydantic's performance was insufficient — high-frequency trading systems, machine learning inference servers, real-time analytics — Jim Crist-Harif released msgspec in 2021. msgspec combines schema definition (via Python's typing module), validation, serialization, and deserialization into a single highly optimized C extension. Its central insight is that knowing the schema in advance eliminates an entire class of runtime work: a schema-aware decoder doesn't need to ask "is this a string or a number?" for every field — it knows in advance, and can take the fast path immediately.
# msgspec (https://github.com/jcrist/msgspec)
import msgspec
from typing import List, Optional
class Score(msgspec.Struct):
value: int
subject: str
class User(msgspec.Struct):
"""
msgspec.Struct is like a faster, frozen dataclass combined with
a serializer and validator. The struct layout is fixed at class
creation time, enabling the C extension to generate highly optimized
encode/decode paths with no dynamic dispatch.
"""
name: str
age: int
scores: List[Score] = []
active: bool = True
# Create encoder/decoder objects once and reuse them.
# This amortizes per-call setup overhead across many serialization operations.
json_enc = msgspec.json.Encoder()
json_dec = msgspec.json.Decoder(User)
user = User(name="Alice", age=30, scores=[Score(value=95, subject="Math")])
encoded = json_enc.encode(user) # b'{"name":"Alice","age":30,...}'
restored = json_dec.decode(encoded) # Validates AND deserializes in one pass
# msgspec also supports MessagePack with the same Struct definition.
# You write the schema once; msgspec handles both JSON and binary formats.
mp_enc = msgspec.msgpack.Encoder()
mp_dec = msgspec.msgpack.Decoder(User)
compact = mp_enc.encode(user) # Binary, ~30% smaller than JSON equivalent
According to msgspec's own benchmarks (which we will examine critically and independently in the next chapter), it can be 5–10× faster than Pydantic v1 for schema-validated JSON serialization and 2–3× faster than orjson for schema-validated encoding. The performance gap closes significantly when Pydantic v2 is the comparison, but msgspec retains a measurable edge in the tightest loops.
Apache Arrow: The Format That Eliminates Serialization (2016)
Wes McKinney — the creator of pandas, the most widely used data manipulation library in Python — spent years frustrated by a specific problem. When a Python process needed to share a pandas DataFrame with another process (a Spark worker, a database connector, an R statistical model), it had to serialize the data to some format, transmit it, and deserialize it on the other end. For large DataFrames — hundreds of millions of rows — this serialization overhead dominated total processing time.
In 2016, McKinney co-founded the Apache Arrow project to build a standardized in-memory columnar format [19]. Arrow's insight is radical in its simplicity: if every system that handles columnar data agrees on the same in-memory representation — same byte layout, same type encoding, same null bitmap convention — then passing data between systems requires no encoding or decoding at all. Just share a pointer.
# Apache Arrow (https://github.com/apache/arrow/tree/main/python)
import pyarrow as pa
import pyarrow.ipc as ipc
import io
# Create an Arrow table in memory.
table = pa.table({
"user_id": pa.array([1001, 1002, 1003], type=pa.int64()),
"name": pa.array(["Alice", "Bob", "Carol"], type=pa.string()),
"score": pa.array([0.92, 0.87, 0.95], type=pa.float64()),
})
# To transfer between processes, use the IPC (Inter-Process Communication) format.
# But the key insight: if both processes share memory (via mmap or shared memory),
# the table can be passed with ZERO COPYING and ZERO SERIALIZATION.
sink = io.BytesIO()
with ipc.new_stream(sink, table.schema) as writer:
writer.write_table(table)
arrow_bytes = sink.getvalue()
# Deserialize (in a different process, or on the other end of a socket):
reader = ipc.open_stream(io.BytesIO(arrow_bytes))
restored = reader.read_all()
# The radical case: when pandas 2.0, DuckDB, Polars, and Spark all use Arrow
# as their internal memory format, data can flow between them without any
# serialization at all — just a pointer handoff.
Arrow has become the connective tissue of the modern data ecosystem. DuckDB, Polars, pandas 2.0, Apache Spark's shuffle layer, and dozens of other tools use Arrow as their internal format. The project embodies a new philosophy: the best serialization is the serialization you never have to do.
Part IX — A Map of the Territory: Understanding Where You Stand
Seven decades of invention have produced a rich, overlapping landscape of serialization formats. Before we move to the benchmarks in the next chapter, it is worth drawing a clear conceptual map — not to memorize, but to understand the axes along which formats differ.
The first and most fundamental axis is text versus binary. Text formats — XML, JSON, YAML, TOML — are human-readable and debuggable with any text editor. They are self-describing in the sense that field names appear in the data itself. They pay a real price in space (field names are repeated for every record) and parse speed (every byte must be scanned to find string boundaries and delimiters). Binary formats — XDR, Protocol Buffers, MessagePack, Avro, Parquet — are compact and fast but opaque without tooling. The choice between them is rarely purely technical: debuggability, team tooling habits, and operational practices matter as much as raw throughput.
The second axis is schema-first versus schemaless. Schema-first formats (Protocol Buffers, Avro, Thrift, Parquet) require you to define your data structure before you can serialize it. This imposes upfront work but provides forward-compatibility guarantees, code generation, static analysis, and auto-generated documentation. Schemaless formats (JSON, MessagePack, BSON, pickle) let you serialize arbitrary data immediately, leaving validation, versioning, and documentation as your ongoing responsibility. The validation renaissance of the 2010s — JSON Schema, Pydantic, msgspec — represents the ecosystem's attempt to bring schema discipline to schemaless formats without sacrificing their flexibility.
The third axis is row-oriented versus columnar. Row-oriented formats (JSON, Protobuf messages, Avro records) write all fields of a record together. They are optimal for transactional workloads where you read and write complete records one at a time. Columnar formats (Parquet, Arrow, ORC) group all values of the same field together across many records. They are optimal for analytical workloads where you query a subset of columns across millions or billions of rows.
The fourth axis is self-describing versus schema-dependent. ASN.1/BER and CBOR embed type tags in every value, allowing any byte stream to be traversed without an external schema. JSON is structurally self-describing but semantically schema-dependent (you can read it without a schema, but you cannot know if a number is a timestamp or a score without one). Protocol Buffers and raw Avro require the schema to interpret the data at all.
Format Taxonomy as of 2025
─────────────────────────────────────────────────────────────────────────
│ Text │ Binary
───────────────────┼────────────────────┼───────────────────────────────
Schemaless │ JSON, YAML │ MessagePack, BSON, CBOR
Schema-first │ XML + XSD, SOAP │ Protobuf, Thrift, Avro
Columnar/analytics │ (rare) │ Parquet, ORC, Arrow
Zero-copy/in-proc │ — │ FlatBuffers, Cap'n Proto
Language-native │ — │ pickle, Java serialization
Validation layer │ JSON Schema │ Schema registries (Confluent)
─────────────────────────────────────────────────────────────────────────
None of these axes determines the right answer for you — your constraints do. A startup building its first REST API should almost certainly use JSON without a schema: the flexibility and tooling ecosystem are worth the trade-offs. A team building an event-driven architecture with years of data retention should use Avro with a schema registry: safe schema evolution is worth the operational investment. A team building a high-throughput internal RPC layer should use Protocol Buffers or MessagePack with msgspec validation: the performance difference at scale justifies the complexity. A data engineering team building analytical pipelines should use Parquet for storage and Arrow for in-process computation: the I/O savings and zero-copy benefits are decisive at volume.
The benchmarks in the next chapter will give you quantitative data for the Python ecosystem specifically. But numbers are only useful when you understand the qualitative history behind them — the trade-offs that led each format's designers to their decisions, and the problems those decisions solved and created in turn. That history is what you have just read.
References
[1] Cohen, D. (1980). "On Holy Wars and a Plea for Peace." IEEE Computer, 14(10), 48–54.
[2] Sun Microsystems. (1987). RFC 1014: XDR: External Data Representation Standard. IETF. https://www.rfc-editor.org/rfc/rfc1014
[3] Eisler, M. (Ed.). (2006). RFC 4506: XDR: External Data Representation Standard. IETF. https://www.rfc-editor.org/rfc/rfc4506
[4] ITU-T. (2021). Recommendation X.680: Abstract Syntax Notation One (ASN.1): Specification of Basic Notation. International Telecommunication Union. https://www.itu.int/rec/T-REC-X.680
[5] Birrell, A. D., & Nelson, B. J. (1984). "Implementing Remote Procedure Calls." ACM Transactions on Computer Systems, 2(1), 39–59. https://doi.org/10.1145/2080.357392
[6] Bray, T., Paoli, J., Sperberg-McQueen, C. M., Maler, E., & Yergeau, F. (2008). Extensible Markup Language (XML) 1.0 (Fifth Edition). W3C Recommendation. https://www.w3.org/TR/xml/
[7] Box, D., Ehnebuske, D., Kakivaya, G., Layman, A., Mendelsohn, N., Nielsen, H. F., Thatte, S., & Winer, D. (2000). Simple Object Access Protocol (SOAP) 1.1. W3C Note. https://www.w3.org/TR/2000/NOTE-SOAP-20000508/
[8] Crockford, D. (2006). RFC 4627: The application/json Media Type for JavaScript Object Notation (JSON). IETF. https://www.rfc-editor.org/rfc/rfc4627
[9] Bray, T. (Ed.). (2014). RFC 7159: The JavaScript Object Notation (JSON) Data Interchange Format. IETF. https://www.rfc-editor.org/rfc/rfc7159
[10] Bray, T. (Ed.). (2017). RFC 8259: The JavaScript Object Notation (JSON) Data Interchange Format. IETF. https://www.rfc-editor.org/rfc/rfc8259
[11] Fielding, R. T. (2000). Architectural Styles and the Design of Network-Based Software Architectures (Doctoral dissertation, University of California, Irvine). https://www.ics.uci.edu/~fielding/pubs/dissertation/top.htm
[12] Varda, K. (2008). Protocol Buffers: Google's Data Interchange Format. Google Open Source Blog. https://opensource.googleblog.com/2008/07/protocol-buffers-googles-data.html
[13] Slee, M., Agarwal, A., & Kwiatkowski, M. (2007). Thrift: Scalable Cross-Language Services Implementation. Facebook White Paper. https://thrift.apache.org/static/files/thrift-20070401.pdf
[14] Ghemawat, S., Gobioff, H., & Leung, S.-T. (2003). "The Google File System." Proceedings of the 19th ACM Symposium on Operating Systems Principles (SOSP '03), 29–43. https://doi.org/10.1145/945445.945450
[15] Dean, J., & Ghemawat, S. (2004). "MapReduce: Simplified Data Processing on Large Clusters." Proceedings of the 6th Symposium on Operating Systems Design and Implementation (OSDI '04), 137–150.
[16] Melnik, S., Gubarev, A., Long, J. J., Romer, G., Shivakumar, S., Tolton, M., & Vassilakis, T. (2010). "Dremel: Interactive Analysis of Web-Scale Datasets." Proceedings of the VLDB Endowment, 3(1–2), 330–339. https://doi.org/10.14778/1920841.1920886
[17] Bormann, C., & Hoffman, P. (2013). RFC 7049: Concise Binary Object Representation (CBOR). IETF. https://www.rfc-editor.org/rfc/rfc7049
[18] Bormann, C., & Hoffman, P. (2020). RFC 8949: Concise Binary Object Representation (CBOR). IETF. https://www.rfc-editor.org/rfc/rfc8949
[19] Apache Software Foundation. (2016). Apache Arrow: A Cross-Language Development Platform for In-Memory Data. https://arrow.apache.org
[20] Kleppmann, M. (2017). Designing Data-Intensive Applications: The Big Ideas Behind Reliable, Scalable, and Maintainable Systems. O'Reilly Media. ISBN 978-1-4493-7332-0.
[21] Abadi, D., Boncz, P., Harizopoulos, S., Idreos, S., & Madden, S. (2013). "The Design and Implementation of Modern Column-Oriented Database Systems." Foundations and Trends in Databases, 5(3), 197–280. https://doi.org/10.1561/1900000024
End of Chapter 1. Chapter 2 proceeds to the benchmark suite design: datasets, fairness constraints, measurement methodology, and results across the Python serializer ecosystem.