Historical Perspective

This page is the “big picture” lens of Serialization 101. You do not need to memorize every product name. The goal is to see a simple pattern: pressure → response. In each era, people felt certain constraints most strongly. The formats of that era were answers to those constraints.

The problem that never goes away

Programs hold rich structure in memory. Nested records, arrays, graphs of objects, different integer widths, and different byte orders all live there. Disks, networks, and many caches only store linear sequences of bytes.

Serialization is the durable answer to a simple question with hard consequences. How do we flatten meaning into bytes and recover it later? The reader may be another machine, another language, or a program years in the future.

Every major format is a bet on which constraints matter most. Human readability is one bet. Portability across CPU architectures is another. Schema evolution means changing the data layout without breaking old readers. Processor cost, memory pressure, and analytics input/output are further bets. History is the story of those bets.

Era map

The table below is a map of the course of history. You do not need to memorize every name. Learn the pressure → response pattern. Ask what was expensive or broken, and what idea fixed it.

Years	Era	Dominant pressure	Representative answers
1950s–1960s	Physical & fixed records	Media limits; batch business data	Punched cards; COBOL fixed-width records; raw memory dumps
1970s–1980s	Network portability	Heterogeneous machines on one network	Network byte order; XDR; ASN.1
Late 1980s–1990s	Distributed objects	Object graphs; “call a method elsewhere”	CORBA IDL; Java / Python native serialization
Mid-1990s–early 2000s	Universal documents	Web-scale multi-vendor interchange	XML; SOAP stack
2000s–2010s	Lightweight web data	Browser & API simplicity	JSON; REST-style APIs
Mid-2000s onward	Efficient services & storage	Datacenter cost; long-lived data	Protobuf, Thrift; MessagePack/BSON/CBOR; Avro
2010s onward	Analytics & zero-copy	Scan huge tables; avoid copy/GC	Parquet/ORC; Arrow; FlatBuffers; Cap’n Proto
~2015 onward	Validation as product	Correctness of dynamic JSON at scale	JSON Schema; typed validators (e.g. Pydantic, msgspec)

1950s–1960s: Physical media and fixed contracts

Early “serialization” was often the medium itself. Herman Hollerith designed punched cards that encoded values as hole patterns in fixed columns. That technology was used in the 1890 census and remained central mid-century. The physical layout was the format. If you knew which columns held a person’s age or occupation, you could “read” the card.

When magnetic tape and disk arrived, two durable ideas competed:

Raw memory image — write the bytes exactly as the CPU lays them out. This was common in FORTRAN-era binary I/O. This is fast on one machine. It is useless or wrong on another machine with a different word size or endianness. Endianness is the byte order of multi-byte values.
Explicit record layout — declare field widths and types once. COBOL DATA DIVISION fixed-width records are an example. Any program that shares the layout can read the bytes. This is interoperable but rigid. If you insert a field, every reader must change or mis-parse the rest of the record.

Lesson still true today: a format is a contract between writer and reader. The simpler and more positional the layout, the harder it is to extend without breaking old readers. That tension still shows up in modern APIs and event schemas.

1970s–1980s: Networks force canonical forms

Connecting incompatible architectures made ad-hoc binary dumps a liability. The PDP-11 was little-endian. IBM machines were big-endian. Later Intel and Motorola worlds differed as well. A multi-byte integer that means 1 on one host can mean millions on another if the byte order differs.

Danny Cohen wrote a 1980 essay called On Holy Wars and a Plea for Peace. It popularized “endian” as a network problem. It argued for a single network byte order (big-endian) for interchange. The choice of big-endian was somewhat arbitrary. Agreement mattered more than which side “won.”

Three threads answered “structured data on the wire” and how to call remote code:

1984: ASN.1

Telecom and ISO standards defined ASN.1. The design was formalized in the mid-1980s. ASN.1 describes abstract types plus encoding rules. Notable rules include BER and DER. Type–Length–Value (TLV) encoding lets a parser skip unknown pieces. That is an early, powerful approach to extensibility. A new field can be added without forcing every old reader to fail. DER-encoded structures still sit under HTTPS certificates (X.509) and other infrastructure you use every day.

1984: RPC as an idea

Birrell & Nelson worked on remote procedure calls. Their work formalized a pattern that never left the industry. The idea is simple to state. Describe the interface once. Generate the code that packs and unpacks arguments. Make a network call feel like a local function call. CORBA, protoc, Thrift, and gRPC are descendants of that DNA.

1987: XDR

Sun’s XDR (External Data Representation) powered NFS and Sun RPC. It is described in RFC 1014 and later RFC 4506. It used fixed alignment rules, big-endian integers, and length-prefixed strings and arrays. It was portable and efficient for its time. It was not self-describing. You needed the agreed procedure and types to interpret the stream. Those were often defined in an .x description file.

Lesson: networks demand a canonical representation that every machine can agree on. That pressure still pushes industry toward an interface description language (IDL) and generated encode/decode code.

Late 1980s–1990s: Objects and native graphs

Object-oriented runtimes introduced graphs. Shared references, cycles, and inheritance all appear. Flat records and simple RPC structs were not enough for “save this object and restore it later in the same ecosystem.”

CORBA (1991) — language-neutral IDL and binary CDR on the wire. It was powerful but operationally heavy. It declined as the web favored looser coupling.
Java serialization (1995) — language work associated with James Gosling and the Java platform. You implement a marker interface. The runtime reflects fields. This is ergonomic inside the JVM. It is not portable to other languages. Versioning via serialVersionUID is brittle. It is unsafe on untrusted bytes. Gadget chains can lead to remote code execution.
Python was created by Guido van Rossum. pickle arrived in the mid-1990s. It is an opcode stream for a small virtual machine. It can reconstruct rich Python objects. With cloudpickle, it can reconstruct many dynamic callables. It is central to much scientific Python. It is Python-only and unsafe on untrusted input.

Lesson: language-native formats maximize convenience inside one trust boundary and one language. They repeatedly fail as universal interchange. They are a poor security boundary when input is untrusted.

Mid-1990s–2000s: The XML decade

The public web needed something language-neutral, hierarchical, and human-inspectable. XML 1.0 was published by the W3C in 1998. Editors included Tim Bray and Jean Paoli. XML has roots in SGML. It wrapped data in named tags. Tooling exploded. XSD, XPath, XSLT, and namespaces all appeared.

Enterprise systems layered SOAP and the WS-* stack on XML over HTTP. In theory this was universal. In practice it was verbose and complex. Parse cost and document weight became obvious at scale.

Lesson: self-description and universality have real processor and bandwidth costs. The industry spent the following decades trying to keep interoperability while reducing that “XML tax.”

2000s–: JSON and the web API default

Douglas Crockford named and popularized JSON in the early 2000s. JSON is essentially JavaScript object literal syntax used as a data format. It was later standardized as RFC 4627, then RFC 8259 / ECMA-404. JSON has a small set of types: null, bool, number, string, array, and object. It is trivial for browsers. It is good enough for most public APIs.

REST-style HTTP APIs made JSON the default public interchange language. REST is an architectural style articulated by Roy Fielding. Mobile clients reinforced that default. Limitations became part of everyday engineering:

There is no standard date or binary type. People use conventions and base64.
Numbers are not a full IEEE 754 taxonomy of int versus float.
Schema is optional. Later work filled that gap with JSON Schema, OpenAPI, and language validators.

Lesson: the “winning” format is often the one that minimizes integration friction for many teams and tools. It is not always the one that wins a microbenchmark on one machine.

2000s–2010s: Schema-driven binary efficiency

Inside large service meshes, repeating field names as text and parsing characters became a measurable datacenter tax. Companies paid real money in CPU time and bandwidth for formats that were convenient but heavy.

Early 2000s: Protocol Buffers

Google’s Protocol Buffers began internal use in the early 2000s and were open-sourced in 2008. Design lineage includes work associated with Jeff Dean, Sanjay Ghemawat, Kenton Varda, and many others. Protocol Buffers revived IDL plus code generation with a compact binary encoding. Field numbers replace field names on the wire. Variable-length integers (varints) keep small numbers small. Evolution rules center on never reusing field numbers for different meanings. The bytes are opaque without the .proto schema. The format is excellent for multi-language services that invest in schema discipline.

2007: Apache Thrift

Facebook’s Thrift (later Apache) combined an IDL, multi-language code generation, and pluggable protocols and transports. It offered RPC-oriented flexibility in environments that used many programming languages.

~2008–2013: Schemaless binary cousins

Not every team wanted a compiler in the loop. Some wanted a binary form of the JSON data model:

Format	Approx.	Intent
MessagePack	~2008 (Sadayuki Furuhashi)	JSON data model, binary tags, smaller/faster than text JSON
BSON	~2009 (MongoDB)	Document storage/wire types (dates, binary) with length prefixes
CBOR	2013 (RFC 7049 / 8949)	Standards-track concise binary objects; strong IoT / constrained-device story

Lesson: once JSON locked a simple data model in people’s heads, the industry cloned that model into binary for speed and size. It reintroduced schemas where long-lived evolution and efficiency dominated.

Late 2000s–2010s: Long-lived data and analytics

Batch and streaming platforms needed formats that survive years of readers and writers coexisting. The Hadoop ecosystem and its successors are examples.

Apache Avro was designed by Doug Cutting and others around 2009. It stores compact binary values. The schema often travels with the data or lives in a registry. It has a strong story for matching a writer’s schema to a reader’s schema by rules (schema resolution). Defaults and reader/writer schema compatibility are first-class ideas. It became a default mental model for event logs. Kafka plus registry patterns is a common example.
Columnar storage — Google’s Dremel paper (2010) popularized nested columnar layout for analytic queries that touch few columns of wide tables. Apache Parquet was open-sourced around 2013 by Julien Le Dem, Nong Li, and community. ORC made the same idea the backbone of data lakes.
Apache Arrow began around 2016. Wes McKinney and co-founders and community led the project. It defines a standard in-memory columnar layout. Systems can share tables with minimal or zero copy instead of endlessly converting.

Lesson: transactional messaging and analytical scanning want different layouts. History gradually splits “messages on the wire” from “tables on disk or in memory for analytics.”

2010s: Zero-copy access

Even fast encode and decode still copy data into language objects. Domains with tight latency or memory budgets pushed further. Games, some telemetry, and certain RPC paths are examples.

Cap’n Proto (Kenton Varda, ~2013) — layout designed so the buffer is the in-memory form. Encode and decode can approach a no-op for simple access patterns.
FlatBuffers (Wouter van Oortmerssen, Google, ~2014) — similar zero-copy access goals with vtable-based optional fields. It has strong mobile and game heritage. It also appears in ML runtime ecosystems.

Trade-off theme: less parse work often means more care around validation, how you mutate data, and day-to-day tooling. Debuggability, proxies, and HTTP-centric infrastructure all change.

Mid-2010s–present: Validation renaissance

Public and internal APIs stayed on JSON for reach. Teams paid for ad-hoc validation. The response was not a single new wire format. It was schema as a product layer:

JSON Schema and OpenAPI — contracts and generated clients and servers for HTTP JSON.
Runtime validators bound to language types — in Python, notably Pydantic and high-performance tools like msgspec. These treat annotations as schema and validate on the way in and out.

Lesson: the popularity of schemaless JSON created demand for optional, enforceable structure. Contracts and validators can fill that demand without always switching the bytes on the wire.

Tensions diagram

History does not converge on a single winner. It accumulates niches along recurring trade-offs:

One pole	↔ tension ↔	Other pole
Human-readable & universal (JSON / XML)		Compact & CPU-cheap (Protobuf / MessagePack / …)
Flexible & ad hoc (JSON / pickle / MessagePack)		Evolvable & explicit (Avro / Protobuf + process)
Whole-record access (messages / documents)		Wide-table analytics (Parquet / Arrow)
Safe across trust boundaries (portable + validated)		Max power in one runtime (native pickle / Java serialization)

Using this history

When someone says “just use X,” ask which pressure they are optimizing. Debugging, multi-language support, long retention, scan speed, and safety under untrusted input are different goals.
Prefer portable, explicit formats when data crosses a language boundary or a trust boundary.
Expect multiple formats in one healthy organization. JSON at the public edge, binary remote procedure calls inside, and Parquet in the data lake is historically normal. It is not a sign of failure.

Selected references

These are entry points, not an exhaustive bibliography. Wikipedia articles for major formats appear on first mention above.

Cohen, D. (1980). “On Holy Wars and a Plea for Peace.” IEEE Computer.
Sun Microsystems / IETF. XDR — RFC 1014; RFC 4506.
ITU-T / ISO. ASN.1 (e.g. X.680 and related encoding rules).
Birrell, A. D., & Nelson, B. J. (1984). “Implementing Remote Procedure Calls.” ACM TOCS.
Bray, T., et al. Extensible Markup Language (XML) 1.0. W3C.
Bray, T. (Ed.). The JavaScript Object Notation (JSON) Data Interchange Format — RFC 8259.
Fielding, R. T. (2000). Architectural Styles and the Design of Network-Based Software Architectures (REST).
Google Protocol Buffers documentation and open-source history.
Slee, M., Agarwal, A., & Kwiatkowski, M. (2007). Thrift white paper; Apache Thrift project.
Apache Avro, Parquet, and Arrow project documentation.
Bormann, C., & Hoffman, P. CBOR — RFC 7049; RFC 8949.
Cap’n Proto and FlatBuffers project documentation.
Kleppmann, M. (2017). Designing Data-Intensive Applications. O’Reilly. (Formats in real systems.)
MessagePack specification; MongoDB BSON specification.