Ch26.13: Serialization and Binary Formats
1. No Separate Serialization Framework
fast_io does not have a serialization framework with schema registration, reflection or a runtime type registry. A binary format is written with the same three tools as any other I/O: a manipulator for a scalar, a byte operation for a blob, and a length or delimiter to frame the record.
The important part is to decide the wire layout explicitly. Do not write a struct by
casting its address to std::byte*: padding, alignment and native endianness
are not part of the protocol. Write each field with a defined width and byte order.
2. Endian Helpers
| Entry | Meaning | Module |
|---|---|---|
byte_swap(value) |
Reverse the byte order of an unsigned integer. | fast_io |
little_endian(value) |
Convert a native integer to little-endian representation. | fast_io |
big_endian(value) |
Convert a native integer to big-endian representation. | fast_io |
le_put<N>(value) |
Write an N-bit little-endian integer. |
header-side |
be_put<N>(value) |
Write an N-bit big-endian integer. |
header-side |
le_get<N>(ref), be_get<N>(ref) |
Read an N-bit integer in the selected byte order. |
header-side |
lebe_put<endian, N>(value) |
The general form behind le_put/be_put. |
header-side |
iec559_le_put, iec559_be_put |
IEEE-754 floating-point byte-order helpers. | header-side |
auto const wire{::fast_io::big_endian(value)};
::fast_io::operations::write_all_bytes(
out, reinterpret_cast<::std::byte const *>(&wire), sizeof(wire));
The conversion functions are constexpr and handle the host's native order, including
non-little/non-big targets. The le_put/le_get family is the
formatter-level alternative: it writes or reads through print/scan
instead of a raw byte copy.
3. LEB128
LEB128 is a variable-length integer encoding. fast_io provides it as a pair of manipulators:
::fast_io::io::print(out, ::fast_io::mnp::leb128_put(size));
::std::uint64_t decoded{};
::fast_io::io::scan(in, ::fast_io::mnp::leb128_get(decoded));
| Entry | Direction | Module |
|---|---|---|
leb128_put(value) |
Write | fast_io |
leb128_get(ref) |
Read | fast_io |
The encoder handles signed and unsigned integers. The decoder checks for truncation and overflow; a malformed stream is a parse error, not a partial value.
4. WebAssembly Encodings
The WebAssembly helpers are header-side aliases over the same primitives:
| Entry | Encoding |
|---|---|
wasm_varint_put, wasm_varint_get |
LEB128 varint. |
wasm_uint32_put, wasm_uint32_get |
32-bit little-endian. |
wasm_uint64_put, wasm_uint64_get |
64-bit little-endian. |
wasm_float_put, wasm_float_get |
IEEE-754 little-endian float. |
They are not in the module export list. Use #include <fast_io.h> when
you need them.
5. Fixed-Size Byte Windows
For a record that is exactly N bytes, use
read_all_bytes / write_all_bytes. For a text token with a known
length, use str_get_all(ref, n):
::std::byte header[16];
::fast_io::operations::read_all_bytes(in, header, sizeof(header));
::std::byte payload[64];
::fast_io::operations::write_all_bytes(out, payload, sizeof(payload));
These operations fail if the full window cannot be filled. A short read is not silently accepted as a complete record; a short write is not silently accepted as a complete message.
6. A Framed Binary Record
A length-prefixed record is the simplest format that avoids delimiter ambiguity. The length is a LEB128 varint; the body is raw bytes:
void write_record(::fast_io::obuf_file &out,
::fast_io::span<::std::byte const> body)
{
::fast_io::io::print(out, ::fast_io::mnp::leb128_put(body.size()));
::fast_io::operations::write_all_bytes(out, body.data(), body.size());
}
std::uint64_t length{};
::fast_io::io::scan(in, ::fast_io::mnp::leb128_get(length));
::fast_io::vector<::std::byte> body;
body.resize(length);
::fast_io::operations::read_all_bytes(in, body.data(), body.size());
Two rules make this safe. First, validate the length against a maximum before allocating. Second, treat a short body as a truncated record and fail; do not process a partial payload. The same pattern works for network frames, file records and memory buffers.
7. What Not to Serialize by Casting
- Do not write a struct as bytes. Padding, alignment and field order are implementation details.
- Do not rely on native endianness. Choose little-endian or big-endian explicitly for every field.
- Do not use
memcpyfrom a pointer field. Serialize the pointed-to data with a length. - Do not use a union as a wire format. The active member is manual; a tagged encoding is clearer and portable.
- Do not trust a length prefix. Bound it before allocation.
The union and bit-field details are in Ch25.1. The byte operations are in Ch26.9.
8. Module and Header Boundary
The fast_io module exports byte_swap,
little_endian, big_endian, leb128_put,
leb128_get and the byte read/write operations. The
le_put/le_get family, IEEE-754 helpers, WebAssembly helpers and
str_get_all are header-side today.
Key Takeaways
- fast_io serializes through manipulators and byte operations, not a schema framework.
byte_swap,little_endianandbig_endianare module-exported;le_put/le_getare header-side.leb128_put/leb128_getare module-exported and handle signed and unsigned values.- WebAssembly varint and float helpers are header-side aliases.
- Use fixed-size byte operations for exact windows; short reads and writes fail.
- Length-prefix records, validate the bound, then read the body.
- Never treat a struct or union as its in-memory byte image.