Ch26.13: Serialization and Binary Formats

中文(简体)

1. No Separate Serialization Framework

fast_io does not have a serialization framework with schema registration, reflection or a runtime type registry. A binary format is written with the same three tools as any other I/O: a manipulator for a scalar, a byte operation for a blob, and a length or delimiter to frame the record.

The important part is to decide the wire layout explicitly. Do not write a struct by casting its address to std::byte*: padding, alignment and native endianness are not part of the protocol. Write each field with a defined width and byte order.

2. Endian Helpers

Entry Meaning Module
byte_swap(value) Reverse the byte order of an unsigned integer. fast_io
little_endian(value) Convert a native integer to little-endian representation. fast_io
big_endian(value) Convert a native integer to big-endian representation. fast_io
le_put<N>(value) Write an N-bit little-endian integer. header-side
be_put<N>(value) Write an N-bit big-endian integer. header-side
le_get<N>(ref), be_get<N>(ref) Read an N-bit integer in the selected byte order. header-side
lebe_put<endian, N>(value) The general form behind le_put/be_put. header-side
iec559_le_put, iec559_be_put IEEE-754 floating-point byte-order helpers. header-side

auto const wire{::fast_io::big_endian(value)};
::fast_io::operations::write_all_bytes(
    out, reinterpret_cast<::std::byte const *>(&wire), sizeof(wire));

The conversion functions are constexpr and handle the host's native order, including non-little/non-big targets. The le_put/le_get family is the formatter-level alternative: it writes or reads through print/scan instead of a raw byte copy.

3. LEB128

LEB128 is a variable-length integer encoding. fast_io provides it as a pair of manipulators:


::fast_io::io::print(out, ::fast_io::mnp::leb128_put(size));

::std::uint64_t decoded{};
::fast_io::io::scan(in, ::fast_io::mnp::leb128_get(decoded));
Entry Direction Module
leb128_put(value) Write fast_io
leb128_get(ref) Read fast_io

The encoder handles signed and unsigned integers. The decoder checks for truncation and overflow; a malformed stream is a parse error, not a partial value.

4. WebAssembly Encodings

The WebAssembly helpers are header-side aliases over the same primitives:

Entry Encoding
wasm_varint_put, wasm_varint_get LEB128 varint.
wasm_uint32_put, wasm_uint32_get 32-bit little-endian.
wasm_uint64_put, wasm_uint64_get 64-bit little-endian.
wasm_float_put, wasm_float_get IEEE-754 little-endian float.

They are not in the module export list. Use #include <fast_io.h> when you need them.

5. Fixed-Size Byte Windows

For a record that is exactly N bytes, use read_all_bytes / write_all_bytes. For a text token with a known length, use str_get_all(ref, n):


::std::byte header[16];
::fast_io::operations::read_all_bytes(in, header, sizeof(header));

::std::byte payload[64];
::fast_io::operations::write_all_bytes(out, payload, sizeof(payload));

These operations fail if the full window cannot be filled. A short read is not silently accepted as a complete record; a short write is not silently accepted as a complete message.

6. A Framed Binary Record

A length-prefixed record is the simplest format that avoids delimiter ambiguity. The length is a LEB128 varint; the body is raw bytes:


void write_record(::fast_io::obuf_file &out,
                  ::fast_io::span<::std::byte const> body)
{
    ::fast_io::io::print(out, ::fast_io::mnp::leb128_put(body.size()));
    ::fast_io::operations::write_all_bytes(out, body.data(), body.size());
}

std::uint64_t length{};
::fast_io::io::scan(in, ::fast_io::mnp::leb128_get(length));

::fast_io::vector<::std::byte> body;
body.resize(length);
::fast_io::operations::read_all_bytes(in, body.data(), body.size());

Two rules make this safe. First, validate the length against a maximum before allocating. Second, treat a short body as a truncated record and fail; do not process a partial payload. The same pattern works for network frames, file records and memory buffers.

7. What Not to Serialize by Casting

The union and bit-field details are in Ch25.1. The byte operations are in Ch26.9.

8. Module and Header Boundary

The fast_io module exports byte_swap, little_endian, big_endian, leb128_put, leb128_get and the byte read/write operations. The le_put/le_get family, IEEE-754 helpers, WebAssembly helpers and str_get_all are header-side today.

Key Takeaways