Ch26.20: Character Encoding

中文(简体)

1. Encoding Schemes

encoding_scheme Meaning
execution_charsetThe compiler's execution character set for the source character type.
utf_leUTF in little-endian code units.
utf_beUTF in big-endian code units.
utfThe native-endian UTF alias.
gb18030GB18030 / GBK.
utf_ebcdicUTF-EBCDIC.

The scheme names describe the encoding of the character units, not the number of characters. A char8_t sequence in utf is UTF-8; a char16_t sequence is UTF-16 in native byte order; a char32_t sequence is UTF-32 in native byte order.

2. Execution Charset Detection

execution_charset_encoding_scheme<char_type>() returns the scheme the compiler selected for that character type. It recognizes GCC/Clang execution-charset names for GB18030/GBK, MSVC's execution-charset macro, EBCDIC targets and wide-character endianness.

execution_charset_encoding<char_type> is the same value as a variable template. Use it when you need a compile-time default; pass an explicit scheme when the wire format has a fixed byte order.


constexpr auto native_scheme{
    ::fast_io::execution_charset_encoding_scheme<char>()};

The helpers and the enum are header-side today. The module exports the two manipulators below, not the scheme names.

3. code_cvt

mnp::code_cvt<src, dst>(value) converts a print alias/scatter during output. The destination character type comes from the output stream; the source type comes from the value.


::fast_io::u16obuf_file out{"utf16.txt"};
::fast_io::io::println(
    out, ::fast_io::mnp::code_cvt<::fast_io::encoding_scheme::utf,
                                     ::fast_io::encoding_scheme::utf>(u8"Hello"));
Form Use
code_cvt<src, dst>(value) Convert a printable value or scatter.
code_cvt_os_c_str<src, dst>(ptr) Convert a NUL-terminated OS string.
code_cvt_os_c_str<src, dst>(ptr, n) Convert a known-length OS string.

The conversion is performed into the output device's reserve buffer; it does not allocate an intermediate string. Invalid source sequences are replaced with the encoding's replacement code units (U+FFFD for Unicode encodings).

4. OS Strings and Execution Charset

A path from argv or an environment variable is in the execution charset, not necessarily UTF-8. Use code_cvt_os_c_str when printing it:


::fast_io::io::println(
    ::fast_io::mnp::code_cvt_os_c_str<::fast_io::encoding_scheme::execution_charset,
                                          ::fast_io::encoding_scheme::utf>(argv[1]));

For files, the device type already chooses the native encoding path; this manipulator is for turning an OS string into text on an output stream.

5. What Is Not a Transcoder

code_cvt is a value-local formatting adapter. It is not the older stateful transcoder pipeline, and it does not expose a stream object. If you need an incremental decoder with an explicit state, use the general general_code_cvt overloads in the header interface; for ordinary output, code_cvt is the smaller tool.

6. Module and Header Boundary

The fast_io module exports mnp::code_cvt and mnp::code_cvt_os_c_str. encoding_scheme, execution_charset_encoding_scheme, execution_charset_encoding, code_cvt_result and the general converter are header-side today. Include <fast_io.h> when you need to name the scheme explicitly.

Key Takeaways