Ch26.20: Character Encoding
1. Encoding Schemes
encoding_scheme |
Meaning |
|---|---|
execution_charset | The compiler's execution character set for the source character type. |
utf_le | UTF in little-endian code units. |
utf_be | UTF in big-endian code units. |
utf | The native-endian UTF alias. |
gb18030 | GB18030 / GBK. |
utf_ebcdic | UTF-EBCDIC. |
The scheme names describe the encoding of the character units, not the number of
characters. A char8_t sequence in utf is UTF-8; a
char16_t sequence is UTF-16 in native byte order; a char32_t
sequence is UTF-32 in native byte order.
2. Execution Charset Detection
execution_charset_encoding_scheme<char_type>() returns the scheme the
compiler selected for that character type. It recognizes GCC/Clang execution-charset
names for GB18030/GBK, MSVC's execution-charset macro, EBCDIC targets and wide-character
endianness.
execution_charset_encoding<char_type> is the same value as a variable
template. Use it when you need a compile-time default; pass an explicit scheme when the
wire format has a fixed byte order.
constexpr auto native_scheme{
::fast_io::execution_charset_encoding_scheme<char>()};
The helpers and the enum are header-side today. The module exports the two manipulators below, not the scheme names.
3. code_cvt
mnp::code_cvt<src, dst>(value) converts a print alias/scatter during
output. The destination character type comes from the output stream; the source type comes
from the value.
::fast_io::u16obuf_file out{"utf16.txt"};
::fast_io::io::println(
out, ::fast_io::mnp::code_cvt<::fast_io::encoding_scheme::utf,
::fast_io::encoding_scheme::utf>(u8"Hello"));
| Form | Use |
|---|---|
code_cvt<src, dst>(value) |
Convert a printable value or scatter. |
code_cvt_os_c_str<src, dst>(ptr) |
Convert a NUL-terminated OS string. |
code_cvt_os_c_str<src, dst>(ptr, n) |
Convert a known-length OS string. |
The conversion is performed into the output device's reserve buffer; it does not allocate an intermediate string. Invalid source sequences are replaced with the encoding's replacement code units (U+FFFD for Unicode encodings).
4. OS Strings and Execution Charset
A path from argv or an environment variable is in the execution charset, not
necessarily UTF-8. Use code_cvt_os_c_str when printing it:
::fast_io::io::println(
::fast_io::mnp::code_cvt_os_c_str<::fast_io::encoding_scheme::execution_charset,
::fast_io::encoding_scheme::utf>(argv[1]));
For files, the device type already chooses the native encoding path; this manipulator is for turning an OS string into text on an output stream.
5. What Is Not a Transcoder
code_cvt is a value-local formatting adapter. It is not the older stateful
transcoder pipeline, and it does not expose a stream object. If you need an
incremental decoder with an explicit state, use the general
general_code_cvt overloads in the header interface; for ordinary output,
code_cvt is the smaller tool.
6. Module and Header Boundary
The fast_io module exports mnp::code_cvt and
mnp::code_cvt_os_c_str. encoding_scheme,
execution_charset_encoding_scheme, execution_charset_encoding, code_cvt_result and the general converter are
header-side today. Include <fast_io.h> when you need to name the scheme
explicitly.
Key Takeaways
encoding_schemenames UTF endian variants, GB18030 and UTF-EBCDIC.utfis the native-endian alias; the character type determines whether it is UTF-8, UTF-16 or UTF-32.execution_charset_encoding_scheme<T>()detects the compiler's execution charset.code_cvt<src, dst>converts one printable value during output.code_cvt_os_c_strconverts NUL-terminated or known-length OS strings.- Invalid sequences become replacement code units, not silent truncation.
- The module exports the two manipulators; the scheme names and general converter are header-side.