Ch11.17: Relationship Between File Layers

Why This Chapter Matters

So far we have met six families of file types: wine, nt, win32, posix, c, and filebuf. They look like separate, parallel APIs — but they are not. They form a stack, and every layer is implemented in terms of the one below it.

Understanding this stack is the key to using fast_io well. It explains why iobuf_file “just works” on every platform, why you can freely mix layers, and what native_file really means.

The Layer Hierarchy

From the lowest layer (closest to the kernel) to the highest layer (closest to the user):

Written as a chain, with the arrow meaning “is built on top of”: 


    wine (1)  =>  nt/zw (2)  =>  win32 (3)  =>  posix (4)  =>  c (5)  =>  filebuf (6)

Each layer wraps the one below it. When that is not possible (because the layer below does not expose a usable API), the layer calls the underlying system call directly and wraps that.

Layer Diagram

A concrete picture of the stack: 


    +------------------------------------------------+
    |       std::filebuf  (C++ stdlib)               |  <- filebuf_file (Layer 6)
    +------------------------------------------------+
                 |
                 v
    +------------------------------------------------+
    |               FILE*  (C stdlib)                |  <- c_file (Layer 5)
    +------------------------------------------------+
                 |
                 v
    +------------------------------------------------+
    |        POSIX file descriptors (int fd)         |  <- posix_file (Layer 4)
    +------------------------------------------------+
                 |
                 v
    +------------------------------------------------+
    |   Win32 HANDLE  (CreateFileW / CreateFileA)    |  <- win32_file (Layer 3)
    +------------------------------------------------+
                 |
                 v
    +------------------------------------------------+
    |   NT HANDLE  (NtCreateFile / ZwCreateFile)     |  <- nt_file (Layer 2)
    +------------------------------------------------+
                 |
                 v
    +------------------------------------------------+
    |   Wine host fd  (Wine translating NT to host)  |  <- wine_file (Layer 1)
    +------------------------------------------------+
                 |
                 v
              [ kernel: ntoskrnl.exe ]

Each layer corresponds to a real system component:

So when you call std::fstream, the call chain on Windows is: C++ stdlib → MSVCRT/ucrt (FILE*) → kernel32.dll (CreateFileW) → ntdll.dll (NtCreateFile) → kernel. On Linux, it is: C++ stdlib → glibc/musl (FILE*) → open()/read() syscalls → kernel.

fast_io lets you skip layers. If you want zero overhead, use native_file to call the kernel directly. If you need compatibility with legacy code, use c_file or filebuf_file. You choose the layer that fits your needs.

On Linux, FreeBSD, and macOS, only the posix branch is used. On Windows, the nt and win32 branches are used. The wine branch is only used when running under Wine on a POSIX host.

Platform availability:

Important: These relationships are not defined by fast_io. They are defined by the operating system and the compiler’s C/C++ runtime vendors. For example, on POSIX systems, the C stdlib’s FILE* is implemented on top of POSIX file descriptors by the libc implementation (glibc, musl, etc.), and the C++ stdlib’s ::std::filebuf is implemented on top of FILE* by the C++ runtime library. fast_io merely summarizes and wraps these existing relationships — it does not invent new ones.

The Golden Rule of Wrapping

fast_io provides a wrapper for every layer, and each wrapper does one of two things:

  1. Wraps the API of the layer below. For example, c_file is built on top of posix_file because the C stdlib on POSIX is itself built on top of POSIX file descriptors.
  2. Calls the lower syscall directly when the runtime below does not expose a usable API. This is often the case going from c_file down to posix_file, because libc does not publicly expose its internal fd — so fast_io goes straight to the syscall and wraps the result.

Converting Between Layers

When you need to move between layers, the direction determines the method:

Cross-layer conversions are also allowed. You don’t need to convert one layer at a time. For example, you can move directly from nt_file (the lowest Windows layer) to filebuf_file (the highest layer), skipping the intermediate win32_file, posix_file, and c_file layers: 


// Cross-layer conversion: skip intermediate layers
::fast_io::nt_file nf(u8"example.txt", ::fast_io::open_mode::out);

// Move directly from nt_file (layer 1) to filebuf_file (layer 6)
::fast_io::filebuf_file fbf(::std::move(nf), ::fast_io::open_mode::out);

// This works because filebuf_file's constructor accepts any lower-layer file type

Similarly, you can static_cast from any higher layer directly to any lower layer’s observer: 


// Direct cast from filebuf_file to nt_io_observer (skipping intermediate layers)
::fast_io::filebuf_file fbf(/* ... */);
auto nt_iob{static_cast<::fast_io::nt_io_observer>(fbf)};
HANDLE handle{nt_iob.native_handle()};  // access the raw NT handle

This flexibility lets you jump directly to the layer you need without tedious step-by-step conversions.

Special case: nt_file and win32_file

Although nt_file (layer 2) and win32_file (layer 3) are conceptually different layers from an API perspective, they both use the same HANDLE type at the OS level. If you need to convert between their observers, just use static_cast: 


// Convert between nt_io_observer and win32_io_observer
::fast_io::nt_io_observer nt_iob(/* ... */);
auto win32_iob{static_cast<::fast_io::win32_io_observer>(nt_iob)};

// The reverse also works
auto nt_iob2{static_cast<::fast_io::nt_io_observer>(win32_iob)};

This is different from true layer transitions (like posix_file → c_file), where the handle type actually changes and must be wrapped in a new abstraction. 


// Standard layer-by-layer conversion
::fast_io::posix_file pf(u8"example.txt", ::fast_io::open_mode::out);
::fast_io::c_file cf(::std::move(pf), ::fast_io::open_mode::out);

// Higher to lower: static_cast to observer for a non-owning view
auto piob{static_cast<::fast_io::posix_io_observer>(cf)};
int fd{piob.fd};  // access the raw handle

What Does native_xxx Mean?

::fast_io::native_file is an alias for the lowest unbuffered layer that fast_io wants to use on the current platform. It is chosen at compile time:

The same aliasing rule applies to the observers: native_io_observer is posix_io_observer on POSIX systems, nt_io_observer on Windows NT.

The buffered type basic_iobuf_file is then defined in terms of basic_native_file: 


template<::std::integral char_type>
using basic_iobuf_file = ::fast_io::basic_iobuf<
    ::fast_io::basic_native_file<char_type>>;

So when you write ::fast_io::iobuf_file f(u8"x.txt", open_mode::out);, you are getting buffered I/O on top of whichever layer fast_io considers native for your platform.

Mixing Layers

Because every layer exposes the same print / scan interface through concepts, you can mix them freely: 


#include <fast_io.h>
#include <fast_io_device.h>
#include <fast_io_legacy.h>

int main()
{
    using namespace ::fast_io::iomnp;

    // Start at the bottom: a POSIX file descriptor.
    ::fast_io::posix_file pf(u8"mixed.txt", ::fast_io::open_mode::out);
    println(pf, u8"Line 1 via posix_file");

    // Move it up to a C FILE*.
    ::fast_io::c_file cf(::std::move(pf), ::fast_io::open_mode::out);
    println(cf, u8"Line 2 via c_file");

    // Move it up again to a C++ std::filebuf.
    ::fast_io::filebuf_file fbf(::std::move(cf), ::fast_io::open_mode::out);
    println(fbf, u8"Line 3 via filebuf_file");

    // All three lines end up in the same file, in order.
}

Each ::std::move promotes the handle one layer upward. Ownership is transferred, not duplicated.

Building an std::fstream from a Syscall-Level Handle

Because every layer can be constructed from the one below it via ::std::move, you can start at the syscall level and ascend through every layer until you reach a fully working ::std::fstream: 


#include <fast_io.h>
#include <fast_io_legacy.h>
#include <fstream>

int main()
{
    using namespace ::fast_io::iomnp;

    // Start at the syscall level (posix on Linux/macOS, nt/win32 on Windows).
    ::fast_io::posix_file pf(u8"syscall.txt", ::fast_io::open_mode::out);

    // Move up: posix -> c
    ::fast_io::c_file cf(::std::move(pf), ::fast_io::open_mode::out);

    // Move up: c -> filebuf
    ::fast_io::filebuf_file fbf(::std::move(cf), ::fast_io::open_mode::out);

    // Transfer the filebuf into an std::ofstream
    ::std::ofstream fout;
    *fout.rdbuf() = ::std::move(*fbf.fb);

    // Now you can use the fstream like normal C++.
    fout << "Hello from a syscall-originated fstream!\n";
}

Each ::std::move promotes the handle one layer upward. Ownership is transferred, not duplicated. This is not a parlor trick — it is the natural consequence of every layer being interoperable with the one next to it.

Getting the Raw File Descriptor Out of an iobuf_file

Because the layers are transparent, you can start at the top and descend all the way back down to a raw int fd. The idiomatic way is through static_cast to the appropriate observer type: 


#include <fast_io.h>
#include <fast_io_device.h>

int main()
{
    using namespace ::fast_io::iomnp;

    // iobuf_file is buffered I/O on top of native_file.
    ::fast_io::iobuf_file bf(u8"demo.txt", ::fast_io::open_mode::out);
    println(bf, u8"Buffered write");

    // Flush so everything hits the underlying file before we peek at it.
    ::fast_io::flush(bf);

    // Step 1: obtain a posix_io_observer from the iobuf_file.
    //         On POSIX systems, iobuf_file's native file is posix_file,
    //         so this static_cast is lossless.
    auto piob{static_cast<::fast_io::posix_io_observer>(bf)};

    // Step 2: pull the raw int fd out of the observer.
    int const fd{piob.fd};

    print("The underlying fd is: ", fd);

    // The fd is still owned by `bf`; we only borrowed a view.
}

On Windows the same idea works, but you descend to ::fast_io::nt_io_observer (or ::fast_io::win32_io_observer) and the raw handle is a void* rather than an int.

The Complete Layer Reference

Layer Owning type Observer type Wraps
wine ::fast_io::wine_file ::fast_io::wine_io_observer Wine host fd
nt ::fast_io::nt_file ::fast_io::nt_io_observer NtCreateFile / ZwCreateFile
win32 ::fast_io::win32_file ::fast_io::win32_io_observer CreateFileW / CreateFileA
posix ::fast_io::posix_file ::fast_io::posix_io_observer open / openat
c ::fast_io::c_file ::fast_io::c_io_observer FILE* / fopen
filebuf ::fast_io::filebuf_file ::fast_io::filebuf_io_observer ::std::filebuf*

Key Takeaways

Memorize these six layers. They are not arbitrary — they reflect how operating systems and compiler runtimes actually implement file I/O. Understanding this hierarchy explains why FILE* behaves the way it does, why mmap is faster than fread, and how different APIs relate to each other under the hood.
← Ch11.16: Wine File APIs ↑ Main Page Ch11.18: Filesystem APIs →