Buffering — why writing to a file doesn't hit disk immediately
Calling write() doesn't mean the data is safely on disk the instant the call returns — most writes land in a buffer first, traded deliberately for speed, and that gap is exactly where a power loss can lose 'already written' data.
Beginner
2 min read
Writing to memory, not to disk, most of the time
Disk I/O is slow relative to memory (the memory hierarchy lesson already
established just how much). If every single write() call had to wait for
data to physically reach the disk before returning, ordinary file writing
would be dramatically slower than it actually is. Instead, the kernel
maintains a buffer (also called a page cache, for file data
specifically) in RAM: a write() call usually just copies data into that
buffer and returns immediately, while the kernel writes the buffer's
contents out to disk later, in its own time.
The trade-off this creates: speed for a real risk window
This is a deliberate, worthwhile trade for almost all ordinary file I/O —
but it creates a genuine gap: between "write() returned successfully" and
"the data is actually, physically on disk," there's a window where a sudden
power loss or a hard crash can lose data the program already believes was
saved. This isn't a bug; it's the explicit, well-known cost of buffering,
and every serious system that cares about durability has to account for it
directly.
Forcing data out of the buffer, when it actually matters
For cases where "the OS will get to it eventually" genuinely isn't good
enough — a database committing a transaction, a critical log entry — there's
an explicit way to force buffered data out to disk immediately:
import oswith open("important.log", "w") as f: f.write("critical event occurred\n") f.flush() # push from Python's own buffer into the OS buffer os.fsync(f.fileno()) # force the OS to write that buffer to physical disk
flush() and fsync() solve two different halves of the same problem —
flush() moves data out of a language-level buffer into the OS's buffer,
while fsync() is what actually forces the OS's buffer out to the physical
disk. Skipping fsync() in code that specifically needs a durability
guarantee is a real, recurring class of bug: everything looks fine until an
actual crash exposes exactly which recent writes never made it past the
buffer.
Why this matters day to day
This is precisely why a database talks about "committing" a transaction as
a distinct, meaningful step, rather than treating every write as
automatically durable — a commit typically includes an explicit fsync()
specifically so the system can promise "once this call returns, your data
survives a crash," at the cost of that call being measurably slower than an
ordinary buffered write. Understanding this trade-off is the difference
between reading "why is fsync so slow" as a mystery and recognizing it as
the literal, unavoidable price of the durability guarantee you actually
asked for.