new | show | ask | jobs Github

jeffbee 7 hours ago

I wonder why the author left out atomic writes with O_APPEND.

▲

ozgrakkurt 7 hours ago | parent | next [-]

This requires O_SYNC and O_DIRECT afaik.

Even then it is only some file systems that guarantee it and even then file size updating isn’t atomic afaik.

Not so sure about file size update being atomic in this case but fairly sure about the rest.

Matklad had some writing or video about this.

Also there is a tool called ALICE and authors of that tool have a white paper about this subject.

Also there was a blog post about how badger database fixed some issues around this problem.

▲

jeffbee 7 hours ago | parent [-]

I don't think any part of your post is right. Aside from NFS, there should not be filesystems where this doesn't work. If there are, those are just bugs. The flags you mentioned are not required or relevant. Setting the fd offset to the end of the file atomically is the entire purpose of O_APPEND.

▲

ozgrakkurt 6 hours ago | parent [-]

It depends on what you mean by atomic. If it is only writing to page cache and you are writing a small amount then yes?

If there is a failure like a crash or power outage etc. then it doesn’t work like that.

You might as well be pushing into an in-memory data structure and writing to disk at program exit in terms of reliability

	▲	jeffbee 6 hours ago \| parent [-]
		You are projecting imaginary features onto O_APPEND and then hypothesizing that your imaginary features might not work. POSIX says that for a file opened with O_APPEND "the file offset shall be set to the end of the file prior to each write." That's it. That's all it does.

▲

zbentley 7 hours ago | parent | prev [-]

Unsure. Aren’t there filesystems which make O_APPEND less durable than it’s specified to be, which might be interpreted to adversely affect atomicity? Could that be it?