A minor gotcha in open(2)

by epilys on 2022-10-22

A test I was writing in C was causing a panic in Rust code that was called via FFI. The panicking line of Rust code was something of the sort:

use nix::sys::stat::Mode;

fn foo(mode: libc::mode_t) {
  let mode: Mode = Mode::from_bits(mode).unwrap();
  ..
}

nix¹ fails to handle a valid (I thought) mode value? There’s gotta be a simpler explanation: the mode value is invalid. Indeed looking at the C code:

char *file = "/tmp/file";
...
fd = open(file, O_WRONLY | O_CREAT | O_TRUNC);
...

example taken from POSIX open specification²

I had forgotten to specify the mode argument in open. However that’s legal behavior in POSIX, and it doesn’t define what happens when you call the variadic³ function open without the mode argument.

The glibc+Linux open(2) manual page with man 2 open explains what happens in their implementation:

The mode argument must be supplied if O_CREAT
or O_TMPFILE is specified in flags; if it is
not supplied, some arbitrary bytes from the
stack will be applied as the file mode [..]

Why am I not surprised! This should not be considered acceptable behavior for a standard library.

Rust friendly bindings to *nix APIs https://crates.io/crates/nix ↩︎
https://pubs.opengroup.org/onlinepubs/007904875/functions/open.html ↩︎
variadic means it accepts a non fixed number of arguments, like printf. Its function signature is
```
int open(const char *path, int oflag, ... );
```
Without historical context, my guess this was to avoid having a different function when you want to specify the mode. Adding extra functions for extra arguments is an acceptable pattern in other libc functions, though.↩︎