jeffr_tech's Journal

Using atomics in struct file.
I'm completely obsessive compulsive about code sometimes. Haven't been able to sleep while thinking about file and descriptor locking. I ended up completely rewriting the unix domain socket garbage collector. It now no longer scans the entire list of descriptors in the system. This enabled me to remove the list of all descriptors and the lock that protected it. Saving a few pointers in every struct file.

In addition to this, I removed all explicit locking of struct file. The reference count and file flags, settable via fcntl(F_SETFL), were protected by a mutex. This was also used for some cases in the garbage collection code that I removed. The reference count is now handled completely with atomics, as are the flags. The flags were somewhat tricky because in some cases you wanted to do:

flags |= SOMEFLAG;
flags &= ~OTHERFLAG;

It's not legal to do this with two separate atomics. The reason is somewhat complex but also pretty interesting. These flags are only set by user-space and inspected by the kernel. In the presence of multiple threads the kernel can not protect against races involving multiple threads setting flags simultaneously. However, one must win. The results can't be a mix of both. So for compound statements I do things like this:

do {
        old = new = fp->f_flags;
        new |= SOMEFLAG:
        new &= ~OTHERFLAG;
}while (atomic_cmpset_int(&fp->f_flags, old, new) == 0);

This loop uses an atomic compare and exchange to update the flags. If f_flags changes after we fetch it into old the loop will restart and we'll try again. In practice restarting the loop is likely to be incredibly rare. For cases which only require updating one bit you can use something like the following in bsd:

atomic_set_int(&fp->f_flag, SOMEFLAG);

Pretty handy. For the atomic reference count we do "if (atomic_fetchadd_int(&fp->f_count, -1) == 1) free the fp". Atomic fetchadd adds -1 to the integer pointed at and returns the old value. If we're closing the last reference we know it's safe to free the structure because it's not possible for another reference to be gained.

Advertisement