jeffr_tech's Journal

Update on the linux scaling situation.
Many people have posted this link http://ozlabs.org/~anton/linux/sysbench/. Which details how switching to the google malloc fixed the problem. However if you follow the lkml thread a little deeper here http://lkml.org/lkml/2007/3/13/159, you'll see that there is more to the issue than userland malloc.

The google malloc, for reasons unknown to me, is apparently not a good general purpose allocator. Further on in the thread it's revealed that the problem may actually be that a process global semaphore is taken on every contested futex acquire and release. So it may very well be that a reasonable allocator design is being penalized by kernel contention. FreeBSD avoids this contention by having two tiers of userland locks (we call them umtx vs futex). One is process local and is used by default for most threading applications and another is system global. In the system global case we also have to traverse the vm datastructures with an expensive lock, however, we don't default to that.

I suspect that linux ended up with the process global lock being used in all cases to support the older clone() based linuxthreads which was really a bad idea for more than this reason.

On the FreeBSD front we've been improving our filedescriptor locking and replacing my tophalf patch with a more general approach via improved sx locks and sleep apis. This has resulted in another 4% or so perf boost with a slightly flatter falloff for us. I'm still working on my patch to break up the scheduler lock which should give us another nice boost.

Advertisement