jeffr_tech ([info]jeffr_tech) wrote,
@ 2007-02-23 19:45:00
Previous Entry  Add to memories!  Tell a Friend!  Next Entry
Exciting new data from the sysbench comparison.


So we got some exciting new data. I realized that we've been running with mysql 5.0.27 on my machine. I updated and re-ran the tests. I also ran with a 2.6.20.1 kernel on linux. The 'contention' refers to a bundle of contention reducing patches that are being tested for stability and inclusion in -CURRENT soon.

I would like to caution everyone against getting too excited before we finish verifying all of the results. I'm talking to some linux kernel guys now who are trying to reproduce this effect. Until then, I'm glad to see that with the same version of mysql we get the same throughput at peak. I have some scheduler changes in the queue that should address the slower ramp-up on freebsd as well.


(Post a new comment)


[info]sniket
2007-02-24 02:37 pm UTC (link)
Jeff, is Transactions/Sec. = Requests/Sec.? I'm asking because in this benchmark [1] Linux drops off as well, even though not as severe as in your test.

[1] http://tweakers.net/reviews/649/9

Maybe rerun the whole thing with Postgres?

Anyway.. thank you for doing this, maybe the kernel guys can spot a bug in the kernel and fix it, which in the end will be good for everyone :)

(Reply to this)(Thread)
Its a usermode problem
(Anonymous)
2007-03-15 09:41 am UTC (link)
Please see here: http://ozlabs.org/~anton/linux/sysbench/

For the lazy: mainly a glibc malloc problem with threading. the last graph on the link looks very "scaly" to me.

(Reply to this)(Parent)

[info]norguhtar
2007-02-24 04:16 pm UTC (link)
Strange. Check this graphs:
http://www.trinity.su/news/280.htm

Try set NOP as IO Schedulers. I'm think problem in IO.

(Reply to this)(Thread)

[info]jeffr_tech
2007-02-24 06:51 pm UTC (link)
This test doesn't do any IO at all. After the first run everything is faulted into memory and no tables are actually modified. This test was selected to help analyze bottlenecks in the operating system.

(Reply to this)(Parent)(Thread)

[info]norguhtar
2007-02-24 07:19 pm UTC (link)
Hmm.... I'm got data from this http://jeffr-tech.livejournal.com/5705.html for one thread.

Linux 2.6.18:
    transactions:                        28581  (476.35 per sec.)
    deadlocks:                           0      (0.00 per sec.)
    read/write requests:                 400134 (6668.83 per sec.)
    other operations:                    57162  (952.69 per sec.)


FreeBSD 7.0 CVS with patch:
    transactions:                        31297  (521.62 per sec.)
    deadlocks:                           0      (0.00 per sec.)
    read/write requests:                 438158 (7302.62 per sec.)
    other operations:                    62594  (1043.23 per sec.)


If MySQL get data from memory why result different more than 5% ?

PS
From my my.cnf :
key_buffer                      = 256M
myisam_sort_buffer_size         = 16M


(Reply to this)(Parent)(Thread)

(Anonymous)
2007-02-24 08:56 pm UTC (link)
That's what makes this an interesting benchmark. If your kernel has a scalability problem, e.g. lots of contention on a single lock causing many threads to back up instead of performing work, then you can find evidence of it by looking at performance with varying workload and then trying to use profiling tools to analyze the cause of the drop-off.

The indications at this stage do seem to point to such a scalability problem in the Linux kernel (as opposed to in mysql, since an identically configured mysql performs well on FreeBSD now).

(Reply to this)(Parent)(Thread)

[info]norguhtar
2007-02-25 07:17 am UTC (link)
May be may be. I'm think need additional test with open solaris :)

(Reply to this)(Parent)

[info]sniket
2007-02-24 07:41 pm UTC (link)
By the way Jeff, are you running SELinux? (default for Fedora Core).
From the Fedora FAQ:
Q: How does SELinux impact system performance?
A: [...] When performance was last measured, the impact was around 7% for completely untuned code. Subsequent changes in system components such as networking are likely to have made that *WORSE* in some cases. [...]

See the complete FAQ here:
http://fedora.redhat.com/docs/selinux-faq-fc5/

To disable SELinux add selinux=0 to your kernel command line.

(Reply to this)(Parent)(Thread)

[info]jeffr_tech
2007-02-24 08:01 pm UTC (link)
I disabled selinux in the installer. Hopefully that worked.

(Reply to this)(Parent)(Thread)

[info]sniket
2007-02-24 08:24 pm UTC (link)
You can check this with 'cat /proc/cmdline' which should say selinux=0 or edit /etc/selinux/config and change the SELINUX line to SELINUX=disabled.

(Reply to this)(Parent)

(Anonymous)
2007-02-25 05:27 pm UTC (link)
I'm wondering it this could be a linux cpu scheduler problem (it's unlikely that this is a threading model problem, everyone uses NTPL these days). Could you post your dmesg output, a snapshot of the "vmstat 1" output and a 'readprofile' output if your system has a /proc/profile file (enabled by booting with "profile=2" boot option) or send it to grundig@terra.es?

(Reply to this)(Parent)

[info]newz_top
2008-01-02 06:46 pm UTC (link)
interesting...

(Reply to this)(Parent)

[info]lostinmind
2007-02-25 03:13 am UTC (link)
Looks awesome. Sorry about not helping with the schedgraph as I was super busy at work, but I got some time free so I'm looking in to that.

(Reply to this)

[info]_dyr
2007-02-25 08:18 am UTC (link)
Good job!

(Reply to this)
linux kernel config
(Anonymous)
2007-02-25 10:42 am UTC (link)
Could you post your linux kernel .config somewhere ?
You should also check that no debug feature are enabled.

(Reply to this)

[info]iskatel
2007-02-25 12:20 pm UTC (link)
Can you test mysql 5.0.33(34) on RHEL4 with all updates or on RHEL5 beta2 ?

(Reply to this)
Please test FreeBSD 6.2
(Anonymous)
2007-02-25 10:49 pm UTC (link)
Can you test FreeBSD 6.2-RELEASE???
To compare the diffrence beetwen 6.2 and 7.0-CURRENT...

Please... :)

(Reply to this)
Response times ?
(Anonymous)
2007-02-25 10:53 pm UTC (link)
Hi !

interesting results, as I've already encountered corner cases with linux 2.6 scheduler, notably antivirus processes freezing for several seconds. It would be interesting to note transaction times. I would bet that they will be really unequal on 2.6. Could you test the same on plain 2.4 (not RHEL nor any O(1) scheduler) ? It will probably be below 2.6 in terms of performance, but the difference in the shape and response times may provide some tuning hints.

(Reply to this)

[info]diegocg
2007-02-26 12:06 am UTC (link)
Apparently a kernel hacker has been able to reproduce the problem - http://lkml.org/lkml/2007/2/25/191

(Reply to this)
Glibc problems?
(Anonymous)
2007-03-12 10:56 pm UTC (link)
http://ozlabs.org/~anton/linux/sysbench/

(Reply to this)
solved (glibc problem)
(Anonymous)
2007-03-13 08:37 am UTC (link)
http://ozlabs.org/~anton/linux/sysbench/

(Reply to this)
Problem found and meta solved
(Anonymous)
2007-03-14 10:36 am UTC (link)
http://ozlabs.org/~anton/linux/sysbench/

(Reply to this)

[info]sjmurdoch
2007-03-15 10:28 am UTC (link)
It appears to be a glibc malloc() problem, as switching to the Google malloc fixes the problem.

(Reply to this)(Thread)

(Anonymous)
2007-03-29 04:22 pm UTC (link)
Very cool, so glibc is the problem. Has GNU been made aware of this issue? Is there a way to track this issue and see if GNU will restructure the malloc code to handle multiple threads better?

(Reply to this)(Parent)

[info]masmedia
2007-04-02 11:28 am UTC (link)
words so I'll just say nice find.

(Reply to this)

[info]7wrc
2007-05-02 03:55 am UTC (link)
Good job.

(Reply to this)

[info]selenation
2007-05-07 11:41 pm UTC (link)
Thanks for the useful information, keep up the good work.

(Reply to this)

[info]amandaqixy
2007-06-01 04:31 pm UTC (link)
Actually great stuff!

(Reply to this)

[info]ttrtt
2007-09-02 01:51 pm UTC (link)
delete

(Reply to this)

(Reply from suspended user)
thanks
[info]oyunlar
2007-12-29 12:35 am UTC (link)
God job thanks
kadın hastalıkları

(Reply to this)

Create an Account
Forgot your login?
Login w/ OpenID
English • Español • Deutsch • Русский…