Techie Notes‎ > ‎

ext4 + MySql performance issues

posted Mar 18, 2010 9:46 PM by Tedman Leung   [ updated Dec 23, 2010 9:19 PM ]
I recently attempted an upgrade of my Fedora 10 system to Fedora 12. Fedora 12 uses ext4 by default. After installing F12 I noticed some of my junit tests were running very slowly. Junit tests that use to take about 1.5 seconds were now taking 9 seconds. The entire suite of test use to take about 1 minute 30 seconds using ext3, it now took close to 5 minutes on ext4.

After some more extensive testing here are some of the results.
  • If you install F12 with ext4 the system is slow, if you do the exact same install with ext3 it is quick.
  • If you test with an external HD, you get mixed results, sometimes it was quick on external usb drives even when formatted with ext4.
  • If you make a spare partition on your main HD, you can see the results by making it ext3 / 4 and restarting your MySql.
  • It's not simply and ext4 overall performance problem though, a test of zipping a slew of small files (source code) showed ext4 was running faster than ext3.
  • Another way to notice the problem but not as severely was to import a mysql database. An imports seemed to take 33% longer than previously.
My final solution was to re-install F12 using ext3 as the file system. This resolved all my problems.

For reference, I was also using encryption on the HD but I also tested it with out encryption with similar slow down results. I also played with turning on and off selinux with no real effect.

-----

Upon further investigation it appears that it has to do with write barriers. http://kernelnewbies.org/Ext4#head-25c0a1275a571f7332fa196d4437c38e79f39f63 allegedly if you mount the FS with barrier=0 it will run a lot faster. There are integrity ramifications though.

-----

Update for Fedora 14 :
Installed Fedora 14 today and unfortuntely the problem still exists.

Here's a few snippets from my ext3 system running FC12:

Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.219 sec
Running org.oscarehr.caisi_integrator.dao.FacilityTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.794 sec
Running org.oscarehr.caisi_integrator.util.MiscUtilsTest
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.017 sec
Running org.oscarehr.caisi_integrator.dao.EventLogTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.853 sec
Running org.oscarehr.caisi_integrator.dao.DemographicLinkTest
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.237 sec
 
Here's the same on ext4 using defaults running FC14:

Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.276 sec
Running org.oscarehr.caisi_integrator.dao.FacilityTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 19.323 sec
Running org.oscarehr.caisi_integrator.util.MiscUtilsTest
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.017 sec
Running org.oscarehr.caisi_integrator.dao.EventLogTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 18.865 sec
Running org.oscarehr.caisi_integrator.dao.DemographicLinkTest
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 23.002 sec

As you can see it's roughly 10x slower... than ext3 on FC12.

Here's ext4 after setting barrier=0 in /etc/fstab on FC14

Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.427 sec
Running org.oscarehr.caisi_integrator.dao.FacilityTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.03 sec
Running org.oscarehr.caisi_integrator.util.MiscUtilsTest
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.019 sec
Running org.oscarehr.caisi_integrator.dao.EventLogTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.854 sec
Running org.oscarehr.caisi_integrator.dao.DemographicLinkTest
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.16 sec

The performance is still about 3x worst than ext3 on FC12.

Here's ext3 and defaults on FC14 :

Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.269 sec
Running org.oscarehr.caisi_integrator.dao.FacilityTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.646 sec
Running org.oscarehr.caisi_integrator.util.MiscUtilsTest
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.015 sec
Running org.oscarehr.caisi_integrator.dao.EventLogTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.745 sec
Running org.oscarehr.caisi_integrator.dao.DemographicLinkTest
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.51 sec

The performance is still 2x worst than ext3 on FC12.

Here's ext4 and noatime,nodiratime,barrier=0,data=journal,commit=30 on FC14 :

Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.276 sec
Running org.oscarehr.caisi_integrator.dao.FacilityTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.747 sec
Running org.oscarehr.caisi_integrator.util.MiscUtilsTest
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.017 sec
Running org.oscarehr.caisi_integrator.dao.EventLogTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.813 sec
Running org.oscarehr.caisi_integrator.dao.DemographicLinkTest
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.528 sec

Performance is now about 1.5x worst than FC12 & ext3.

Here's ext3 and noatime,nodiratime,data=journal,commit=30 on FC14 :

Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.28 sec
Running org.oscarehr.caisi_integrator.dao.FacilityTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.98 sec
Running org.oscarehr.caisi_integrator.util.MiscUtilsTest
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.018 sec
Running org.oscarehr.caisi_integrator.dao.EventLogTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.939 sec
Running org.oscarehr.caisi_integrator.dao.DemographicLinkTest
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.371 sec

YAYAYA!! kind of. It's still about 10% slower than FC12 & ext3 but that's acceptable for me - for now.

The kickers is that I had to enable data=journal which wasn't enabled on my FC12 system it was using the default writeback. For fun... I went back to the FC12 system and enabled journal. As expected it was slower... by about 20% (the unexpected result is that journalling was faster in FC14...).