I-Space Research Labs

GFS Tuning and Iran

by on Jun.24, 2009, under GFS, Tech Stuff

I’m not surprised that the Iranian government stacked the deck and tried to screw their own people over. People who are in power illegitimately ususally go to any means to ensure that they STAY in power. I’m also disappointed in our own talking heads, especially McCain, who think that we should be charging in somehow and fixing this for the Iranians. Because it worked so well in Iraq. A democracy that is forced on people isn’t a democracy. Countries like Iraq, who have spent the last few decades under the opressive rule of a dictator, they don’t know what to do with the democracy they’ve been given. The fate of the Iranian people lies in their own hands, all we can do as a responsible nation is to make sure that they don’t get all machinegunny on their own people.

Anyway, today’s topic is….GFS tuning. I was doing some benchmarking with a cluster of three nodes all tied back to a shared GFS filesystem that is shared via iSCSI. I don’t think the underlying network is gigabit, likely Fast Ethernet and the iSCSI server is a measly little workstation. With the default parameters, I was getting initially 5 megabytes/sec dd’ing /dev/urandom to a file until it was a gigabyte in size. Once we had that baseline down, I did a few tests. This is the standard command I used on all my tests:

dd if=/dev/urandom of=/my/gfs/file bs=1024 count=1000000 <- Random garbage of about a gig in size

First test, enable directio. This can be enabled at the directory or file level, or invoked directly from an application that is directio aware. Directio bypasses system caching and talks directly to disk

touch /my/gfs/file

gfs_tool setflag directio /my/gfs/file <- This sets directio on a file called “file”

Conduct dd test, and I get throughput of less than 2MB/sec. Wow. That’s less than half.  Maybe directio works better on small files or something. rm’ing the file also kills the directio bit. If you dont’ know whether a file or directory has directio enabled on it, do a gfs_tool stat <file> or <directory>, you’ll see if any flags are enabled at the bottom of the output. Clearing the directio flag without killing the file is a simple matter of gfs_tool clearflag directio <file>

Next we try enabling statfs_fast, which can improve our times potentially…

gfs_tool settune /my/gfs statfs_fast 1

dd again, and we see about 5.1 MB/sec. Slight improvement over defaults, but at least we’re not cutting our throughput

Turn off statfs_fast and now we’ll try data journaling. According to documentation, this works better for smaller files. Instead of journaling just metadata, the data itself gets journaled

touch /my/gfs/file

gfs_tool setflag jdata /my/gfs/file

Throughput is about 2.5MB/sec. Not good for a gig file.

The noatime option is interesting. It basically doesn’t update the access time entry for files, so we’re eliminating potentially unecessary IO on files we’re touching. This is not a gfs_tool option, it’s a mount option, so we will need to remount the GFS volume:

mount -t gfs /my/gfs /dev/mapper/myvg/mylv -o noatime,remount

dd, and we see about 5.2MB/sec. Not a huge gain, but at least we’re not cutting our throughput.

So now I started experimenting with combinations of various options and to no surprise, mounting with noatime and enabling statfs_fast gives me an overall throughput of 6.2MB/sec. I ran the dd test again with just all the defaults and got about 6MB/sec the second time around, so not huge gains to be had here. I am keeping in mind that this is NOT an optimal setup I’m using: my cluster nodes are using the same interface for cluster, network and iSCSI traffic, the iSCSI host is a crappy workstation, the network is far from optimized. I haven’t even tried some other options like glock_purge yet. Hopefully I will be able to play with those tomorrow and see how much that makes a difference. I haven’t even upgraded to GFS2 yet, so there may be a performance boost there yet.

Most importantly, remember that all of these GFS tuning tweaks are NOT persistant across umounts/mounts,  so if you find some tweaks that work for you, you need to stick them in rc.local or something


Leave a Reply

Looking for something?

Use the form below to search the site:

Still not finding what you're looking for? Drop a comment on a post or contact us so we can take care of it!

Visit our friends!

A few highly recommended friends...


All entries, chronologically...