I-Space Research Labs

More GFS tuning

by on Jun.25, 2009, under GFS, Tech Stuff

Finally had some time to do some more GFS tuning on my test cluster.

First thing I’ve discovered that even writing small (~1Mb) files, using directio cuts your throughput in half. It’s fail, don’t use it. Same with data journaling. Don’t bother.

But where the SWEET stuff is with glock_purging and demote_secs. On a 100Mbps network connection to an old cranky Dell workstation with the iSCSI target running… 3 servers writing 1000 1-Mb files to random locations on the GFS filesystem, I saw up to 6.8Mb throughput on all 3 servers at the same time. Hopefully I’ll get some real SAN hardware soon so I can get some real performance.

The two parameters are glock_purge and demote_secs. You set them with:

gfs_tool settune /my/gfs glock_purge X

gfs_tool settune /my/gfs demote_secs X

glock_purge accepts an argument that tells gfsd what percentage of unused locks to purge every 5 seconds. Redhat recommends starting at 50 and working your way up. I’m currently pushing 90 right now, but I think that may be a bit too aggressive, but then I’m just doing some benchmarking. Production may turn out to be different.

demote_secs is the number of seconds that gfsd will wake up and demote locks and flush data to disk. So it stands that a lower number may be beneficial. I’m currently at 5, but this may be too silly, but I like to see what the extremes look like as I dial in. The default is 300 seconds.

You can read more about them here

Here’s how I set up my mounts on all 3 servers:

mount -t gfs /dev/myvg/mygfs /mnt/gfs -o acl,noatime,nodiratime

gfs_tool settune /mnt/gfs statfs_fast 1

gfs_tool settune /mnt/gfs glock_purge 90

gfs_tool settune /mnt/gfs demote_secs 5

Remember these numbers are probably not good for production.

On one of the servers, I do a little for loop to set up the test:

for i in {1..1000}; do mkdir /mnt/gfs/$i;done

This creates 1000 folders on the gfs mount.

Then a short bash script:

## gfshammer.sh

##GFS testing script. Yay.


echo “Starting: “`date`>>~/timefile

for i in {1..1000}


NUM=`let R=$RAND%1000;echo $R`

SIZE=`let S=$RAND%1000;echo $S`


dd if=/dev/urandom of=/mnt/gfs/$NUM/test$i bs=1024 count=$MYCOUNT


This will creat randomly sized files full of random data in random places on the GFS filesystem. I ran this on all 3 nodes at the same time and saw lows of 4MB/sec to highs of 6.8MB/sec, usually around 6MB/sec. That ain’t bad given the underlying infrastructure: 100Mbps LAN, single spindle on an old workstation. I think at this point I’m being bottlenecked by the network. I was getting around 6MB/sec with just a single node without any glock tuning the other day, so this seems like a big jump forward.

Also, I tried GFS2, and I’m sad to report that its performance is nowhere near what I was getting with GFS. I can’t tune glocks as GFS2 is supposed to be self-tuning, but I saw a pretty significant drop in throughput when I tried it, so back to GFS we go…


Leave a Reply

Looking for something?

Use the form below to search the site:

Still not finding what you're looking for? Drop a comment on a post or contact us so we can take care of it!

Visit our friends!

A few highly recommended friends...


All entries, chronologically...