Sorry, I forgot to include informatino about my testbed.<br><br>4 machines - Gluster (as described below)<br>3 machines - WWW server, load balance using LVS<br>1 machine - LVS direct routing and weighted lease connection<br>
3 machines - clients, 100 threads per client.<br><br><div class="gmail_quote">2009/7/12 Somsak Sriprayoonsakul <span dir="ltr"><<a href="mailto:somsaks@gmail.com">somsaks@gmail.com</a>></span><br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
Hello,<br><br>We have been evaluating the choice for the new platform for a webboard system. <br>The webboard is PHP scripts that generate/modify HTML page when user posting/add comment to the page, resulting topic is actually stored as a HTML file with all related file (file attach to the topic, etc.. )stored in its own directory for each topic. In general, the web site mostly serve a lot of small static files using Apache while using PHP to do other dynamic contents. This system has been working very well in the past, with the increasing page view rate, it is very likely that we will need some kind of Cluster file system as backend very soon.<br>
<br>We have set up a test system using Grinder as stress test tool. The test system is 11 machines of Intel Dual Core x86_64 CentOS5 with stock Apache (prefork, since the goal is to use this with PHP), linked together with Gigabit Ethernet. We try to compare the performance of either using single NFS server in sync mode against using 4 Gluster nodes (distribute of 2 replicated nodes) through Fuse. However, the transaction per second (TPS) result is not good.<br>
<br>NFS (single server, sync mode)<br> - 100 thread of client - Peak TPS = 1716.67, Avg. TPS = 1066, mean response time = 61.63 ms<br> - 200 threads - Peak TPS = 2790, Avg. TPS = 1716, mean rt = 87.33 ms<br> - 400 threads - Peak TPS = 3810, Avg. TPS = 1800, mean rt = 165ms<br>
- 600 threads - Peak TPS = 4506.67, Avg. TPS = 1676.67, mean rt = 287.33ms<br><br>4 nodes Gluster (2 distribute of replicated 2 node)<br>- 100 thread - peak TPS = 1293.33, Avg. TPS = 430, mean rt = 207.33ms<br>- 200 threads - Peak TPS = 974.67, Avg. TPS = 245.33, mean rt = 672.67ms<br>
- 300 threads - Peak TPS = 861.33, Avg. TPS = 210, mean rt = 931.33<br>(no 400-600 threads since we run out of client machine, sorry).<br><br>gfsd is configured to use 32 thread of iothread as brick. gfs-client is configured to use io-cache->write-behind->readahead->distribute->replicate. io-cache cache-size is 256MB. I used patched Fuse downloaded from Gluster web-site (build through DKMS).<br>
<br>As the result yield, it seems that Gluster performance worse with increasing no. of client. One observation is that the glusterfs process on client is taking about 100% of CPU during all the tests. glusterfsd is utilizing only 70-80% of CPUs during the test time. Note that system is Dual core.<br>
<br>I also tried using modglusterfs and not using fuse at all to serve all the static files and conduct another test with Grinder. The result is about the same, 1000+ peak TPS with 2-400 avg. TPS. A problem arise in this test that each Apache prefork process used more about twice more memory and we need to lower number of httpd processes by about half.<br>
<br>I tried disable EnableMMAP and it didn't help much. Adjusting readahead, write behind according to GlusterOptimization page didn't help much either.<br><br>My question is, there seems to be bottleneck in this setup, but how can I track this? Note that, I didn't do any other optimization other than what said above. Are there any best practice configuration for using Apache to serve a bunch of small static files like this around?<br>
<br>Regards,<br><font color="#888888"><br>Somsak<br><br><br><br>
</font></blockquote></div><br>