Good news!<div><br></div><div>That seems to have improved performance quite a bit so I'd like to share what I've done. Originally, with only distribute configured on a volume, I was seeing 100MB/s writes. When moving to distribute/replicate, I was getting 10MB/s or less. Avati suggested that I'm running out of extended attribute space for inodes.</div>
<div><br></div><div>I have reformatted /dev/sdb which is what I'm currently using as my gluster export. I have created a single primary partition (/dev/sdb1). My version (CentOS 5) of mke2fs (mkfs.ext3) has an undocumented option for increasing the inode-size attribute:</div>
<div><br></div><div>/sbin/mkfs.ext3 -I 512 /dev/sdb1 </div><div><br></div><div>Recreating my volume with dist/replicate:</div><div><br></div><div><div>[root@vm-container-0-3 ~]# gluster volume info pifs</div><div><br></div>
<div>Volume Name: pifs</div><div>Type: Distributed-Replicate</div><div>Status: Started</div><div>Number of Bricks: 2 x 2 = 4</div><div>Transport-type: tcp</div><div>Bricks:</div><div>Brick1: vm-container-0-0:/gluster</div>
<div>Brick2: vm-container-0-1:/gluster</div><div>Brick3: vm-container-0-2:/gluster</div><div>Brick4: vm-container-0-3:/gluster</div><div><br></div><div>and I'm consistently seeing 30+ MB/s writes with no changes to the network setup.</div>
<div><br></div><div>Thanks Avati!!</div><div><br></div><div> --joey</div><div><br></div><div><br></div><br><div class="gmail_quote">On Tue, Aug 16, 2011 at 9:31 AM, Joey McDonald <span dir="ltr"><<a href="mailto:joey@scare.org">joey@scare.org</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;"><div class="gmail_quote"><div>Hi Avati,</div><div class="im"><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="gmail_quote"><div><div></div></div><div>
Write performance in replicate is not only a throughput factor of disk and network, but also involves xattr performance. xattr performance is a function of the inode size in most of the disk filesystems. Can you give some more details about the backend filesystem, specifically the inode size with which it was formatted? If it was ext3 with the default 128byte inode, it is very likely you might be running out of in-inode xattr space (due to enabling marker-related features like geo-sync or quota?) and hitting data blocks. If so, please reformat with 512byte or 1KB inode size.</div>
<div><br></div><div>Also, what about read performance in replicate?</div></div></blockquote><div><br></div></div><div>Thanks for your insight on this issue, we are using ext3 for the gluster partition with CentOS 5 default inode size:</div>
<div><div><br></div><div>[root@vm-container-0-0 ~]# tune2fs -l /dev/sdb1 | grep Inode </div><div>Inode count: 244219904</div><div>Inodes per group: 32768</div><div>Inode blocks per group: 1024</div>
<div>Inode size: 128</div></div><div><br></div><div>I'll reformat sdb1 with 512 bytes and recreate my gluster volumes with distribute/replicate and run my benchmark tests again. </div><div><br></div><font color="#888888"><div>
<br></div><div> --joey</div><div><br></div><div></div></font></div><br><div><br></div>
</blockquote></div><br></div>