<div dir="ltr"><div class="gmail_default" style="font-family:tahoma,sans-serif"><span style="font-family:arial">On Sun, Mar 23, 2014 at 6:10 AM, Josh Boon </span><span dir="ltr" style="font-family:arial"><<a href="mailto:gluster@joshboon.com" target="_blank">gluster@joshboon.com</a>></span><span style="font-family:arial"> wrote:</span><br>
</div><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">Thanks for those options. My machines tend to be self-healing rather frequently. Doing a gluster volume heal VMARRAY info the file list cycles through most of my high IOPS machines<br>
<br>
Also what's the best way to apply those options with out bricking the running VM's? I just made a rough stab and took the cluster down.<br>
<div class="im"><br></div></blockquote><div><br></div><div class="gmail_default" style="font-family:tahoma,sans-serif">The CPU problem thing sounds a lot like what I ran into with my ovirt on gluster deployment (same boxes). What I did to solve that was use cgroups to limit the CPU usage glusterd and glusterfsd is allowed to use. [1]<br>
</div><div class="gmail_default" style="font-family:tahoma,sans-serif"><br></div><div class="gmail_default" style="font-family:tahoma,sans-serif">I'm not completely sure if libgfapi uses the glusterd process to access the storage, could someone else comment? However, I know by limiting glusterfsd we can slow down the replication process by limiting the CPU it sees, thus not bringing the entire system to a halt.</div>
<div class="gmail_default" style="font-family:tahoma,sans-serif"><br></div><div class="gmail_default" style="font-family:tahoma,sans-serif">[1] <a href="http://www.andrewklau.com/controlling-glusterfsd-cpu-outbreaks-with-cgroups/">http://www.andrewklau.com/controlling-glusterfsd-cpu-outbreaks-with-cgroups/</a></div>
<div class="gmail_default" style="font-family:tahoma,sans-serif"><br></div><div class="gmail_default" style="font-family:tahoma,sans-serif"><br></div><div class="gmail_default" style="font-family:tahoma,sans-serif"><br></div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div class="im">
----- Original Message -----<br>
From: "Vijay Bellur" <<a href="mailto:vbellur@redhat.com">vbellur@redhat.com</a>><br>
To: "Josh Boon" <<a href="mailto:gluster@joshboon.com">gluster@joshboon.com</a>>, "Nick Majeran" <<a href="mailto:nmajeran@gmail.com">nmajeran@gmail.com</a>><br>
Cc: "<a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a> List" <<a href="mailto:gluster-users@gluster.org">gluster-users@gluster.org</a>><br>
</div><div class="im">Sent: Saturday, March 22, 2014 1:36:09 PM<br>
Subject: Re: [Gluster-users] Optimizing Gluster (gfapi) for high IOPS<br>
<br>
</div><div class=""><div class="h5">On 03/21/2014 09:50 PM, Josh Boon wrote:<br>
> Hardware RAID 5 on SSD's using LVM formatted with XFS default options<br>
> mounted with noatime<br>
><br>
> Also I don't a lot of history for this current troubled machine but the<br>
> sysctl additions don't appear to have made a significant difference<br>
<br>
Performance tunables in [1] are normally recommended for qemu -<br>
libgfapi. The last two options are related to quorum and the remaining<br>
tunables are related to performance. It might be worth a check to see if<br>
these options help provide better performance.<br>
<br>
Do you happen to know if self-healing was in progress when the machines<br>
stall?<br>
<br>
-Vijay<br>
<br>
[1]<br>
<a href="https://github.com/gluster/glusterfs/blob/master/extras/group-virt.example" target="_blank">https://github.com/gluster/glusterfs/blob/master/extras/group-virt.example</a><br>
<br>
<br>
_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
<a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://supercolony.gluster.org/mailman/listinfo/gluster-users</a><br>
</div></div></blockquote></div><br></div></div>