<font><font face="verdana,sans-serif"><br></font></font><br><div class="gmail_quote">On Sat, Aug 11, 2012 at 9:41 AM, Brian Candler <span dir="ltr"><<a href="mailto:B.Candler@pobox.com" target="_blank">B.Candler@pobox.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">On Sat, Aug 11, 2012 at 08:31:51AM -0700, Harry Mangalam wrote:<br>
> Re the size difference, I'll explicitly rebalance the brick after the<br>
> fix-layout finishes, but I'm even more worried about this fantastic<br>
> increase in CPU usage and its effect on user performance.<br>
<br>
This presumably means you were originally running the cluster with fewer<br>
nodes, and then added some later?<br></blockquote><div><br></div><div>No, but the unbalanced current situation suggests that at some point, it got out of balance.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
> In the fix-layout routines (still running), I've seen CPU usage of<br>
> glusterfsd rise to ~400% and loadavg go up to >15 on all the servers<br>
> (except the pbs3, the one that originally had that problem). That high<br>
> load does not last long tho (maybe a few mintes - we've just installed<br>
> nagios on these nodes and I'm getting a ton of emails about load<br>
> increasing and then decreasing on all the nodes (except pbs3). When<br>
> the load goes very high on a server node, the user-end performance<br>
> drops appreciably.<br>
<br>
Maybe worth trying an strace (strace -f -p <pid> 2>strace.out) on the<br>
glusterfsd process, or whatever it is which is causing the high load, during<br>
such a burst, just for a few seconds. The output might give some clues.<br></blockquote><div><br></div><div>Good idea. I'll watch and when it goes wacko and post the filtered results.</div><div><br></div><div>Thanks</div>
<div>Harry</div></div><br><br clear="all"><div><br></div>-- <br>Harry Mangalam - Research Computing, OIT, Rm 225 MSTB, UC Irvine<br>[m/c 2225] / 92697 Google Voice Multiplexer: (949) 478-4487<br>415 South Circle View Dr, Irvine, CA, 92697 [shipping]<br>
MSTB Lat/Long: (33.642025,-117.844414) (paste into Google Maps)<br><br>