<div><font><font face="verdana,sans-serif">Thanks for your comments.</font></font></div><div><font><font face="verdana,sans-serif"><br></font></font></div><font><font face="verdana,sans-serif">I use mdadm on many servers and I've seen md numbering like this a fair bit. Usually it occurs after a another RAID has been created and the numbering shifts. Neil Brown (mdadm's author) , seems to think it's fine. So I don't think that's the problem. And you're right - this is a Frankengluster made from a variety of chassis and controllers and normally it's fine. As Brian noted, it's all the same to gluster, mod some small local differences in IO performance.</font></font><div>
<font><font face="verdana,sans-serif"><br></font></font></div><div><font><font face="verdana,sans-serif">Re the size difference, I'll explicitly rebalance the brick after the fix-layout finishes, but I'm even more worried about this fantastic increase in CPU usage and its effect on user performance.</font></font></div>
<div><font><font face="verdana,sans-serif"><br></font></font></div><div><font><font face="verdana,sans-serif">In the fix-layout routines (still running), I've seen CPU usage of glusterfsd rise to ~400% and loadavg go up to >15 on all the servers (except the pbs3, the one that originally had that problem). That high load does not last long tho (maybe a few mintes - we've just installed nagios on these nodes and I'm getting a ton of emails about load increasing and then decreasing on all the nodes (except pbs3). When the load goes very high on a server node, the user-end performance drops appreciably.</font></font></div>
<div><font><font face="verdana,sans-serif"><br></font></font></div><div><font><font face="verdana,sans-serif">hjm</font></font></div>
<div><font><font face="verdana,sans-serif"><br></font></font></div><div><font><font face="verdana,sans-serif"><br>
</font></font><br><div class="gmail_quote">On Sat, Aug 11, 2012 at 4:20 AM, Brian Candler <span dir="ltr"><<a href="mailto:B.Candler@pobox.com" target="_blank">B.Candler@pobox.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
On Sat, Aug 11, 2012 at 12:11:39PM +0100, Nux! wrote:<br>
> On 10.08.2012 22:16, Harry Mangalam wrote:<br>
> >pbs3:/dev/md127 8.2T 5.9T 2.3T 73% /bducgl <---<br>
><br>
> Harry,<br>
><br>
> The name of that md device (127) indicated there may be something<br>
> dodgy going on there. A device shouldn't be named 127 unless some<br>
> problems occured. Are you sure your drives are OK?<br>
<br>
I have systems with /dev/md127 all the time, and there's no problem. It<br>
seems to number downwards from /dev/md127 - if I create md array on the same<br>
system it is /dev/md126.<br>
<br>
However, this does suggest that the nodes are not configured identically:<br>
two are /dev/sda or /dev/sdb, which suggests either plain disk or hardware<br>
RAID, while two are /dev/md0 or /dev/127, which is software RAID.<br>
<br>
Although this could explain performance differences between the nodes, this<br>
is transparent to gluster and doesn't explain why the files are unevenly<br>
balanced - unless there is one huge file which happens to have been<br>
allocated to this node.<br>
<br>
Regards,<br>
<br>
Brian.<br>
<br>
_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="http://gluster.org/cgi-bin/mailman/listinfo/gluster-users" target="_blank">http://gluster.org/cgi-bin/mailman/listinfo/gluster-users</a><br>
</blockquote></div><br><br clear="all"><div><br></div>-- <br>Harry Mangalam - Research Computing, OIT, Rm 225 MSTB, UC Irvine<br>[m/c 2225] / 92697 Google Voice Multiplexer: <a href="tel:%28949%29%20478-4487" value="+19494784487" target="_blank">(949) 478-4487</a><br>
415 South Circle View Dr, Irvine, CA, 92697 [shipping]<br>
MSTB Lat/Long: (33.642025,-117.844414) (paste into Google Maps)<br><br>
</div>