<font><font face="verdana,sans-serif"><div>running 3.3 distributed on IPoIB on 4 nodes, 1 brick per node. Any idea why, on one of those nodes, glusterfsd would go berserk, running up to 370% CPU and driving load to >30 (file performance on the clients slows to a crawl). While very slow, it continued to serve out files. This is the second time this has happened in about a week. I had turned on the gluster nfs services, but wasn't using it when this happened. It's now off.</div>
<div><br></div><div>kill -HUP did nothing to either glusterd or glusterfsd, so I had to kill both and restart glusterd. That solved the overload on glusterfsd and performance is back to near normal. I'm now doing a rebalance/fix-layout which is running as expected, but will take the weekend to complete. I did notice that the affected node (pbs3) has more files than the others, tho I'm not sure that this is significant.</div>
<div><br></div><div>Filesystem Size Used Avail Use% Mounted on</div><div>pbs1:/dev/sdb 6.4T 1.9T 4.6T 29% /bducgl</div><div>pbs2:/dev/md0 8.2T 2.4T 5.9T 30% /bducgl</div><div>pbs3:/dev/md127 8.2T 5.9T 2.3T 73% /bducgl <---</div>
<div>pbs4:/dev/sda 6.4T 1.8T 4.6T 29% /bducgl</div><div><br></div></font></font><div><br></div>-- <br>Harry Mangalam - Research Computing, OIT, Rm 225 MSTB, UC Irvine<br>[m/c 2225] / 92697 Google Voice Multiplexer: (949) 478-4487<br>
415 South Circle View Dr, Irvine, CA, 92697 [shipping]<br>MSTB Lat/Long: (33.642025,-117.844414) (paste into Google Maps)<br><br>