<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=us-ascii"><meta name=Generator content="Microsoft Word 14 (filtered medium)"><!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";
color:black;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p
{mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:12.0pt;
font-family:"Times New Roman","serif";
color:black;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0in;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";
color:black;}
p.MsoAcetate, li.MsoAcetate, div.MsoAcetate
{mso-style-priority:99;
mso-style-link:"Balloon Text Char";
margin:0in;
margin-bottom:.0001pt;
font-size:8.0pt;
font-family:"Tahoma","sans-serif";
color:black;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";
color:black;}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:Consolas;}
span.BalloonTextChar
{mso-style-name:"Balloon Text Char";
mso-style-priority:99;
mso-style-link:"Balloon Text";
font-family:"Tahoma","sans-serif";}
span.EmailStyle23
{mso-style-type:personal;
font-family:"Arial","sans-serif";
color:windowtext;
font-weight:normal;
font-style:normal;}
span.EmailStyle24
{mso-style-type:personal;
font-family:"Arial","sans-serif";
color:windowtext;
font-weight:normal;
font-style:normal;}
span.EmailStyle25
{mso-style-type:personal;
font-family:"Arial","sans-serif";
color:windowtext;
font-weight:normal;
font-style:normal;}
span.EmailStyle26
{mso-style-type:personal;
font-family:"Arial","sans-serif";
color:windowtext;
font-weight:normal;
font-style:normal;}
span.EmailStyle27
{mso-style-type:personal-reply;
font-family:"Arial","sans-serif";
color:windowtext;
font-weight:normal;
font-style:normal;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:175968176;
mso-list-type:hybrid;
mso-list-template-ids:186411800 1243623632 67698691 67698693 67698689 67698691 67698693 67698689 67698691 67698693;}
@list l0:level1
{mso-level-start-at:0;
mso-level-number-format:bullet;
mso-level-text:-;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:.75in;
text-indent:-.25in;
font-family:"Arial","sans-serif";
mso-fareast-font-family:Calibri;}
@list l0:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:1.25in;
text-indent:-.25in;
font-family:"Courier New";}
@list l0:level3
{mso-level-number-format:bullet;
mso-level-text:"&\#61607";
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:1.75in;
text-indent:-.25in;
font-family:Wingdings;}
@list l0:level4
{mso-level-number-format:bullet;
mso-level-text:"&\#61623";
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:2.25in;
text-indent:-.25in;
font-family:Symbol;}
@list l0:level5
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:2.75in;
text-indent:-.25in;
font-family:"Courier New";}
@list l0:level6
{mso-level-number-format:bullet;
mso-level-text:"&\#61607";
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:3.25in;
text-indent:-.25in;
font-family:Wingdings;}
@list l0:level7
{mso-level-number-format:bullet;
mso-level-text:"&\#61623";
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:3.75in;
text-indent:-.25in;
font-family:Symbol;}
@list l0:level8
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:4.25in;
text-indent:-.25in;
font-family:"Courier New";}
@list l0:level9
{mso-level-number-format:bullet;
mso-level-text:"&\#61607";
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:4.75in;
text-indent:-.25in;
font-family:Wingdings;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body bgcolor=white lang=EN-US link=blue vlink=purple><div class=WordSection1><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif";color:windowtext'> Follow up: still seeing extremely high CPU loads on all bricks during any file activity. e.g. from top:<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif";color:windowtext'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif";color:windowtext'> </span><span style='font-size:10.0pt;font-family:"Courier New";color:windowtext'>630 root 20 0 319m 85m 2676 R <span style='background:yellow;mso-highlight:yellow'>134.8</span> 0.3 1023:21 glusterfs<o:p></o:p></span></p><p class=MsoNormal style='text-indent:.5in'><span style='font-size:10.0pt;font-family:"Courier New";color:windowtext'>26486 root 20 0 1921m 72m 2124 S <span style='background:yellow;mso-highlight:yellow'>41.5</span> 0.2 418:27.29 glusterfsd<o:p></o:p></span></p><p class=MsoNormal style='text-indent:.5in'><span style='font-size:10.0pt;font-family:"Arial","sans-serif";color:windowtext'><o:p> </o:p></span></p><p class=MsoNormal style='text-indent:.5in'><span style='font-size:10.0pt;font-family:"Arial","sans-serif";color:windowtext'>The 134.8% CPU is curious reporting (peaks up to 200% or so) but these readings are typical of any of the eight brick systems in the volume when either files are being read / written or an application stored in the volume is in use on any of the systems in the brick. Is this kind of load expected in an eight-brick 2x replicated volume? When I grabbed the readings above no files were being accesses but an application file was in use. Is there any way I can get more detailed performance metrics while in use? <o:p></o:p></span></p><p class=MsoNormal style='text-indent:.5in'><span style='font-size:10.0pt;font-family:"Arial","sans-serif";color:windowtext'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif";color:windowtext'> Thanks,<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif";color:windowtext'> ~Mike C. <o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif";color:windowtext'><o:p> </o:p></span></p><p class=MsoNormal><a name="_MailEndCompose"><span style='font-size:10.0pt;font-family:"Arial","sans-serif";color:windowtext'><o:p> </o:p></span></a></p><div><div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in'><p class=MsoNormal><b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif";color:windowtext'>From:</span></b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif";color:windowtext'> gluster-users-bounces@gluster.org [mailto:gluster-users-bounces@gluster.org] <b>On Behalf Of </b>Joe Julian<br><b>Sent:</b> Sunday, February 03, 2013 11:47 AM<br><b>To:</b> gluster-users@gluster.org<br><b>Subject:</b> Re: [Gluster-users] high CPU load on all bricks<o:p></o:p></span></p></div></div><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>On 02/03/2013 11:22 AM, Michael Colonno wrote:<br><br><o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'> Having taken a lot more data it does seem the glusterfsd and glusterd processes (along with several ksoftirqd) spike up to near 100% on both client and brick servers during any file transport across the mount. Thankfully this is short-lived for the most part but I’m wondering if this is expected behavior or what others have experienced(?) I’m a little surprised such a large CPU load would be required to move small files and / or use an application within a Gluster mount point. </span><o:p></o:p></p><p class=MsoNormal><br>If you're getting ksoftirqd spikes, that sounds like a hardware issue to me. I never see huge spikes like that on my servers nor clients.<br><br><br><o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'> </span><o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'> I wanted to test this against an NFS mount of the same Gluster volume. I managed to get rstatd installed and running but my attempts to mount the volume via NFS are met with: </span><o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'> </span><o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'> </span><span style='font-size:10.0pt;font-family:"Courier New"'>mount.nfs: requested NFS version or transport protocol is not supported</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'> </span><o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'> Relevant line in /etc/fstab:</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'> </span><o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'> </span><span style='font-size:10.0pt;font-family:"Courier New"'>node1:/volume /volume nfs defaults,_netdev,vers=3,mountproto=tcp 0 0 </span><o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'> </span><o:p></o:p></p><p class=MsoNormal style='text-indent:.5in'><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'>It looks like CentOS 6.x has NFS version 4 built into everything. So a few questions:</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'> </span><o:p></o:p></p><p class=MsoListParagraph style='margin-left:.75in;text-indent:-.25in;mso-list:l0 level1 lfo2'><![if !supportLists]><span style='font-family:"Arial","sans-serif"'><span style='mso-list:Ignore'>-<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'>Has anyone else noted significant performance differences between a glusterfs mount and NFS mount for volumes of 8+ bricks? </span><o:p></o:p></p><p class=MsoListParagraph style='margin-left:.75in;text-indent:-.25in;mso-list:l0 level1 lfo2'><![if !supportLists]><span style='font-family:"Arial","sans-serif"'><span style='mso-list:Ignore'>-<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'>Is there a straightforward way to make the newer versions of CentOS play nice with NFS version 3 + Gluster? </span><o:p></o:p></p><p class=MsoListParagraph style='margin-left:.75in;text-indent:-.25in;mso-list:l0 level1 lfo2'><![if !supportLists]><span style='font-family:"Arial","sans-serif"'><span style='mso-list:Ignore'>-<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'>Are there any general performance tuning guidelines I can follow to improve CPU performance? I found a few references to the cache settings but nothing solid. </span><o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'> </span><o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'>If the consensus is that NFS will not gain anything then I won’t waste the time setting it all up. </span><o:p></o:p></p><p class=MsoNormal><br>NFS gains you the use of FSCache to cache directories and file stats making directory listings faster, but it adds overhead decreasing the overall throughput (from all the reports I've seen).<br><br>I would suspect that you have the kernel nfs server running on your servers. Make sure it's disabled.<br><br><br><o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'> </span><o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'>Thanks,</span><o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'>~Mike C. </span><o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'> </span><o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'> </span><o:p></o:p></p><div><div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in'><p class=MsoNormal><b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>From:</span></b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'> <a href="mailto:gluster-users-bounces@gluster.org">gluster-users-bounces@gluster.org</a> [<a href="mailto:gluster-users-bounces@gluster.org">mailto:gluster-users-bounces@gluster.org</a>] <b>On Behalf Of </b>Michael Colonno<br><b>Sent:</b> Friday, February 01, 2013 4:46 PM<br><b>To:</b> <a href="mailto:gluster-users@gluster.org">gluster-users@gluster.org</a><br><b>Subject:</b> Re: [Gluster-users] high CPU load on all bricks</span><o:p></o:p></p></div></div><p class=MsoNormal> <o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'> Update: after a few hours the CPU usage seems to have dropped down to a small value. I did not change anything with respect to the configuration or unmount / stop anything as I wanted to see if this would persist for a long period of time. Both the client and the self-mounted bricks are now showing CPU < 1% (as reported by top). Prior to the larger CPU loads I installed a bunch of software into the volume (~ 5 GB total). Is this kind a transient behavior – by which I mean larger CPU loads after a lot of filesystem activity in short time – typical? This is not a problem in my deployment; I just want to know what to expect in the future and to complete this thread for future users. If this is expected behavior we can wrap up this thread. If not then I’ll do more digging into the logs on the client and brick sides. </span><o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'> </span><o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'> Thanks,</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'> ~Mike C. </span><o:p></o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'> </span><o:p></o:p></p><div><div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in'><p class=MsoNormal><b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>From:</span></b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'> Joe Julian [<a href="mailto:joe@julianfamily.org">mailto:joe@julianfamily.org</a>] <br><b>Sent:</b> Friday, February 01, 2013 2:08 PM<br><b>To:</b> Michael Colonno; <a href="mailto:gluster-users@gluster.org">gluster-users@gluster.org</a><br><b>Subject:</b> Re: [Gluster-users] high CPU load on all bricks</span><o:p></o:p></p></div></div><p class=MsoNormal> <o:p></o:p></p><p class=MsoNormal style='margin-bottom:12.0pt'>Check the client log(s). <o:p></o:p></p><div><p class=MsoNormal>Michael Colonno <<a href="mailto:mcolonno@stanford.edu">mcolonno@stanford.edu</a>> wrote:<o:p></o:p></p><p class=MsoNormal> Forgot to mention: on a client system (not a brick) the glusterfs process is consuming ~ 68% CPU continuously. This is a much less powerful desktop system so the CPU load can’t be compared 1:1 with the systems comprising the bricks but still very high. So the issue seems to exist with both glusterfsd and glusterfs processes. <o:p></o:p></p><p> <o:p></o:p></p><p class=MsoNormal> Thanks,<o:p></o:p></p><p class=MsoNormal> ~Mike C. <o:p></o:p></p><p> <o:p></o:p></p><div><div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in'><p class=MsoNormal><b>From:</b> <a href="mailto:gluster-users-bounces@gluster.org">gluster-users-bounces@gluster.org</a> [<a href="mailto:gluster-users-bounces@gluster.org">mailto:gluster-users-bounces@gluster.org</a>] <b>On Behalf Of </b>Michael Colonno<br><b>Sent:</b> Friday, February 01, 2013 12:46 PM<br><b>To:</b> <a href="mailto:gluster-users@gluster.org">gluster-users@gluster.org</a><br><b>Subject:</b> [Gluster-users] high CPU load on all bricks<o:p></o:p></p></div></div><p> <o:p></o:p></p><p class=MsoNormal> Gluster gurus ~<o:p></o:p></p><p> <o:p></o:p></p><p class=MsoNormal> I’ve deployed and 8-brick (2x replicate) Gluster 3.3.1 volume on CentOS 6.3 with tcp transport. I was able to build, start, mount, and use the volume. On each system contributing a brick, however, my CPU usage (glusterfsd) is hovering around 20% (virtually zero memory usage thankfully). These are brand new, fairly beefy servers so 20% CPU load is quite a bit. The deployment is pretty plain with each brick mounting the volume to itself via a glusterfs mount. I assume this type of CPU usage is atypically high; is there anything I can do to investigate what’s soaking up CPU and minimize it? Total usable volume size is only about 22 TB (about 45 TB total with 2x replicate). <o:p></o:p></p><p> <o:p></o:p></p><p class=MsoNormal> Thanks,<o:p></o:p></p><p class=MsoNormal> ~Mike C. <o:p></o:p></p><p> <o:p></o:p></p><pre style='text-align:center;white-space:pre-wrap;word-wrap:break-word'><span style='font-family:"Arial","sans-serif"'><hr size=2 width="100%" align=center></span></pre><pre><span style='font-family:"Arial","sans-serif"'><o:p> </o:p></span></pre><pre><span style='font-family:"Arial","sans-serif"'>Gluster-users mailing list<o:p></o:p></span></pre><pre><span style='font-family:"Arial","sans-serif"'><a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><o:p></o:p></span></pre><pre><span style='font-family:"Arial","sans-serif"'><a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users">http://supercolony.gluster.org/mailman/listinfo/gluster-users</a></span><o:p></o:p></pre></div><p class=MsoNormal><br><br><br><o:p></o:p></p><pre>_______________________________________________<o:p></o:p></pre><pre>Gluster-users mailing list<o:p></o:p></pre><pre><a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><o:p></o:p></pre><pre><a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users">http://supercolony.gluster.org/mailman/listinfo/gluster-users</a><o:p></o:p></pre><p class=MsoNormal><o:p> </o:p></p></div></body></html>