<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#FFFFFF">
On 02/03/2013 11:22 AM, Michael Colonno wrote:<br>
<blockquote cite="mid:015101ce0243$d464c000$7d2e4000$@stanford.edu"
type="cite">
<meta http-equiv="Content-Type" content="text/html;
charset=ISO-8859-1">
<meta name="Generator" content="Microsoft Word 14 (filtered
medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]-->
<style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p
{mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0in;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";}
p.MsoAcetate, li.MsoAcetate, div.MsoAcetate
{mso-style-priority:99;
mso-style-link:"Balloon Text Char";
margin:0in;
margin-bottom:.0001pt;
font-size:8.0pt;
font-family:"Tahoma","sans-serif";}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:Consolas;}
span.BalloonTextChar
{mso-style-name:"Balloon Text Char";
mso-style-priority:99;
mso-style-link:"Balloon Text";
font-family:"Tahoma","sans-serif";}
span.EmailStyle22
{mso-style-type:personal;
font-family:"Arial","sans-serif";
color:windowtext;
font-weight:normal;
font-style:normal;}
span.EmailStyle23
{mso-style-type:personal;
font-family:"Arial","sans-serif";
color:windowtext;
font-weight:normal;
font-style:normal;}
span.EmailStyle24
{mso-style-type:personal;
font-family:"Arial","sans-serif";
color:windowtext;
font-weight:normal;
font-style:normal;}
span.EmailStyle25
{mso-style-type:personal-reply;
font-family:"Arial","sans-serif";
color:windowtext;
font-weight:normal;
font-style:normal;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:175968176;
mso-list-type:hybrid;
mso-list-template-ids:186411800 1243623632 67698691 67698693 67698689 67698691 67698693 67698689 67698691 67698693;}
@list l0:level1
{mso-level-start-at:0;
mso-level-number-format:bullet;
mso-level-text:-;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:.75in;
text-indent:-.25in;
font-family:"Arial","sans-serif";
mso-fareast-font-family:Calibri;}
@list l0:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:1.25in;
text-indent:-.25in;
font-family:"Courier New";}
@list l0:level3
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:1.75in;
text-indent:-.25in;
font-family:Wingdings;}
@list l0:level4
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:2.25in;
text-indent:-.25in;
font-family:Symbol;}
@list l0:level5
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:2.75in;
text-indent:-.25in;
font-family:"Courier New";}
@list l0:level6
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:3.25in;
text-indent:-.25in;
font-family:Wingdings;}
@list l0:level7
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:3.75in;
text-indent:-.25in;
font-family:Symbol;}
@list l0:level8
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:4.25in;
text-indent:-.25in;
font-family:"Courier New";}
@list l0:level9
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:4.75in;
text-indent:-.25in;
font-family:Wingdings;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
<div class="WordSection1">
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Arial","sans-serif"">
Having taken a lot more data it does seem the glusterfsd and
glusterd processes (along with several ksoftirqd) spike up
to near 100% on both client and brick servers during any
file transport across the mount. Thankfully this is
short-lived for the most part but I’m wondering if this is
expected behavior or what others have experienced(?) I’m a
little surprised such a large CPU load would be required to
move small files and / or use an application within a
Gluster mount point. </span></p>
</div>
</blockquote>
<br>
If you're getting ksoftirqd spikes, that sounds like a hardware
issue to me. I never see huge spikes like that on my servers nor
clients.<br>
<br>
<blockquote cite="mid:015101ce0243$d464c000$7d2e4000$@stanford.edu"
type="cite">
<div class="WordSection1">
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Arial","sans-serif""><o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Arial","sans-serif"">
I wanted to test this against an NFS mount of the same
Gluster volume. I managed to get rstatd installed and
running but my attempts to mount the volume via NFS are met
with: <o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Arial","sans-serif"">
</span><span style="font-size:10.0pt;font-family:"Courier
New"">mount.nfs: requested NFS version or transport
protocol is not supported<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Arial","sans-serif"">
Relevant line in /etc/fstab:<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Arial","sans-serif"">
</span><span style="font-size:10.0pt;font-family:"Courier
New"">node1:/volume /volume nfs
defaults,_netdev,vers=3,mountproto=tcp 0 0 <o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></p>
<p class="MsoNormal" style="text-indent:.5in"><span
style="font-size:10.0pt;font-family:"Arial","sans-serif"">It
looks like CentOS 6.x has NFS version 4 built into
everything. So a few questions:<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></p>
<p class="MsoListParagraph"
style="margin-left:.75in;text-indent:-.25in;mso-list:l0 level1
lfo1"><!--[if !supportLists]--><span
style="font-size:10.0pt;font-family:"Arial","sans-serif""><span
style="mso-list:Ignore">-<span style="font:7.0pt
"Times New Roman""> </span></span></span><!--[endif]--><span
style="font-size:10.0pt;font-family:"Arial","sans-serif"">Has
anyone else noted significant performance differences
between a glusterfs mount and NFS mount for volumes of 8+
bricks? <o:p></o:p></span></p>
<p class="MsoListParagraph"
style="margin-left:.75in;text-indent:-.25in;mso-list:l0 level1
lfo1"><!--[if !supportLists]--><span
style="font-size:10.0pt;font-family:"Arial","sans-serif""><span
style="mso-list:Ignore">-<span style="font:7.0pt
"Times New Roman""> </span></span></span><!--[endif]--><span
style="font-size:10.0pt;font-family:"Arial","sans-serif"">Is
there a straightforward way to make the newer versions of
CentOS play nice with NFS version 3 + Gluster? <o:p></o:p></span></p>
<p class="MsoListParagraph"
style="margin-left:.75in;text-indent:-.25in;mso-list:l0 level1
lfo1"><!--[if !supportLists]--><span
style="font-size:10.0pt;font-family:"Arial","sans-serif""><span
style="mso-list:Ignore">-<span style="font:7.0pt
"Times New Roman""> </span></span></span><!--[endif]--><span
style="font-size:10.0pt;font-family:"Arial","sans-serif"">Are
there any general performance tuning guidelines I can follow
to improve CPU performance? I found a few references to the
cache settings but nothing solid. <o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span
style="font-size:10.0pt;font-family:"Arial","sans-serif"">If
the consensus is that NFS will not gain anything then I
won’t waste the time setting it all up. </span></p>
</div>
</blockquote>
<br>
NFS gains you the use of FSCache to cache directories and file stats
making directory listings faster, but it adds overhead decreasing
the overall throughput (from all the reports I've seen).<br>
<br>
I would suspect that you have the kernel nfs server running on your
servers. Make sure it's disabled.<br>
<br>
<blockquote cite="mid:015101ce0243$d464c000$7d2e4000$@stanford.edu"
type="cite">
<div class="WordSection1">
<p class="MsoNormal" style="margin-left:.5in"><span
style="font-size:10.0pt;font-family:"Arial","sans-serif""><o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span
style="font-size:10.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span
style="font-size:10.0pt;font-family:"Arial","sans-serif"">Thanks,<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span
style="font-size:10.0pt;font-family:"Arial","sans-serif"">~Mike
C. <o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></p>
<p class="MsoNormal"><a moz-do-not-send="true"
name="_MailEndCompose"><span
style="font-size:10.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></a></p>
<div>
<div style="border:none;border-top:solid #B5C4DF
1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span
style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span
style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">
<a class="moz-txt-link-abbreviated" href="mailto:gluster-users-bounces@gluster.org">gluster-users-bounces@gluster.org</a>
[<a class="moz-txt-link-freetext" href="mailto:gluster-users-bounces@gluster.org">mailto:gluster-users-bounces@gluster.org</a>] <b>On Behalf
Of </b>Michael Colonno<br>
<b>Sent:</b> Friday, February 01, 2013 4:46 PM<br>
<b>To:</b> <a class="moz-txt-link-abbreviated" href="mailto:gluster-users@gluster.org">gluster-users@gluster.org</a><br>
<b>Subject:</b> Re: [Gluster-users] high CPU load on all
bricks<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Arial","sans-serif"">
Update: after a few hours the CPU usage seems to have
dropped down to a small value. I did not change anything
with respect to the configuration or unmount / stop anything
as I wanted to see if this would persist for a long period
of time. Both the client and the self-mounted bricks are now
showing CPU < 1% (as reported by top). Prior to the
larger CPU loads I installed a bunch of software into the
volume (~ 5 GB total). Is this kind a transient behavior –
by which I mean larger CPU loads after a lot of filesystem
activity in short time – typical? This is not a problem in
my deployment; I just want to know what to expect in the
future and to complete this thread for future users. If this
is expected behavior we can wrap up this thread. If not then
I’ll do more digging into the logs on the client and brick
sides. <o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Arial","sans-serif"">
Thanks,<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Arial","sans-serif"">
~Mike C. <o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #B5C4DF
1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span
style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span
style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">
Joe Julian [<a moz-do-not-send="true"
href="mailto:joe@julianfamily.org">mailto:joe@julianfamily.org</a>]
<br>
<b>Sent:</b> Friday, February 01, 2013 2:08 PM<br>
<b>To:</b> Michael Colonno; <a moz-do-not-send="true"
href="mailto:gluster-users@gluster.org">gluster-users@gluster.org</a><br>
<b>Subject:</b> Re: [Gluster-users] high CPU load on all
bricks<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal" style="margin-bottom:12.0pt">Check the
client log(s). <o:p></o:p></p>
<div>
<p class="MsoNormal">Michael Colonno <<a
moz-do-not-send="true" href="mailto:mcolonno@stanford.edu">mcolonno@stanford.edu</a>>
wrote:<o:p></o:p></p>
<p class="MsoNormal"> Forgot to mention: on a
client system (not a brick) the glusterfs process is
consuming ~ 68% CPU continuously. This is a much less
powerful desktop system so the CPU load can’t be compared
1:1 with the systems comprising the bricks but still very
high. So the issue seems to exist with both glusterfsd and
glusterfs processes. <o:p></o:p></p>
<p> <o:p></o:p></p>
<p class="MsoNormal"> Thanks,<o:p></o:p></p>
<p class="MsoNormal"> ~Mike C. <o:p></o:p></p>
<p> <o:p></o:p></p>
<div>
<div style="border:none;border-top:solid #B5C4DF
1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b>From:</b> <a
moz-do-not-send="true"
href="mailto:gluster-users-bounces@gluster.org">gluster-users-bounces@gluster.org</a>
[<a moz-do-not-send="true"
href="mailto:gluster-users-bounces@gluster.org">mailto:gluster-users-bounces@gluster.org</a>]
<b>On Behalf Of </b>Michael Colonno<br>
<b>Sent:</b> Friday, February 01, 2013 12:46 PM<br>
<b>To:</b> <a moz-do-not-send="true"
href="mailto:gluster-users@gluster.org">gluster-users@gluster.org</a><br>
<b>Subject:</b> [Gluster-users] high CPU load on all
bricks<o:p></o:p></p>
</div>
</div>
<p> <o:p></o:p></p>
<p class="MsoNormal"> Gluster gurus ~<o:p></o:p></p>
<p> <o:p></o:p></p>
<p class="MsoNormal"> I’ve deployed and 8-brick (2x
replicate) Gluster 3.3.1 volume on CentOS 6.3 with tcp
transport. I was able to build, start, mount, and use the
volume. On each system contributing a brick, however, my CPU
usage (glusterfsd) is hovering around 20% (virtually zero
memory usage thankfully). These are brand new, fairly beefy
servers so 20% CPU load is quite a bit. The deployment is
pretty plain with each brick mounting the volume to itself
via a glusterfs mount. I assume this type of CPU usage is
atypically high; is there anything I can do to investigate
what’s soaking up CPU and minimize it? Total usable volume
size is only about 22 TB (about 45 TB total with 2x
replicate). <o:p></o:p></p>
<p> <o:p></o:p></p>
<p class="MsoNormal"> Thanks,<o:p></o:p></p>
<p class="MsoNormal"> ~Mike C. <o:p></o:p></p>
<p> <o:p></o:p></p>
<pre style="text-align:center;white-space:pre-wrap;word-wrap:break-word"><span style="font-family:"Arial","sans-serif""><hr align="center" size="2" width="100%"></span></pre>
<pre><span style="font-family:"Arial","sans-serif"">
Gluster-users mailing list
<a moz-do-not-send="true" href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a>
<a moz-do-not-send="true" href="http://supercolony.gluster.org/mailman/listinfo/gluster-users">http://supercolony.gluster.org/mailman/listinfo/gluster-users</a><o:p></o:p></span></pre>
</div>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
Gluster-users mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a>
<a class="moz-txt-link-freetext" href="http://supercolony.gluster.org/mailman/listinfo/gluster-users">http://supercolony.gluster.org/mailman/listinfo/gluster-users</a></pre>
</blockquote>
<br>
</body>
</html>