<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=us-ascii"><meta name=Generator content="Microsoft Word 14 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:PMingLiU;
panose-1:2 2 5 0 0 0 0 0 0 0;}
@font-face
{font-family:PMingLiU;
panose-1:2 2 5 0 0 0 0 0 0 0;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:PMingLiU;
panose-1:2 2 5 0 0 0 0 0 0 0;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";
color:black;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML \9810\8A2D\683C\5F0F \5B57\5143";
margin:0cm;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";
color:black;}
span.HTML
{mso-style-name:"HTML \9810\8A2D\683C\5F0F \5B57\5143";
mso-style-priority:99;
mso-style-link:"HTML \9810\8A2D\683C\5F0F";
font-family:"Courier New";
color:black;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 90.0pt 72.0pt 90.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body bgcolor=white lang=ZH-TW link=blue vlink=purple><div class=WordSection1><p class=MsoNormal><span lang=EN-US style='font-family:"Calibri","sans-serif";color:#1F497D'>Actually I had the same experience when I was using 3.4.2<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-family:"Calibri","sans-serif";color:#1F497D'><a href="https://www.mail-archive.com/gluster-users@gluster.org/msg15850.html">https://www.mail-archive.com/gluster-users@gluster.org/msg15850.html</a><o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-family:"Calibri","sans-serif";color:#1F497D'>If I understand, I should be using FULL heal rather than DIFF for large vm-images?<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-family:"Calibri","sans-serif";color:#1F497D'>I was not sure throttling was working for 3.4.2 or not. I attempted to recover the entire volume filled with VM-images ranging from size 10G to 500G<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-family:"Calibri","sans-serif";color:#1F497D'>I saw it was recovering 2 images at a time rather than all at once.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-family:"Calibri","sans-serif";color:#1F497D'>Thanks,<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-family:"Calibri","sans-serif";color:#1F497D'>Adrian<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><div><div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm'><p class=MsoNormal><b><span lang=EN-US style='font-size:10.0pt;font-family:"Tahoma","sans-serif";color:windowtext'><o:p> </o:p></span></b></p><p class=MsoNormal><b><span lang=EN-US style='font-size:10.0pt;font-family:"Tahoma","sans-serif";color:windowtext'><o:p> </o:p></span></b></p><p class=MsoNormal><b><span lang=EN-US style='font-size:10.0pt;font-family:"Tahoma","sans-serif";color:windowtext'>From:</span></b><span lang=EN-US style='font-size:10.0pt;font-family:"Tahoma","sans-serif";color:windowtext'> gluster-users-bounces@gluster.org [mailto:gluster-users-bounces@gluster.org] <b>On Behalf Of </b>Pranith Kumar Karampuri<br><b>Sent:</b> Tuesday, November 18, 2014 7:02 PM<br><b>To:</b> Lindsay Mathieson; gluster-users<br><b>Subject:</b> Re: [Gluster-users] glusterfsd process thrashing CPU<o:p></o:p></span></p></div></div><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><div><p class=MsoNormal><span lang=EN-US>On 11/18/2014 04:14 PM, Lindsay Mathieson wrote:<o:p></o:p></span></p></div><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt'><pre><span lang=EN-US>On Tue, 18 Nov 2014 02:36:19 PM Pranith Kumar Karampuri wrote:<o:p></o:p></span></pre><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt'><pre><span lang=EN-US>On 11/18/2014 01:17 PM, Lindsay Mathieson wrote:<o:p></o:p></span></pre><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt'><pre><span lang=EN-US>On 18 November 2014 17:40, Pranith Kumar Karampuri <a href="mailto:pkarampu@redhat.com"><pkarampu@redhat.com></a> <o:p></o:p></span></pre></blockquote></blockquote><pre><span lang=EN-US>wrote:<o:p></o:p></span></pre><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt'><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt'><pre><span lang=EN-US><o:p> </o:p></span></pre><pre><span lang=EN-US>However given the files are tens of GB in size, won't it thrash my<o:p></o:p></span></pre><pre><span lang=EN-US>network?<o:p></o:p></span></pre></blockquote><pre><span lang=EN-US><o:p> </o:p></span></pre><pre><span lang=EN-US>Yes you are right. I wonder why thrashing of the network is never<o:p></o:p></span></pre><pre><span lang=EN-US>reported till now.<o:p></o:p></span></pre></blockquote><pre><span lang=EN-US><o:p> </o:p></span></pre><pre><span lang=EN-US>Not sure if you are being sarcastic or not :) But from what I've observed, <o:p></o:p></span></pre><pre><span lang=EN-US>sync operations seem to self throttle, I've not seen them use more than 50% of <o:p></o:p></span></pre><pre><span lang=EN-US>bandwidth, and given most setups have a dedicated network for the servers <o:p></o:p></span></pre><pre><span lang=EN-US>maybe they just don't notice if it takes a while?<o:p></o:p></span></pre></blockquote><p class=MsoNormal><span lang=EN-US>No, I was not being sarcastic :-). I am genuinely wondering why it is not reported till now. May be Joe will have more inputs there, that is the reason I CCed him.<br><br><o:p></o:p></span></p><pre><span lang=EN-US><o:p> </o:p></span></pre><pre><span lang=EN-US><o:p> </o:p></span></pre><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt'><pre><span lang=EN-US>I still need to think about how best to solve this problem.<o:p></o:p></span></pre></blockquote><pre><span lang=EN-US><o:p> </o:p></span></pre><pre><span lang=EN-US>Setup a array of queues for self healing, sorted by size maybe?<o:p></o:p></span></pre><pre><span lang=EN-US><o:p> </o:p></span></pre><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt'><pre><span lang=EN-US><o:p> </o:p></span></pre><pre><span lang=EN-US>Let me tell you a bit more about this issue:<o:p></o:p></span></pre><pre><span lang=EN-US>there are two processes which heal the VM images:<o:p></o:p></span></pre><pre><span lang=EN-US>1) self-heal-daemon. 2) Mount process.<o:p></o:p></span></pre><pre><span lang=EN-US>Self-heal daemon heals one VM image at a time. But mount process<o:p></o:p></span></pre><pre><span lang=EN-US>triggers self-heals for all the opened files(VM image is nothing but an<o:p></o:p></span></pre><pre><span lang=EN-US>opened file from filesystem's perspective) when a brick goes down and<o:p></o:p></span></pre><pre><span lang=EN-US>comes backup.<o:p></o:p></span></pre></blockquote><pre><span lang=EN-US><o:p> </o:p></span></pre><pre><span lang=EN-US><o:p> </o:p></span></pre><pre><span lang=EN-US>Thanks, interesting to know.<o:p></o:p></span></pre><pre><span lang=EN-US><o:p> </o:p></span></pre><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt'><pre><span lang=EN-US>So we need to come up with a scheme to throttle self-heals<o:p></o:p></span></pre><pre><span lang=EN-US>on the mount point to prevent this issue. I will update you as soon as I<o:p></o:p></span></pre><pre><span lang=EN-US>come up with a fix. This should not be hard to do. Need some time to<o:p></o:p></span></pre><pre><span lang=EN-US>choose the best approach. Thanks a lot for bringing up this issue.<o:p></o:p></span></pre></blockquote><pre><span lang=EN-US><o:p> </o:p></span></pre><pre><span lang=EN-US>Thanks you for looking at it!<o:p></o:p></span></pre><pre><span lang=EN-US><o:p> </o:p></span></pre><pre><span lang=EN-US>Cheers,<o:p></o:p></span></pre><pre><span lang=EN-US><o:p> </o:p></span></pre><pre><span lang=EN-US><o:p> </o:p></span></pre><p class=MsoNormal><span lang=EN-US><br><br><br><o:p></o:p></span></p><pre><span lang=EN-US>_______________________________________________<o:p></o:p></span></pre><pre><span lang=EN-US>Gluster-users mailing list<o:p></o:p></span></pre><pre><span lang=EN-US><a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><o:p></o:p></span></pre><pre><span lang=EN-US><a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users">http://supercolony.gluster.org/mailman/listinfo/gluster-users</a><o:p></o:p></span></pre><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p></div></body></html>