<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=us-ascii"><meta name=Generator content="Microsoft Word 14 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:PMingLiU;
panose-1:2 2 5 0 0 0 0 0 0 0;}
@font-face
{font-family:PMingLiU;
panose-1:2 2 5 0 0 0 0 0 0 0;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:PMingLiU;
panose-1:2 2 5 0 0 0 0 0 0 0;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";
color:black;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML \9810\8A2D\683C\5F0F \5B57\5143";
margin:0cm;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";
color:black;}
span.moz-smiley-s2
{mso-style-name:moz-smiley-s2;}
span.HTML
{mso-style-name:"HTML \9810\8A2D\683C\5F0F \5B57\5143";
mso-style-priority:99;
mso-style-link:"HTML \9810\8A2D\683C\5F0F";
font-family:"Courier New";
color:black;}
span.EmailStyle20
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 90.0pt 72.0pt 90.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body bgcolor=white lang=ZH-TW link=blue vlink=purple><div class=WordSection1><p class=MsoNormal><span lang=EN-US style='font-family:"Calibri","sans-serif";color:#1F497D'>Hi all,<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-family:"Calibri","sans-serif";color:#1F497D'>I’d like to know if we should be using “full” for self-heal algorithm rather than “diff” for storing large files especially for VM images?<br>I’ve tried healing a 100gb image by having one of the node to go down for a few minutes and bring it back up. The self-heal is in progress with one of the node’s CPU going up to 100% at all time for over 12 hours and continuing. I’m wondering if it is syncing anything or not. If I use “full”, I don’t get such an issue and the self-heal would complete in just about 10 minutes.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-family:"Calibri","sans-serif";color:#1F497D'>Thanks,<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-family:"Calibri","sans-serif";color:#1F497D'>Adrian<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><div><div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm'><p class=MsoNormal><b><span lang=EN-US style='font-size:10.0pt;font-family:"Tahoma","sans-serif";color:windowtext'>From:</span></b><span lang=EN-US style='font-size:10.0pt;font-family:"Tahoma","sans-serif";color:windowtext'> gluster-users-bounces@gluster.org [mailto:gluster-users-bounces@gluster.org] <b>On Behalf Of </b>Ivano Talamo<br><b>Sent:</b> Wednesday, May 28, 2014 3:19 PM<br><b>To:</b> Ravishankar N; gluster-users@gluster.org<br><b>Subject:</b> Re: [Gluster-users] gluster 3.4 self-heal<o:p></o:p></span></p></div></div><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><div><p class=MsoNormal><span lang=EN-US>Hi Ravishankar,<br><br>thank you for the explanation.<br>I expected a performance hit after such a long shutdown, the only problem is I couldn't understand<br>if the healing was going or not.<br>After launching the gluster volume heal vol1 full I can see files in the .glusterfs/indices/xattrop/ directory<br>to decrease, but to this rate it would take two weeks to finish, maybe I would rather delete and recreate the volume<br>from scratch and with 3.5.<br><br>Thanks<br>Ivano<br><br>On 5/27/14 7:35 PM, Ravishankar N wrote:<o:p></o:p></span></p></div><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt'><div><p class=MsoNormal><span lang=EN-US>On 05/27/2014 08:47 PM, Ivano Talamo wrote:<o:p></o:p></span></p></div><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt'><p class=MsoNormal style='margin-bottom:12.0pt'><span lang=EN-US>Dear all, <br><br>we have a replicated volume (2 servers with 1 brick each) on Scientific Linux 6.2 with gluster 3.4. <br>Everything was running fine until we shutdown of of the two and kept it down for 2 months. <br>When it came up again the volume could be healed and we have the following symptoms <br>(call #1 the always-up server, #2 the server that was kept down): <br><br>-doing I/O on the volume has very bad performances (impossible to keep VM images on it) <o:p></o:p></span></p></blockquote><p class=MsoNormal><span lang=EN-US>A replica's bricks are not supposed to be intentionally kept down even for hours, leave alone months <span class=moz-smiley-s2>:-( </span>; If you do; then when it does come backup, there would be tons of stuff to heal, so a performance hit is expected.<br><br><o:p></o:p></span></p><p class=MsoNormal style='margin-bottom:12.0pt'><span lang=EN-US>-on #1 there's 3997354 files on .glusterfs/indices/xattrop/ and the number doesn't go down <o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>When #2 was down, did the I/O involve directory renames? (see if there are entries on .glusterfs/landfill on #2). If yes then this is a known issue and a fix is in progress : <a href="http://review.gluster.org/#/c/7879/">http://review.gluster.org/#/c/7879/</a><br><br><br><o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>-on #1 gluster volume heal vol1 info the first time takes a lot to end and doesn't show nothing. <o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>This is fixed in glusterfs 3.5 where heal info is much more responsive.<br><br><o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>after that it prints "Another transaction is in progress. Please try again after sometime." <br><br>Furthermore on #1 glusterhd.log is full of messages like this: <br>[2014-05-27 15:07:44.145326] W [client-rpc-fops.c:1538:client3_3_inodelk_cbk] 0-vol1-client-0: remote operation failed: No such file or directory <br>[2014-05-27 15:07:44.145880] W [client-rpc-fops.c:1640:client3_3_entrylk_cbk] 0-vol1-client-0: remote operation failed: No such file or directory <br>[2014-05-27 15:07:44.146070] E [afr-self-heal-entry.c:2296:afr_sh_post_nonblocking_entry_cbk] 0-vol1-replicate-0: Non Blocking entrylks failed for <gfid:bfbe65db-7426-4ca0-bf0b-7d1a28de2052>. <br>[2014-05-27 15:13:34.772856] E [afr-self-heal-data.c:1270:afr_sh_data_open_cbk] 0-vol1-replicate-0: open of <gfid:18a358e0-23d3-4f56-8d74-f5cc38a0d0ea> failed on child vol1-client-0 (No such file or directory) <br><br>On #2 bricks I see some updates, ie. new filenames appearing and .glusterfs/indices/xattrop/ is usually empy. <br><br>Do you know what's happening? How can we fix this? <o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>You could try a `gluster volume heal vol1 full` to see if the bricks get synced.<br><br>Regards,<br>Ravi<br><br><o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><br>thank you, <br>Ivano <br><br><br><br><br><o:p></o:p></span></p><pre><span lang=EN-US>_______________________________________________<o:p></o:p></span></pre><pre><span lang=EN-US>Gluster-users mailing list<o:p></o:p></span></pre><pre><span lang=EN-US><a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><o:p></o:p></span></pre><pre><span lang=EN-US><a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users">http://supercolony.gluster.org/mailman/listinfo/gluster-users</a><o:p></o:p></span></pre><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p></blockquote><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p></div></body></html>