<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoAcetate, li.MsoAcetate, div.MsoAcetate
{mso-style-priority:99;
mso-style-link:"Balloon Text Char";
margin:0in;
margin-bottom:.0001pt;
font-size:8.0pt;
font-family:"Tahoma","sans-serif";}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
span.EmailStyle18
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:windowtext;}
span.EmailStyle19
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.BalloonTextChar
{mso-style-name:"Balloon Text Char";
mso-style-priority:99;
mso-style-link:"Balloon Text";
font-family:"Tahoma","sans-serif";}
span.EmailStyle22
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:1449936980;
mso-list-type:hybrid;
mso-list-template-ids:1306133326 -548366592 67698691 67698693 67698689 67698691 67698693 67698689 67698691 67698693;}
@list l0:level1
{mso-level-number-format:bullet;
mso-level-text:\F0D8;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;
mso-fareast-font-family:Calibri;
mso-bidi-font-family:"Times New Roman";}
@list l0:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Courier New";}
@list l0:level3
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;}
@list l0:level4
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Symbol;}
@list l0:level5
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Courier New";}
@list l0:level6
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;}
@list l0:level7
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Symbol;}
@list l0:level8
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Courier New";}
@list l0:level9
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="color:#1F497D">Hi, <o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span style="color:#1F497D">Can someone please advise on this issue. ?? Urgent. Selfheal is working every 10 minutes only. ??<o:p></o:p></span></b></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<div>
<p class="MsoNormal"><span style="color:black">Thanks & Regards,<o:p></o:p></span></p>
<p class="MsoNormal"><b><span style="font-size:9.0pt;font-family:"Arial","sans-serif";color:black">Bobby Jacob<o:p></o:p></span></b></p>
</div>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> Bobby Jacob
<br>
<b>Sent:</b> Tuesday, December 03, 2013 8:51 AM<br>
<b>To:</b> gluster-users@gluster.org<br>
<b>Subject:</b> FW: Self Heal Issue GlusterFS 3.3.1<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">Just and addition: on the node where the self heal is not working when I check /var/log/glusterd/glustershd.log, I see the following:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">[2013-12-03 05:49:18.348637] E [afr-self-heald.c:685:_link_inode_update_loc] 0-glustervol-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">[2013-12-03 05:49:18.350273] E [afr-self-heald.c:685:_link_inode_update_loc] 0-glustervol-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">[2013-12-03 05:49:18.354813] E [afr-self-heald.c:685:_link_inode_update_loc] 0-glustervol-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">[2013-12-03 05:49:18.355893] E [afr-self-heald.c:685:_link_inode_update_loc] 0-glustervol-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">[2013-12-03 05:49:18.356901] E [afr-self-heald.c:685:_link_inode_update_loc] 0-glustervol-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">[2013-12-03 05:49:18.357730] E [afr-self-heald.c:685:_link_inode_update_loc] 0-glustervol-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">[2013-12-03 05:49:18.359136] E [afr-self-heald.c:685:_link_inode_update_loc] 0-glustervol-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">[2013-12-03 05:49:18.360276] E [afr-self-heald.c:685:_link_inode_update_loc] 0-glustervol-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">[2013-12-03 05:49:18.361168] E [afr-self-heald.c:685:_link_inode_update_loc] 0-glustervol-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">[2013-12-03 05:49:18.362135] E [afr-self-heald.c:685:_link_inode_update_loc] 0-glustervol-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">[2013-12-03 05:49:18.363569] E [afr-self-heald.c:685:_link_inode_update_loc] 0-glustervol-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">[2013-12-03 05:49:18.364232] E [afr-self-heald.c:685:_link_inode_update_loc] 0-glustervol-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">[2013-12-03 05:49:18.364872] E [afr-self-heald.c:685:_link_inode_update_loc] 0-glustervol-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">[2013-12-03 05:49:18.365777] E [afr-self-heald.c:685:_link_inode_update_loc] 0-glustervol-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">[2013-12-03 05:49:18.367383] E [afr-self-heald.c:685:_link_inode_update_loc] 0-glustervol-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">[2013-12-03 05:49:18.368075] E [afr-self-heald.c:685:_link_inode_update_loc] 0-glustervol-replicate-0: inode link failed on the inode (00000000-0000-0000-0000-000000000000)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<div>
<p class="MsoNormal"><span style="color:black">Thanks & Regards,<o:p></o:p></span></p>
<p class="MsoNormal"><b><span style="font-size:9.0pt;font-family:"Arial","sans-serif";color:black">Bobby Jacob<o:p></o:p></span></b></p>
</div>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> gluster-users-bounces@gluster.org [mailto:gluster-users-bounces@gluster.org]
<b>On Behalf Of </b>Bobby Jacob<br>
<b>Sent:</b> Tuesday, December 03, 2013 8:48 AM<br>
<b>To:</b> gluster-users@gluster.org<br>
<b>Subject:</b> [Gluster-users] Self Heal Issue GlusterFS 3.3.1<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Hi,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I’m running glusterFS 3.3.1 on Centos 6.4. <o:p></o:p></p>
<p class="MsoListParagraph" style="text-indent:-.25in;mso-list:l0 level1 lfo2"><![if !supportLists]><span style="font-family:Wingdings"><span style="mso-list:Ignore">Ø<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]>Gluster volume status<o:p></o:p></p>
<p class="MsoListParagraph"><o:p> </o:p></p>
<p class="MsoListParagraph">Status of volume: glustervol<o:p></o:p></p>
<p class="MsoListParagraph">Gluster process Port Online Pid<o:p></o:p></p>
<p class="MsoListParagraph">------------------------------------------------------------------------------<o:p></o:p></p>
<p class="MsoListParagraph">Brick KWTOCUATGS001:/mnt/cloudbrick 24009 Y 20031<o:p></o:p></p>
<p class="MsoListParagraph">Brick KWTOCUATGS002:/mnt/cloudbrick 24009 Y 1260<o:p></o:p></p>
<p class="MsoListParagraph">NFS Server on localhost 38467 Y 43320<o:p></o:p></p>
<p class="MsoListParagraph">Self-heal Daemon on localhost N/A Y 43326<o:p></o:p></p>
<p class="MsoListParagraph">NFS Server on KWTOCUATGS002 38467 Y 5842<o:p></o:p></p>
<p class="MsoListParagraph">Self-heal Daemon on KWTOCUATGS002 N/A Y 5848<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">The self heal stops working and application write only to 1 brick and it doesn’t replicate. When I check /var/log/glusterfs/glustershd.log I see the following.:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">[2013-12-03 05:42:32.033563] W [socket.c:410:__socket_keepalive] 0-socket: failed to set keep idle on socket 8<o:p></o:p></p>
<p class="MsoNormal">[2013-12-03 05:42:32.033646] W [socket.c:1876:socket_server_event_handler] 0-socket.glusterfsd: Failed to set keep-alive: Operation not supported<o:p></o:p></p>
<p class="MsoNormal">[2013-12-03 05:42:32.790473] I [client-handshake.c:1614:select_server_supported_programs] 0-glustervol-client-1: Using Program GlusterFS 3.3.2, Num (1298437), Version (330)<o:p></o:p></p>
<p class="MsoNormal">[2013-12-03 05:42:32.790840] I [client-handshake.c:1411:client_setvolume_cbk] 0-glustervol-client-1: Connected to 172.16.95.153:24009, attached to remote volume '/mnt/cloudbrick'.<o:p></o:p></p>
<p class="MsoNormal">[2013-12-03 05:42:32.790884] I [client-handshake.c:1423:client_setvolume_cbk] 0-glustervol-client-1: Server and Client lk-version numbers are not same, reopening the fds<o:p></o:p></p>
<p class="MsoNormal">[2013-12-03 05:42:32.791003] I [afr-common.c:3685:afr_notify] 0-glustervol-replicate-0: Subvolume 'glustervol-client-1' came back up; going online.<o:p></o:p></p>
<p class="MsoNormal">[2013-12-03 05:42:32.791161] I [client-handshake.c:453:client_set_lk_version_cbk] 0-glustervol-client-1: Server lk version = 1<o:p></o:p></p>
<p class="MsoNormal">[2013-12-03 05:42:32.795103] E [afr-self-heal-data.c:1321:afr_sh_data_open_cbk] 0-glustervol-replicate-0: open of <gfid:a7e88fd1-6e32-40ab-90f6-ea452242a7c6> failed on child glustervol-client-0 (Transport endpoint is not connected)<o:p></o:p></p>
<p class="MsoNormal">[2013-12-03 05:42:32.798064] E [afr-self-heal-data.c:1321:afr_sh_data_open_cbk] 0-glustervol-replicate-0: open of <gfid:081c6657-301a-42a4-9f95-6eeba6c67413> failed on child glustervol-client-0 (Transport endpoint is not connected)<o:p></o:p></p>
<p class="MsoNormal">[2013-12-03 05:42:32.799278] E [afr-self-heal-data.c:1321:afr_sh_data_open_cbk] 0-glustervol-replicate-0: open of <gfid:565f1358-449c-45e2-8535-93b5632c0d1e> failed on child glustervol-client-0 (Transport endpoint is not connected)<o:p></o:p></p>
<p class="MsoNormal">[2013-12-03 05:42:32.800636] E [afr-self-heal-data.c:1321:afr_sh_data_open_cbk] 0-glustervol-replicate-0: open of <gfid:9c7010ac-5c11-4561-8b86-5c4d6561f34e> failed on child glustervol-client-0 (Transport endpoint is not connected)<o:p></o:p></p>
<p class="MsoNormal">[2013-12-03 05:42:32.802223] E [afr-self-heal-data.c:1321:afr_sh_data_open_cbk] 0-glustervol-replicate-0: open of <gfid:25fd406f-63e0-4037-bb01-da282cbe4d76> failed on child glustervol-client-0 (Transport endpoint is not connected)<o:p></o:p></p>
<p class="MsoNormal">[2013-12-03 05:42:32.803339] E [afr-self-heal-data.c:1321:afr_sh_data_open_cbk] 0-glustervol-replicate-0: open of <gfid:a109c429-5885-499e-8711-09fdccd396f2> failed on child glustervol-client-0 (Transport endpoint is not connected)<o:p></o:p></p>
<p class="MsoNormal">[2013-12-03 05:42:32.804308] E [afr-self-heal-data.c:1321:afr_sh_data_open_cbk] 0-glustervol-replicate-0: open of <gfid:5a8fd3bf-9215-444c-b974-5c280f5699a6> failed on child glustervol-client-0 (Transport endpoint is not connected)<o:p></o:p></p>
<p class="MsoNormal">[2013-12-03 05:42:32.804877] I [client-handshake.c:1614:select_server_supported_programs] 0-glustervol-client-0: Using Program GlusterFS 3.3.2, Num (1298437), Version (330)<o:p></o:p></p>
<p class="MsoNormal">[2013-12-03 05:42:32.807517] I [client-handshake.c:1411:client_setvolume_cbk] 0-glustervol-client-0: Connected to 172.16.107.154:24009, attached to remote volume '/mnt/cloudbrick'.<o:p></o:p></p>
<p class="MsoNormal">[2013-12-03 05:42:32.807562] I [client-handshake.c:1423:client_setvolume_cbk] 0-glustervol-client-0: Server and Client lk-version numbers are not same, reopening the fds<o:p></o:p></p>
<p class="MsoNormal">[2013-12-03 05:42:32.810357] I [client-handshake.c:453:client_set_lk_version_cbk] 0-glustervol-client-0: Server lk version = 1<o:p></o:p></p>
<p class="MsoNormal">[2013-12-03 05:42:32.827437] E [afr-self-heal-data.c:764:afr_sh_data_fxattrop_fstat_done] 0-glustervol-replicate-0: Unable to self-heal contents of '<gfid:1262d40d-46a3-4e57-b07b-0fcc972c8403>' (possible split-brain). Please delete the
file from all but the preferred subvolume.<o:p></o:p></p>
<p class="MsoNormal">[2013-12-03 05:42:39.205157] E [afr-self-heal-metadata.c:472:afr_sh_metadata_fix] 0-glustervol-replicate-0: Unable to self-heal permissions/ownership of '<gfid:c590e3fb-a376-4ac9-86a6-14a80814e06f>' (possible split-brain). Please fix the
file on all backend volumes<o:p></o:p></p>
<p class="MsoNormal">[2013-12-03 05:42:39.215793] E [afr-self-heal-metadata.c:472:afr_sh_metadata_fix] 0-glustervol-replicate-0: Unable to self-heal permissions/ownership of '<gfid:c0660768-289f-48ac-b8e5-e5b5a3a4b965>' (possible split-brain). Please fix the
file on all backend volumes<o:p></o:p></p>
<p class="MsoNormal"><b><o:p> </o:p></b></p>
<p class="MsoNormal"><b><o:p> </o:p></b></p>
<p class="MsoNormal"><b>PLEASE ADVICE.<o:p></o:p></b></p>
<p class="MsoNormal"><b><o:p> </o:p></b></p>
<p class="MsoNormal"><span style="color:black">Thanks & Regards,<o:p></o:p></span></p>
<p class="MsoNormal"><b><span style="font-size:9.0pt;font-family:"Arial","sans-serif";color:black">Bobby Jacob<o:p></o:p></span></b></p>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</body>
</html>