<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=us-ascii"><meta name=Generator content="Microsoft Word 14 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";
mso-fareast-language:EN-US;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoAcetate, li.MsoAcetate, div.MsoAcetate
{mso-style-priority:99;
mso-style-link:"Balloon Text Char";
margin:0cm;
margin-bottom:.0001pt;
font-size:8.0pt;
font-family:"Tahoma","sans-serif";
mso-fareast-language:EN-US;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri","sans-serif";
color:windowtext;}
span.BalloonTextChar
{mso-style-name:"Balloon Text Char";
mso-style-priority:99;
mso-style-link:"Balloon Text";
font-family:"Tahoma","sans-serif";}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 90.0pt 72.0pt 90.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=FR-CA link=blue vlink=purple><div class=WordSection1><p class=MsoNormal>Hello,<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><span lang=EN-CA>We are experiencing strange behavior when writing file on the Gluster mount point. On some occasion, when writing to the Gluster Mount we have an Open Stream error. We’ve looked the gluster logs and found the following faulty entries :<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-CA><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-CA>[From /var/log/glusterfs/mnt-gv0.log]<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-CA><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-CA>[2013-11-18 18:18:43.052446] W [afr-common.c:1411:afr_conflicting_iattrs] 0-gv0-replicate-0: /aa/aa/aa/aa: filetype differs on subvolumes (0, 1)<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-CA>[2013-11-18 18:18:43.052468] E [afr-self-heal-common.c:1409:afr_sh_common_lookup_cbk] 0-gv0-replicate-0: Conflicting entries for /aa/aa/aa/aa<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-CA>[2013-11-18 18:18:43.052757] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-gv0-replicate-0: background meta-data data entry missing-entry gfid self-heal<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-CA> failed on /aa/aa/aa/aa/aa<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-CA>[2013-11-18 18:18:43.052780] W [fuse-bridge.c:292:fuse_entry_cbk] 0-glusterfs-fuse: 439382194: LOOKUP() /aa/aa/aa/aa/aa => -1 (Input/output error)<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-CA><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-CA>We’ve looked at the log file etc-glusterfs-glusterd.vol.log but we found nothing related to this problem. Then, we’ve looked at the log From /var/log/glusterfs/bricks/mnt-data.log and we found 70 gig of logs of the same type :<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-CA><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-CA>[2013-11-19 17:13:32.269757] W [posix-handle.c:538:posix_handle_soft] 0-gv0-posix: symlink ../../ab/fe/abfeb61c-501d-4417-b8fb-0accdd57146f/cf -> /mnt/data/.glusterfs/ab/fe/abfeb61c-501d-4417-b8fb-0accdd57146f/cf failed (No such file or directory)<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-CA>[2013-11-19 17:13:32.269978] W [posix-handle.c:538:posix_handle_soft] 0-gv0-posix: symlink ../../c7/8b/c78be78f-cc95-47b2-a27f-4217f1759b67/d2 -> /mnt/data/.glusterfs/c7/8b/c78be78f-cc95-47b2-a27f-4217f1759b67/d2 failed (No such file or directory)<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-CA>[2013-11-19 17:13:32.270190] W [posix-handle.c:538:posix_handle_soft] 0-gv0-posix: symlink ../../5a/8f/5a8fa43c-4ccc-4d88-9122-a96bc8ffaebc/f2 -> /mnt/data/.glusterfs/5a/8f/5a8fa43c-4ccc-4d88-9122-a96bc8ffaebc/f2 failed (No such file or directory)<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-CA><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-CA>And it does not stop logging. It seems that the self heal is not working properly when there are broking symlinks in the gluster. It is worth saying also that this log is only produce on a single node but the write fail on several node though. Also, we try to clean the symlinks manually but it always come back.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-CA><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-CA>Is it possible to recover from broken symlinks?<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-CA><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-CA>Configuration :<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-CA>Gluster Version : 3.3.2<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-CA>Cluster setup : 4 X 2<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-CA>OS : Ubuntu<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-CA>On Fuse<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-CA><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-CA>Thanks,<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-CA><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-CA>Alexandre<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-CA style='mso-fareast-language:FR-CA'><o:p> </o:p></span></p></div></body></html>