<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<div class="moz-cite-prefix">On 11/19/2013 10:49 PM, Alexandre
Fournier wrote:<br>
</div>
<blockquote
cite="mid:CB5152E8C4C5F54983F4B38A0DB3F52505076EA3@MDC-MAIL-CMS01.ubisoft.org"
type="cite">
<meta http-equiv="Content-Type" content="text/html;
charset=ISO-8859-1">
<meta name="Generator" content="Microsoft Word 14 (filtered
medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";
mso-fareast-language:EN-US;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoAcetate, li.MsoAcetate, div.MsoAcetate
{mso-style-priority:99;
mso-style-link:"Balloon Text Char";
margin:0cm;
margin-bottom:.0001pt;
font-size:8.0pt;
font-family:"Tahoma","sans-serif";
mso-fareast-language:EN-US;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri","sans-serif";
color:windowtext;}
span.BalloonTextChar
{mso-style-name:"Balloon Text Char";
mso-style-priority:99;
mso-style-link:"Balloon Text";
font-family:"Tahoma","sans-serif";}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 90.0pt 72.0pt 90.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
<div class="WordSection1">
<p class="MsoNormal">Hello,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span lang="EN-CA">We are experiencing
strange behavior when writing file on the Gluster mount
point. On some occasion, when writing to the Gluster Mount
we have an Open Stream error. We’ve looked the gluster
logs and found the following faulty entries :<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">[From
/var/log/glusterfs/mnt-gv0.log]<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">[2013-11-18
18:18:43.052446] W
[afr-common.c:1411:afr_conflicting_iattrs]
0-gv0-replicate-0: /aa/aa/aa/aa: filetype differs on
subvolumes (0, 1)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">[2013-11-18
18:18:43.052468] E
[afr-self-heal-common.c:1409:afr_sh_common_lookup_cbk]
0-gv0-replicate-0: Conflicting entries for /aa/aa/aa/aa<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">[2013-11-18
18:18:43.052757] E
[afr-self-heal-common.c:2160:afr_self_heal_completion_cbk]
0-gv0-replicate-0: background meta-data data entry
missing-entry gfid self-heal<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA"> failed on
/aa/aa/aa/aa/aa<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">[2013-11-18
18:18:43.052780] W [fuse-bridge.c:292:fuse_entry_cbk]
0-glusterfs-fuse: 439382194: LOOKUP() /aa/aa/aa/aa/aa =>
-1 (Input/output error)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">We’ve looked at the log
file etc-glusterfs-glusterd.vol.log but we found nothing
related to this problem. Then, we’ve looked at the log From
/var/log/glusterfs/bricks/mnt-data.log and we found 70 gig
of logs of the same type :<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">[2013-11-19
17:13:32.269757] W [posix-handle.c:538:posix_handle_soft]
0-gv0-posix: symlink
../../ab/fe/abfeb61c-501d-4417-b8fb-0accdd57146f/cf ->
/mnt/data/.glusterfs/ab/fe/abfeb61c-501d-4417-b8fb-0accdd57146f/cf
failed (No such file or directory)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">[2013-11-19
17:13:32.269978] W [posix-handle.c:538:posix_handle_soft]
0-gv0-posix: symlink
../../c7/8b/c78be78f-cc95-47b2-a27f-4217f1759b67/d2 ->
/mnt/data/.glusterfs/c7/8b/c78be78f-cc95-47b2-a27f-4217f1759b67/d2
failed (No such file or directory)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">[2013-11-19
17:13:32.270190] W [posix-handle.c:538:posix_handle_soft]
0-gv0-posix: symlink
../../5a/8f/5a8fa43c-4ccc-4d88-9122-a96bc8ffaebc/f2 ->
/mnt/data/.glusterfs/5a/8f/5a8fa43c-4ccc-4d88-9122-a96bc8ffaebc/f2
failed (No such file or directory)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA"><o:p> </o:p></span></p>
</div>
</blockquote>
<br>
This looks like a bug, unless there is something wrong with the
set-up. I have copied gluster-devel in this thread as I think they
might help.<br>
<br>
Just curious, is all your gluster nodes have equal time (i.e. ntp
synced).<br>
<br>
<blockquote
cite="mid:CB5152E8C4C5F54983F4B38A0DB3F52505076EA3@MDC-MAIL-CMS01.ubisoft.org"
type="cite">
<div class="WordSection1">
<p class="MsoNormal"><span lang="EN-CA">And it does not stop
logging. It seems that the self heal is not working
properly when there are broking symlinks in the gluster. It
is worth saying also that this log is only produce on a
single node but the write fail on several node though.
Also, we try to clean the symlinks manually but it always
come back.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">Is it possible to
recover from broken symlinks?<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">Configuration :<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">Gluster Version : 3.3.2<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">Cluster setup : 4 X 2<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">OS : Ubuntu<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">On Fuse<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">Thanks,<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">Alexandre<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:FR-CA"
lang="EN-CA"><o:p> </o:p></span></p>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
Gluster-users mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a>
<a class="moz-txt-link-freetext" href="http://supercolony.gluster.org/mailman/listinfo/gluster-users">http://supercolony.gluster.org/mailman/listinfo/gluster-users</a></pre>
</blockquote>
<br>
</body>
</html>