<html>
<head>
<meta content="text/html; charset=UTF-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<div class="moz-cite-prefix">On 02/28/2014 01:09 PM, Zhang Huan
wrote:<br>
</div>
<blockquote
cite="mid:CAM8aEsGwicV0xVsEjST67aa=cnVp48FKRophA7om9JEcbTFbHg@mail.gmail.com"
type="cite">
<div dir="ltr">
<div class="gmail_default"
style="font-family:arial,helvetica,sans-serif"><span
style="font-family:arial">On Fri, Feb 28, 2014 at 12:07 PM,
Ravishankar N </span><span dir="ltr"
style="font-family:arial"><<a moz-do-not-send="true"
href="mailto:ravishankar@redhat.com" target="_blank">ravishankar@redhat.com</a>></span><span
style="font-family:arial"> wrote:</span><br>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<div>
<div>On 02/28/2014 07:28 AM, Zhang Huan wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div
style="font-family:arial,helvetica,sans-serif">Hello <span
style="font-family:arial,sans-serif;font-size:13px">Ravi,</span></div>
<div
style="font-family:arial,helvetica,sans-serif">
<span
style="font-family:arial,sans-serif;font-size:13px"><br>
</span></div>
<div
style="font-family:arial,helvetica,sans-serif"><span
style="font-family:arial,sans-serif;font-size:13px">Thanks for your
reply.</span></div>
<div
style="font-family:arial,helvetica,sans-serif"><span
style="font-family:arial,sans-serif;font-size:13px"><br>
</span></div>
<div
style="font-family:arial,helvetica,sans-serif">
<span
style="font-family:arial,sans-serif;font-size:13px">Sorry
that I have a typo in my mail. It should by
"underlying corruption" instead of "underlying
correction".</span></div>
<div
style="font-family:arial,helvetica,sans-serif">
<br>
</div>
<div><font face="arial, sans-serif">I guess the
logic of eliminating zero byte files from all
innocent nodes is working for preventing
underlying corruption to propagate to other
brick. Asked in another way, if the underlying
brick finds some file is corrupted, anything
it could do to tell glusterfs to fix it?</font></div>
<div
style="font-family:arial,helvetica,sans-serif"><span
style="font-family:arial,sans-serif;font-size:13px"><br>
</span></div>
</div>
</blockquote>
</div>
Hi Zhang,<br>
If all nodes are innocent (from AFR's point of view)
,then AFR cannot use the changelog attributes to
determine which is source. In this case, the safest bet
is to mark all zero byte files as sink, so that we don't
end up healing in the wrong direction. Like I said
earlier, AFR can only use the changelog attributes
(xattrs) to determine the source/sinks. It cannot detect
underlying on disk file system corruptions outside the
scope of the xattrs. <br>
<br>
If you are sure that a particular brick is the right
source despite the xattrs saying otherwise, you can
manually change the attributes of the file on all bricks
so that AFR now sees that brick as the source and heals
in the expected direction.<br>
<br>
-Ravi</div>
</blockquote>
<div><br>
</div>
<div class="gmail_default"
style="font-family:arial,helvetica,sans-serif">Hello Ravi,</div>
<div class="gmail_default"
style="font-family:arial,helvetica,sans-serif"><br>
</div>
<div class="gmail_default"
style="font-family:arial,helvetica,sans-serif"> IMO,
changing the attributes might be dangerous, since
concurrent access with glusterfs is introduced. Not sure
if glusterfs has already provided some mechanism for this.</div>
<div class="gmail_default"
style="font-family:arial,helvetica,sans-serif"> <br>
</div>
</div>
</div>
</div>
</blockquote>
You are right Zhang. My assumption was that the file wouldn't be
modified from the mount point while you are modifying the xattrs at
the bricks.<br>
<blockquote
cite="mid:CAM8aEsGwicV0xVsEjST67aa=cnVp48FKRophA7om9JEcbTFbHg@mail.gmail.com"
type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div class="gmail_default"
style="font-family:arial,helvetica,sans-serif">My
suggestion is to eliminate the zero-byte file from heal
source even if is marked as a source. If the underlying
filesystem finds some corruption (by scrubbing daemon
after checking data checksum), it could truncate it to 0
and let glusterfs to do the healing job.</div>
</div>
</div>
</div>
</blockquote>
If there is underlying FS corruption and we need to make gluster
aware of it, then something like bit rot detection would be the way
to go. You can find more information about some work in progress on
the gluster website/ mailing list archives:<br>
<a class="moz-txt-link-freetext" href="http://www.gluster.org/community/documentation/index.php/Arch/BitRot_Detection">http://www.gluster.org/community/documentation/index.php/Arch/BitRot_Detection</a><br>
<a class="moz-txt-link-freetext" href="http://lists.nongnu.org/archive/html/gluster-devel/2014-01/msg00209.html">http://lists.nongnu.org/archive/html/gluster-devel/2014-01/msg00209.html</a><br>
<a class="moz-txt-link-freetext" href="https://lists.gnu.org/archive/html/gluster-devel/2014-01/msg00006.html">https://lists.gnu.org/archive/html/gluster-devel/2014-01/msg00006.html</a><br>
<br>
-Ravi<br>
<br>
<blockquote
cite="mid:CAM8aEsGwicV0xVsEjST67aa=cnVp48FKRophA7om9JEcbTFbHg@mail.gmail.com"
type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div class="gmail_default"
style="font-family:arial,helvetica,sans-serif"> Here is
several cases of analysis in my mind.</div>
<div class="gmail_default"
style="font-family:arial,helvetica,sans-serif">1. If this
corrupted file is marked as the only source, then there is
no correct replica in the filesystem (actually all are
fools), just pick any one as the source to heal is OK;</div>
<div class="gmail_default"
style="font-family:arial,helvetica,sans-serif">2. If the
corrupted file is one of the potential sources, eliminate
this one should keep healing in the right direction
without further corrupting other correct replicas.</div>
<div class="gmail_default"
style="font-family:arial,helvetica,sans-serif">3. If the
corrupted file is not marked as a source, some other
replica will be chosen as a source and this file will be
overwritten with correct data.</div>
<div class="gmail_default"
style="font-family:arial,helvetica,sans-serif">4. If there
is no one is marked as clean by attribute, it is quite
unlikely this file is chosen as a source as its size is 0.
Even it is chosen as a source, there is no further
corruption of file content after heal.</div>
<div class="gmail_default"
style="font-family:arial,helvetica,sans-serif"><br>
</div>
<div class="gmail_default"
style="font-family:arial,helvetica,sans-serif">
<div style="font-family:arial">Zhang Huan</div>
<div style="font-family:arial"><br>
</div>
</div>
</div>
</div>
</div>
</blockquote>
<br>
</body>
</html>