<html>
  <head>
    <meta content="text/html; charset=UTF-8" http-equiv="Content-Type">
  </head>
  <body bgcolor="#FFFFFF" text="#000000">
    <div class="moz-cite-prefix">On 02/28/2014 01:09 PM, Zhang Huan
      wrote:<br>
    </div>
    <blockquote
cite="mid:CAM8aEsGwicV0xVsEjST67aa=cnVp48FKRophA7om9JEcbTFbHg@mail.gmail.com"
      type="cite">
      <div dir="ltr">
        <div class="gmail_default"
          style="font-family:arial,helvetica,sans-serif"><span
            style="font-family:arial">On Fri, Feb 28, 2014 at 12:07 PM,
            Ravishankar N </span><span dir="ltr"
            style="font-family:arial">&lt;<a moz-do-not-send="true"
              href="mailto:ravishankar@redhat.com" target="_blank">ravishankar@redhat.com</a>&gt;</span><span
            style="font-family:arial"> wrote:</span><br>
        </div>
        <div class="gmail_extra">
          <div class="gmail_quote">
            <blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
              <div bgcolor="#FFFFFF" text="#000000">
                <div>
                  <div>On 02/28/2014 07:28 AM, Zhang Huan wrote:<br>
                  </div>
                  <blockquote type="cite">
                    <div dir="ltr">
                      <div
                        style="font-family:arial,helvetica,sans-serif">Hello <span
style="font-family:arial,sans-serif;font-size:13px">Ravi,</span></div>
                      <div
                        style="font-family:arial,helvetica,sans-serif">
                        <span
                          style="font-family:arial,sans-serif;font-size:13px"><br>
                        </span></div>
                      <div
                        style="font-family:arial,helvetica,sans-serif"><span
style="font-family:arial,sans-serif;font-size:13px">Thanks for your
                          reply.</span></div>
                      <div
                        style="font-family:arial,helvetica,sans-serif"><span
style="font-family:arial,sans-serif;font-size:13px"><br>
                        </span></div>
                      <div
                        style="font-family:arial,helvetica,sans-serif">
                        <span
                          style="font-family:arial,sans-serif;font-size:13px">Sorry


                          that I have a typo in my mail. It should by
                          "underlying corruption" instead of "underlying
                          correction".</span></div>
                      <div
                        style="font-family:arial,helvetica,sans-serif">
                        <br>
                      </div>
                      <div><font face="arial, sans-serif">I guess the
                          logic of eliminating zero byte files from all
                          innocent nodes is working for preventing
                          underlying corruption to propagate to other
                          brick. Asked in another way, if the underlying
                          brick finds some file is corrupted, anything
                          it could do to tell glusterfs to fix it?</font></div>
                      <div
                        style="font-family:arial,helvetica,sans-serif"><span
style="font-family:arial,sans-serif;font-size:13px"><br>
                        </span></div>
                    </div>
                  </blockquote>
                </div>
                Hi Zhang,<br>
                If all nodes are innocent (from AFR's point of view)
                ,then AFR cannot  use the changelog attributes to
                determine which is source. In this case, the safest bet
                is to mark all zero byte files as sink, so that we don't
                end up healing in the wrong direction.  Like I said
                earlier, AFR can only use the changelog attributes
                (xattrs) to determine the source/sinks. It cannot detect
                underlying on disk file system corruptions outside the
                scope of the xattrs. <br>
                <br>
                If you are sure that a particular brick is the right
                source despite the xattrs saying otherwise, you can
                manually change the attributes of the file on all bricks
                so that AFR now sees that brick as the source and heals
                in the expected direction.<br>
                <br>
                -Ravi</div>
            </blockquote>
            <div><br>
            </div>
            <div class="gmail_default"
              style="font-family:arial,helvetica,sans-serif">Hello Ravi,</div>
            <div class="gmail_default"
              style="font-family:arial,helvetica,sans-serif"><br>
            </div>
            <div class="gmail_default"
              style="font-family:arial,helvetica,sans-serif"> IMO,
              changing the attributes might be dangerous, since
              concurrent access with glusterfs is introduced. Not sure
              if glusterfs has already provided some mechanism for this.</div>
            <div class="gmail_default"
              style="font-family:arial,helvetica,sans-serif"> <br>
            </div>
          </div>
        </div>
      </div>
    </blockquote>
    You are right Zhang. My assumption was that the file wouldn't be
    modified from the mount point while you are modifying the xattrs at
    the bricks.<br>
    <blockquote
cite="mid:CAM8aEsGwicV0xVsEjST67aa=cnVp48FKRophA7om9JEcbTFbHg@mail.gmail.com"
      type="cite">
      <div dir="ltr">
        <div class="gmail_extra">
          <div class="gmail_quote">
            <div class="gmail_default"
              style="font-family:arial,helvetica,sans-serif">My
              suggestion is to eliminate the zero-byte file from heal
              source even if is marked as a source. If the underlying
              filesystem finds some corruption (by scrubbing daemon
              after checking data checksum), it could truncate it to 0
              and let glusterfs to do the healing job.</div>
          </div>
        </div>
      </div>
    </blockquote>
    If there is underlying FS corruption and we need to make gluster
    aware of it, then something like bit rot detection would be the way
    to go. You can find more information about some work in progress on
    the gluster website/ mailing list archives:<br>
<a class="moz-txt-link-freetext" href="http://www.gluster.org/community/documentation/index.php/Arch/BitRot_Detection">http://www.gluster.org/community/documentation/index.php/Arch/BitRot_Detection</a><br>
<a class="moz-txt-link-freetext" href="http://lists.nongnu.org/archive/html/gluster-devel/2014-01/msg00209.html">http://lists.nongnu.org/archive/html/gluster-devel/2014-01/msg00209.html</a><br>
<a class="moz-txt-link-freetext" href="https://lists.gnu.org/archive/html/gluster-devel/2014-01/msg00006.html">https://lists.gnu.org/archive/html/gluster-devel/2014-01/msg00006.html</a><br>
    <br>
    -Ravi<br>
    <br>
    <blockquote
cite="mid:CAM8aEsGwicV0xVsEjST67aa=cnVp48FKRophA7om9JEcbTFbHg@mail.gmail.com"
      type="cite">
      <div dir="ltr">
        <div class="gmail_extra">
          <div class="gmail_quote">
            <div class="gmail_default"
              style="font-family:arial,helvetica,sans-serif"> Here is
              several cases of analysis in my mind.</div>
            <div class="gmail_default"
              style="font-family:arial,helvetica,sans-serif">1. If this
              corrupted file is marked as the only source, then there is
              no correct replica in the filesystem (actually all are
              fools), just pick any one as the source to heal is OK;</div>
            <div class="gmail_default"
              style="font-family:arial,helvetica,sans-serif">2. If the
              corrupted file is one of the potential sources, eliminate
              this one should keep healing in the right direction
              without further corrupting other correct replicas.</div>
            <div class="gmail_default"
              style="font-family:arial,helvetica,sans-serif">3. If the
              corrupted file is not marked as a source, some other
              replica will be chosen as a source and this file will be
              overwritten with correct data.</div>
            <div class="gmail_default"
              style="font-family:arial,helvetica,sans-serif">4. If there
              is no one is marked as clean by attribute, it is quite
              unlikely this file is chosen as a source as its size is 0.
              Even it is chosen as a source, there is no further
              corruption of file content after heal.</div>
            <div class="gmail_default"
              style="font-family:arial,helvetica,sans-serif"><br>
            </div>
            <div class="gmail_default"
              style="font-family:arial,helvetica,sans-serif">
              <div style="font-family:arial">Zhang Huan</div>
              <div style="font-family:arial"><br>
              </div>
            </div>
          </div>
        </div>
      </div>
    </blockquote>
    <br>
  </body>
</html>