Pranith, the md5sum mismatch was noticed Tuesday (9/6) morning via an automated process that verifies file checksum consistency between what is stored on Gluster and the source.  However, the file was written to Gluster on 8/31.  We&#39;re not sure if something happened between 8/31 and 9/6 that would make the replicas inconsistent.  It was only after looking at the file via backend storage on Tuesday we noticed that there was an inconsistency between the two replica copies of the file.<br>
<br>What is odd is that the fattr and stat data of the two replica files are identical.  However, when we try to unzip the file, the file that gives us the incorrect md5sum also gives us an error when unzipping which makes the replica file unusable.  I&#39;m just curious to know why Gluster can&#39;t see the difference between the two files.  Gzip isn&#39;t capable of reading the file with the erroneous checksum but according to the fattr and stat data, they&#39;re the same file when compared to the good copy.<br>
<br>All of our processes read/write/modify file i/o through the Gluster mount point whether its via glusterfs or NFS - never through the backend storage with the exception of recent non-intrusive troubleshooting (getfattr and stat) of backend storage.<br>
<br>Thank you<br><br><div class="gmail_quote">On Wed, Sep 7, 2011 at 10:45 PM, Pranith Kumar K <span dir="ltr">&lt;<a href="mailto:pranithk@gluster.com">pranithk@gluster.com</a>&gt;</span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<u></u>

  
    
  
  <div bgcolor="#ffffff" text="#000000">
    hi Anthony,<br>
          Thanks for the outputs. Nothing suspicious. When did you
    notice that the md5sums are not matching? As soon as it is created
    or something happened before the file ended up in this situation.<br><font color="#888888">
    <br>
    Pranith.</font><div><div></div><div class="h5"><br>
    On 09/08/2011 02:10 AM, Anthony Delviscio wrote:
    <blockquote type="cite">
      
      Pranith, thank you for clarifying.<br>
      <br>
      The folder was moved earlier today but the issue of md5sums of the
      file being different still persists.<br>
      The files are compressed and when decompressing the files, the
      file with the correct (or expected md5sum) decompresses without
      error.  However, the file with the incorrect mdsum doesn&#39;t
      decompress properly and cites crc and length errors.<br>
      <br>
      The pastie output of stat and fattr of the directory housing the
      file can be found here:<br>
      <a href="http://pastie.org/2499242" target="_blank">http://pastie.org/2499242</a><br>
      <br>
      Thank you<br>
      Anthony <br>
      <br>
      <div class="gmail_quote">
        On Wed, Sep 7, 2011 at 1:24 PM, Pranith Kumar K <span dir="ltr">&lt;<a href="mailto:pranithk@gluster.com" target="_blank">pranithk@gluster.com</a>&gt;</span>
        wrote:<br>
        <blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204, 204, 204);padding-left:1ex">
          <div bgcolor="#ffffff" text="#000000"> hi Anthony,<br>
                Parent directory is directory that contains the file.<br>
            <font color="#888888"> <br>
              Pranith</font>
            <div>
              <div><br>
                <br>
                On 09/07/2011 08:18 PM, Anthony Delviscio wrote:
                <blockquote type="cite">
                  <p class="MsoNormal"><span style="font-size:11pt;color:rgb(31, 73, 125)">Pranith, by parent
                      directory, do you mean the directory that contains
                      the file or the top level directory of the brick?</span></p>
                  <p class="MsoNormal"><span style="font-size:11pt;color:rgb(31, 73, 125)"> </span></p>
                  <p class="MsoNormal"><span style="font-size:11pt;color:rgb(31, 73, 125)">My gluster volume info:</span></p>
                  <p class="MsoNormal"><span style="font-size:11pt;color:rgb(31, 73, 125)"><a href="http://pastie.org/2493045" target="_blank">http://pastie.org/2493045</a></span></p>
                  <p class="MsoNormal"><span style="font-size:11pt;color:rgb(31, 73, 125)">The hostnames used in
                      the gluster volume info are DNS hostnames that
                      resolve to 10GB interfaces on the Gluster nodes.<span> 
                      </span></span></p>
                  <p class="MsoNormal"><span style="font-size:11pt;color:rgb(31, 73, 125)">The hostname used in the
                      mount options is a RR DNS hostname that resolves
                      to all eight Gluster nodes.</span></p>
                  <p class="MsoNormal"><span style="font-size:11pt;color:rgb(31, 73, 125)"> </span></p>
                  <p class="MsoNormal"><span style="font-size:11pt;color:rgb(31, 73, 125)">Stat/md5sum/getfattr
                      data of the identical files with different
                      md5sums.</span></p>
                  <p class="MsoNormal"><span style="font-size:11pt;color:rgb(31, 73, 125)"><a href="http://pastie.org/2497461" target="_blank">http://pastie.org/2497461</a></span></p>
                  <p class="MsoNormal"><span style="font-size:11pt;color:rgb(31, 73, 125)"> </span></p>
                  <p class="MsoNormal"><span style="font-size:11pt;color:rgb(31, 73, 125)">Thank you</span></p>
                  <br>
                  <br>
                  <div class="gmail_quote">On Wed, Sep 7, 2011 at 4:43
                    AM, Pranith Kumar K <span dir="ltr">&lt;<a href="mailto:pranithk@gluster.com" target="_blank">pranithk@gluster.com</a>&gt;</span>
                    wrote:<br>
                    <blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204, 204, 204);padding-left:1ex">
                      <div bgcolor="#ffffff" text="#000000"> hi Anthony,<br>
                              Could you send the output of the getfattr
                        -d -m . -e hex &lt;filepath&gt; on both the
                        bricks and also the stat output on the both the
                        backends. Give the outputs for its parent
                        directory also.<br>
                        <br>
                        Pranith.
                        <div>
                          <div><br>
                            <br>
                            On 09/07/2011 04:22 AM, Anthony Delviscio
                            wrote: </div>
                        </div>
                        <blockquote type="cite">
                          <div>
                            <div>
                              <div>
                                <div>
                                  <p class="MsoNormal">I was wondering
                                    if anyone would be able to shed some
                                    light on how a file could end up
                                    with inconsistent md5sums on Gluster
                                    backend storage.</p>
                                  <p class="MsoNormal"> </p>
                                  <p class="MsoNormal">Our configuration
                                    is running on Gluster v3.1.5 in a
                                    distribute-replicate setup
                                    consisting of 8 bricks.</p>
                                  <p class="MsoNormal">Our OS is Red Hat
                                    5.6 x86_64.<span>  </span>Backend
                                    storage is an ext3 RAID 5.</p>
                                  <p class="MsoNormal"> </p>
                                  <p class="MsoNormal">The 8 bricks are
                                    in RR DNS and are mounted for
                                    reading/writing via NFS automounts.<span> 
                                    </span></p>
                                  <p class="MsoNormal"> </p>
                                  <p class="MsoNormal">When comparing
                                    md5sums of the file from two
                                    different NFS clients, they were
                                    different.<span>  </span></p>
                                  <p class="MsoNormal"> </p>
                                  <p class="MsoNormal">The extended
                                    attributes of the files on backend
                                    storage are identical.<span>  </span>The
                                    file size and permissions are
                                    identical.<span>  </span>The stat
                                    data (excluding inode on backend
                                    storage file system) is identical.</p>
                                  <p class="MsoNormal">However, running
                                    md5sum on the two files, results in
                                    two different md5sums.</p>
                                  <p class="MsoNormal"> </p>
                                  <p class="MsoNormal">Copying both
                                    files to another location/server and
                                    running the md5sum also results in
                                    no change – they’re still different.</p>
                                  <p class="MsoNormal"> </p>
                                  <p class="MsoNormal">Gluster logs do
                                    not show anything related to the
                                    filename in question.<span>  </span>Triggering
                                    a self-healing operation didn’t seem
                                    to do anything and it may have to do
                                    with the fact that the extended
                                    attributes are identical.</p>
                                  <p class="MsoNormal"> </p>
                                  <p class="MsoNormal">If more
                                    information is required, let me know
                                    and I will try to accommodate.<span> 
                                    </span></p>
                                  <p class="MsoNormal">Thank you</p>
                                </div>
                              </div>
                            </div>
                          </div>
                          <pre><fieldset></fieldset>
_______________________________________________
Gluster-users mailing list
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a>
<a href="http://gluster.org/cgi-bin/mailman/listinfo/gluster-users" target="_blank">http://gluster.org/cgi-bin/mailman/listinfo/gluster-users</a>
</pre>
                        </blockquote>
                        <br>
                      </div>
                    </blockquote>
                  </div>
                  <br>
                </blockquote>
                <br>
              </div>
            </div>
          </div>
        </blockquote>
      </div>
      <br>
    </blockquote>
    <br>
  </div></div></div>

</blockquote></div><br>