Pranith, the md5sum mismatch was noticed Tuesday (9/6) morning via an automated process that verifies file checksum consistency between what is stored on Gluster and the source. However, the file was written to Gluster on 8/31. We're not sure if something happened between 8/31 and 9/6 that would make the replicas inconsistent. It was only after looking at the file via backend storage on Tuesday we noticed that there was an inconsistency between the two replica copies of the file.<br>
<br>What is odd is that the fattr and stat data of the two replica files are identical. However, when we try to unzip the file, the file that gives us the incorrect md5sum also gives us an error when unzipping which makes the replica file unusable. I'm just curious to know why Gluster can't see the difference between the two files. Gzip isn't capable of reading the file with the erroneous checksum but according to the fattr and stat data, they're the same file when compared to the good copy.<br>
<br>All of our processes read/write/modify file i/o through the Gluster mount point whether its via glusterfs or NFS - never through the backend storage with the exception of recent non-intrusive troubleshooting (getfattr and stat) of backend storage.<br>
<br>Thank you<br><br><div class="gmail_quote">On Wed, Sep 7, 2011 at 10:45 PM, Pranith Kumar K <span dir="ltr"><<a href="mailto:pranithk@gluster.com">pranithk@gluster.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<u></u>
<div bgcolor="#ffffff" text="#000000">
hi Anthony,<br>
Thanks for the outputs. Nothing suspicious. When did you
notice that the md5sums are not matching? As soon as it is created
or something happened before the file ended up in this situation.<br><font color="#888888">
<br>
Pranith.</font><div><div></div><div class="h5"><br>
On 09/08/2011 02:10 AM, Anthony Delviscio wrote:
<blockquote type="cite">
Pranith, thank you for clarifying.<br>
<br>
The folder was moved earlier today but the issue of md5sums of the
file being different still persists.<br>
The files are compressed and when decompressing the files, the
file with the correct (or expected md5sum) decompresses without
error. However, the file with the incorrect mdsum doesn't
decompress properly and cites crc and length errors.<br>
<br>
The pastie output of stat and fattr of the directory housing the
file can be found here:<br>
<a href="http://pastie.org/2499242" target="_blank">http://pastie.org/2499242</a><br>
<br>
Thank you<br>
Anthony <br>
<br>
<div class="gmail_quote">
On Wed, Sep 7, 2011 at 1:24 PM, Pranith Kumar K <span dir="ltr"><<a href="mailto:pranithk@gluster.com" target="_blank">pranithk@gluster.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204, 204, 204);padding-left:1ex">
<div bgcolor="#ffffff" text="#000000"> hi Anthony,<br>
Parent directory is directory that contains the file.<br>
<font color="#888888"> <br>
Pranith</font>
<div>
<div><br>
<br>
On 09/07/2011 08:18 PM, Anthony Delviscio wrote:
<blockquote type="cite">
<p class="MsoNormal"><span style="font-size:11pt;color:rgb(31, 73, 125)">Pranith, by parent
directory, do you mean the directory that contains
the file or the top level directory of the brick?</span></p>
<p class="MsoNormal"><span style="font-size:11pt;color:rgb(31, 73, 125)"> </span></p>
<p class="MsoNormal"><span style="font-size:11pt;color:rgb(31, 73, 125)">My gluster volume info:</span></p>
<p class="MsoNormal"><span style="font-size:11pt;color:rgb(31, 73, 125)"><a href="http://pastie.org/2493045" target="_blank">http://pastie.org/2493045</a></span></p>
<p class="MsoNormal"><span style="font-size:11pt;color:rgb(31, 73, 125)">The hostnames used in
the gluster volume info are DNS hostnames that
resolve to 10GB interfaces on the Gluster nodes.<span>
</span></span></p>
<p class="MsoNormal"><span style="font-size:11pt;color:rgb(31, 73, 125)">The hostname used in the
mount options is a RR DNS hostname that resolves
to all eight Gluster nodes.</span></p>
<p class="MsoNormal"><span style="font-size:11pt;color:rgb(31, 73, 125)"> </span></p>
<p class="MsoNormal"><span style="font-size:11pt;color:rgb(31, 73, 125)">Stat/md5sum/getfattr
data of the identical files with different
md5sums.</span></p>
<p class="MsoNormal"><span style="font-size:11pt;color:rgb(31, 73, 125)"><a href="http://pastie.org/2497461" target="_blank">http://pastie.org/2497461</a></span></p>
<p class="MsoNormal"><span style="font-size:11pt;color:rgb(31, 73, 125)"> </span></p>
<p class="MsoNormal"><span style="font-size:11pt;color:rgb(31, 73, 125)">Thank you</span></p>
<br>
<br>
<div class="gmail_quote">On Wed, Sep 7, 2011 at 4:43
AM, Pranith Kumar K <span dir="ltr"><<a href="mailto:pranithk@gluster.com" target="_blank">pranithk@gluster.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204, 204, 204);padding-left:1ex">
<div bgcolor="#ffffff" text="#000000"> hi Anthony,<br>
Could you send the output of the getfattr
-d -m . -e hex <filepath> on both the
bricks and also the stat output on the both the
backends. Give the outputs for its parent
directory also.<br>
<br>
Pranith.
<div>
<div><br>
<br>
On 09/07/2011 04:22 AM, Anthony Delviscio
wrote: </div>
</div>
<blockquote type="cite">
<div>
<div>
<div>
<div>
<p class="MsoNormal">I was wondering
if anyone would be able to shed some
light on how a file could end up
with inconsistent md5sums on Gluster
backend storage.</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">Our configuration
is running on Gluster v3.1.5 in a
distribute-replicate setup
consisting of 8 bricks.</p>
<p class="MsoNormal">Our OS is Red Hat
5.6 x86_64.<span> </span>Backend
storage is an ext3 RAID 5.</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">The 8 bricks are
in RR DNS and are mounted for
reading/writing via NFS automounts.<span>
</span></p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">When comparing
md5sums of the file from two
different NFS clients, they were
different.<span> </span></p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">The extended
attributes of the files on backend
storage are identical.<span> </span>The
file size and permissions are
identical.<span> </span>The stat
data (excluding inode on backend
storage file system) is identical.</p>
<p class="MsoNormal">However, running
md5sum on the two files, results in
two different md5sums.</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">Copying both
files to another location/server and
running the md5sum also results in
no change – they’re still different.</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">Gluster logs do
not show anything related to the
filename in question.<span> </span>Triggering
a self-healing operation didn’t seem
to do anything and it may have to do
with the fact that the extended
attributes are identical.</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">If more
information is required, let me know
and I will try to accommodate.<span>
</span></p>
<p class="MsoNormal">Thank you</p>
</div>
</div>
</div>
</div>
<pre><fieldset></fieldset>
_______________________________________________
Gluster-users mailing list
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a>
<a href="http://gluster.org/cgi-bin/mailman/listinfo/gluster-users" target="_blank">http://gluster.org/cgi-bin/mailman/listinfo/gluster-users</a>
</pre>
</blockquote>
<br>
</div>
</blockquote>
</div>
<br>
</blockquote>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</blockquote>
<br>
</div></div></div>
</blockquote></div><br>