Compatibility was broken between 3.1.4 (and pre) servers and 3.1.5 clients (results in a hang when replicate translator is used). This compat breakage was "necessary" in order to fix a hang issue which was present in all 3.1.x till then. New servers should work fine with old clients. Upgrade all your servers before upgrading the clients.<div>
<br></div><div>Avati<br><br><div class="gmail_quote">On Wed, Jun 29, 2011 at 8:23 PM, Burnash, James <span dir="ltr"><<a href="mailto:jburnash@knight.com">jburnash@knight.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<div lang="EN-US" link="blue" vlink="purple"><div><p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">I’m sorry – I think I wasn’t clear.<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">The problem is that a 3.1.5 client used to write a file to GlusterFS native mount point on a server running 3.1.3 hangs.<u></u><u></u></span></p><p class="MsoNormal">
<span style="font-size:11.0pt;color:#1F497D"><u></u> <u></u></span></p><p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">Are you saying that the clients are known to not be backward compatible within the 3.1.x series?<u></u><u></u></span></p>
<div class="im"><p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D"><u></u> <u></u></span></p><p class="MsoNormal"><span style="font-size:9.0pt;color:#003D79">James Burnash<u></u><u></u></span></p><p class="MsoNormal">
<span style="font-size:9.0pt;color:#003D79">Unix Engineer<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:8.0pt;color:#003D79">Knight Capital Group<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D"><u></u> <u></u></span></p>
</div><p class="MsoNormal"><b><span style="font-size:10.0pt">From:</span></b><span style="font-size:10.0pt"> Anand Avati [mailto:<a href="mailto:anand.avati@gmail.com" target="_blank">anand.avati@gmail.com</a>] <br><b>Sent:</b> Wednesday, June 29, 2011 10:46 AM<br>
<b>To:</b> Burnash, James<br><b>Cc:</b> <a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a><br><b>Subject:</b> Re: [Gluster-users] Possible new bug in 3.1.5 discovered<u></u><u></u></span></p>
<div><div></div><div class="h5"><p class="MsoNormal"><u></u> <u></u></p><p class="MsoNormal">James,<u></u><u></u></p><div><p class="MsoNormal"> Both in 3.1.5 and 3.2.1 there were necessary locks hang fixes which went in and as a side effect clients and servers result in a hang when used across versions. Please upgrade your clients to 3.1.5 as well. This is a known, and hard to fix compatibility issue.<u></u><u></u></p>
</div><div><p class="MsoNormal"><u></u> <u></u></p></div><div><p class="MsoNormal" style="margin-bottom:12.0pt">Avati<u></u><u></u></p><div><p class="MsoNormal">On Wed, Jun 29, 2011 at 8:05 PM, Burnash, James <<a href="mailto:jburnash@knight.com" target="_blank">jburnash@knight.com</a>> wrote:<u></u><u></u></p>
<div><div><p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">“May you live in interesting times”</span><u></u><u></u></p><p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">Is this a curse or a blessing? </span><span style="font-size:11.0pt;font-family:Wingdings;color:#1F497D">J</span><u></u><u></u></p><p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">I’ve just tested a 3.1.5 GlusterFS native client against a 3.1.3 storage pool using this volume:</span><u></u><u></u></p><p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">Volume Name: pfs-rw1</span><u></u><u></u></p><p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">Type: Distributed-Replicate</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">Status: Started</span><u></u><u></u></p><p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">Number of Bricks: 2 x 2 = 4</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">Transport-type: tcp</span><u></u><u></u></p><p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">Bricks:</span><u></u><u></u></p><p class="MsoNormal">
<span style="font-size:11.0pt;color:#1F497D">Brick1: jc1letgfs16-pfs1:/export/read-write/g01</span><u></u><u></u></p><p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">Brick2: jc1letgfs13-pfs1:/export/read-write/g01</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">Brick3: jc1letgfs16-pfs1:/export/read-write/g02</span><u></u><u></u></p><p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">Brick4: jc1letgfs13-pfs1:/export/read-write/g02</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">Options Reconfigured:</span><u></u><u></u></p><p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">performance.cache-size: 2GB</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">performance.stat-prefetch: 0</span><u></u><u></u></p><p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">network.ping-timeout: 10</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">diagnostics.client-log-level: ERROR</span><u></u><u></u></p><p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D"> </span><u></u><u></u></p><p class="MsoNormal">
<span style="font-size:11.0pt;color:#1F497D">Any attempt to write to that volume mounted on a native client using version 3.1.5 results in a hang at the command line, which I can only break out of by killing my ssh session into the client. Upon logging back into the same client, I see a zombie process from the attempt to write:</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D"> </span><u></u><u></u></p><p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">21172 ? D 0:00 touch /pfs1/test/junk1</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D"> </span><u></u><u></u></p><p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">Anybody else run into this situation?</span><u></u><u></u></p><p class="MsoNormal">
<span style="font-size:11.0pt;color:#1F497D"> </span><u></u><u></u></p><p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">Client mount log (/var/log/glusterfs/pfs2.log) below:</span><u></u><u></u></p><p class="MsoNormal">
<span style="font-size:11.0pt;color:#1F497D"> </span><u></u><u></u></p><p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">[2011-06-29 10:28:07.860519] E [afr-self-heal-metadata.c:522:afr_sh_metadata_fix] 0-pfs-ro1-replicate-6: Unable to self-heal permissions/ownership of '/' (possible split-brain). Please fix the file on all backend volumes</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">[2011-06-29 10:28:07.860668] E [afr-self-heal-metadata.c:522:afr_sh_metadata_fix] 0-pfs-ro1-replicate-1: Unable to self-heal permissions/ownership of '/' (possible split-brain). Please fix the file on all backend volumes</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">s/ownership of '/' (possible split-brain). Please fix the file on all backend volumes</span><u></u><u></u></p><p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">s/ownership of '/' (possible split-brain). Please fix the file on all backend volumes</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">s/ownership of '/' (possible split-brain). Please fix the file on all backend volumes</span><u></u><u></u></p><p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">s/ownership of '/' (possible split-brain). Please fix the file on all backend volumes</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">ns/ownership of '/' (possible split-brain). Please fix the file on all backend volumes</span><u></u><u></u></p><p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">data self-heal failed on /</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">data self-heal failed on /</span><u></u><u></u></p><p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">s/ownership of '/' (possible split-brain). Please fix the file on all backend volumes</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">ns/ownership of '/' (possible split-brain). Please fix the file on all backend volumes</span><u></u><u></u></p><p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">data self-heal failed on /</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">data self-heal failed on /</span><u></u><u></u></p><p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">data self-heal failed on /</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">s/ownership of '/' (possible split-brain). Please fix the file on all backend volumes</span><u></u><u></u></p><p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">s/ownership of '/' (possible split-brain). Please fix the file on all backend volumes</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">data self-heal failed on /</span><u></u><u></u></p><p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">data self-heal failed on /</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">data self-heal failed on /</span><u></u><u></u></p><p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">s/ownership of '/' (possible split-brain). Please fix the file on all backend volumes</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">data self-heal failed on /</span><u></u><u></u></p><p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">data self-heal failed on /</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">data self-heal failed on /</span><u></u><u></u></p><p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">data self-heal failed on /</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D"> </span><u></u><u></u></p><div><p class="MsoNormal"><span style="font-size:9.0pt;color:#003D79">James Burnash</span><u></u><u></u></p><p class="MsoNormal">
<span style="font-size:9.0pt;color:#003D79">Unix Engineer</span><u></u><u></u></p><p class="MsoNormal"><span style="font-size:8.0pt;color:#003D79">Knight Capital Group</span><u></u><u></u></p></div><p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D"> </span><u></u><u></u></p>
</div><p class="MsoNormal" style="margin-bottom:12.0pt"><u></u> <u></u></p><p><span style="font-size:7.5pt;color:navy">DISCLAIMER:<br>This e-mail, and any attachments thereto, is intended only for use by the addressee(s)named herein and<br>
may contain legally privileged and/or confidential information. If you are not the intended recipient of this<br>e-mail, you are hereby notified that any dissemination, distribution or copying of this e-mail and any attachments</span><span style="color:navy"><br>
</span><span style="font-size:7.5pt;color:navy">thereto, is strictly prohibited. If you have received this in error, please immediately notify me and permanently<br>delete the original and any printout thereof.</span><span style="font-size:7.5pt"> <span style="color:navy">E-mail transmission cannot be guaranteed to be secure or error-free.<br>
The sender therefore does not accept liability for any errors or omissions in the contents of this message which<br>arise as a result of e-mail transmission.<br>NOTICE REGARDING PRIVACY AND CONFIDENTIALITY<br>Knight Capital Group may, at its discretion, monitor and review the content of all e-mail communications.</span></span><u></u><u></u></p>
<p><a href="http://www.knight.com/" target="_blank"><span style="font-size:7.5pt;color:navy">http://www.knight.com</span></a><u></u><u></u></p><p class="MsoNormal"> <u></u><u></u></p></div><p class="MsoNormal" style="margin-bottom:12.0pt">
<br>_______________________________________________<br>Gluster-users mailing list<br><a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br><a href="http://gluster.org/cgi-bin/mailman/listinfo/gluster-users" target="_blank">http://gluster.org/cgi-bin/mailman/listinfo/gluster-users</a><u></u><u></u></p>
</div><p class="MsoNormal"><u></u> <u></u></p></div></div></div></div></div></blockquote></div><br></div>