<html dir="ltr">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<style>
<!--
@font-face
{font-family:Wingdings}
@font-face
{font-family:Wingdings}
@font-face
{font-family:Calibri}
@font-face
{font-family:Tahoma}
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif"}
a:link, span.MsoHyperlink
{color:blue;
text-decoration:underline}
a:visited, span.MsoHyperlinkFollowed
{color:purple;
text-decoration:underline}
p
{margin-right:0in;
margin-left:0in;
font-size:12.0pt;
font-family:"Times New Roman","serif"}
p.MsoAcetate, li.MsoAcetate, div.MsoAcetate
{margin:0in;
margin-bottom:.0001pt;
font-size:8.0pt;
font-family:"Tahoma","sans-serif"}
span.BalloonTextChar
{font-family:"Tahoma","sans-serif"}
span.EmailStyle20
{font-family:"Calibri","sans-serif";
color:#1F497D}
span.EmailStyle21
{font-family:"Calibri","sans-serif";
color:#1F497D}
.MsoChpDefault
{font-size:10.0pt}
@page WordSection1
{margin:1.0in 1.0in 1.0in 1.0in}
-->
</style><style id="owaParaStyle" type="text/css">P {margin-top:0;margin-bottom:0;}</style>
</head>
<body ocsi="0" fpstyle="1" lang="EN-US" link="blue" vlink="purple">
<div style="direction: ltr;font-family: Tahoma;color: #000000;font-size: 10pt;">James - if that is indeed the case, then yes, we should update that.<br>
<br>
I will investigate and make sure the docs reflect reality.<br>
<br>
Thanks for bringing this to our attention.<br>
<br>
-JM<br>
Gluster Community Guy<br>
<br>
<br>
<div style="font-family: Times New Roman; color: rgb(0, 0, 0); font-size: 16px;">
<hr tabindex="-1">
<div style="direction: ltr;" id="divRpF703446"><font face="Tahoma" color="#000000" size="2"><b>From:</b> gluster-users-bounces@gluster.org [gluster-users-bounces@gluster.org] on behalf of Burnash, James [jburnash@knight.com]<br>
<b>Sent:</b> Thursday, June 30, 2011 10:03 AM<br>
<b>To:</b> 'Anand Avati'<br>
<b>Cc:</b> gluster-users@gluster.org<br>
<b>Subject:</b> Re: [Gluster-users] Possible new bug in 3.1.5 discovered<br>
</font><br>
</div>
<div></div>
<div>
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">Just to follow up on this – I went through the release notes for 3.1.5, 3.2, and 3.2.1. There is no mention of this issue – in fact, under the
compatibility section, it says:</span></p>
<p class="MsoNormal"><span style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);"> </span></p>
<p class="MsoNormal"><span style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">“Release 3.1.5 of GlusterFS is compatible with 3.1.x releases of GlusterFS.”</span></p>
<p class="MsoNormal"><span style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);"> </span></p>
<p class="MsoNormal"><span style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">This is obviously not true in the case of 3.1.5 clients with earlier 3.1.x servers, as mentioned by Avati, and should be so reflected in the documentation.</span></p>
<p class="MsoNormal"><span style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);"> </span></p>
<div>
<p class="MsoNormal"><span style="font-size: 9pt; font-family: "Arial","sans-serif"; color: rgb(0, 61, 121);">James Burnash</span></p>
<p class="MsoNormal"><span style="font-size: 9pt; font-family: "Arial","sans-serif"; color: rgb(0, 61, 121);">Unix Engineer</span></p>
<p class="MsoNormal"><span style="font-size: 8pt; font-family: "Arial","sans-serif"; color: rgb(0, 61, 121);">Knight Capital Group</span></p>
</div>
<p class="MsoNormal"><span style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);"> </span></p>
<div>
<div style="border-width: 1pt medium medium; border-style: solid none none; border-color: rgb(181, 196, 223) -moz-use-text-color -moz-use-text-color; -moz-border-top-colors: none; -moz-border-right-colors: none; -moz-border-bottom-colors: none; -moz-border-left-colors: none; -moz-border-image: none; padding: 3pt 0in 0in;">
<p class="MsoNormal"><b><span style="font-size: 10pt; font-family: "Tahoma","sans-serif";">From:</span></b><span style="font-size: 10pt; font-family: "Tahoma","sans-serif";"> gluster-users-bounces@gluster.org [mailto:gluster-users-bounces@gluster.org]
<b>On Behalf Of </b>Burnash, James<br>
<b>Sent:</b> Wednesday, June 29, 2011 11:26 AM<br>
<b>To:</b> 'Anand Avati'<br>
<b>Cc:</b> gluster-users@gluster.org<br>
<b>Subject:</b> Re: [Gluster-users] Possible new bug in 3.1.5 discovered</span></p>
</div>
</div>
<p class="MsoNormal"> </p>
<p class="MsoNormal"><span style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">Thank you.</span></p>
<p class="MsoNormal"><span style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);"> </span></p>
<p class="MsoNormal"><span style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">Would you be able to point me to the documentation of this in terms of release notes? It seems clear that I should read them more closely
</span><span style="font-size: 11pt; font-family: Wingdings; color: rgb(31, 73, 125);">J</span><span style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);"></span></p>
<p class="MsoNormal"><span style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);"> </span></p>
<p class="MsoNormal"><span style="font-size: 9pt; font-family: "Arial","sans-serif"; color: rgb(0, 61, 121);">James Burnash</span></p>
<p class="MsoNormal"><span style="font-size: 9pt; font-family: "Arial","sans-serif"; color: rgb(0, 61, 121);">Unix Engineer</span></p>
<p class="MsoNormal"><span style="font-size: 8pt; font-family: "Arial","sans-serif"; color: rgb(0, 61, 121);">Knight Capital Group</span></p>
<p class="MsoNormal"><span style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);"> </span></p>
<p class="MsoNormal"><b><span style="font-size: 10pt; font-family: "Tahoma","sans-serif";">From:</span></b><span style="font-size: 10pt; font-family: "Tahoma","sans-serif";"> Anand Avati [mailto:anand.avati@gmail.com]
<br>
<b>Sent:</b> Wednesday, June 29, 2011 11:17 AM<br>
<b>To:</b> Burnash, James<br>
<b>Cc:</b> gluster-users@gluster.org<br>
<b>Subject:</b> Re: [Gluster-users] Possible new bug in 3.1.5 discovered</span></p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">Compatibility was broken between 3.1.4 (and pre) servers and 3.1.5 clients (results in a hang when replicate translator is used). This compat breakage was "necessary" in order to fix a hang issue which was present in all 3.1.x till then.
New servers should work fine with old clients. Upgrade all your servers before upgrading the clients.</p>
<div>
<p class="MsoNormal"> </p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom: 12pt;">Avati</p>
<div>
<p class="MsoNormal">On Wed, Jun 29, 2011 at 8:23 PM, Burnash, James <<a href="mailto:jburnash@knight.com" target="_blank">jburnash@knight.com</a>> wrote:</p>
<div>
<div>
<p class="MsoNormal" style=""><span style="font-size: 11pt; color: rgb(31, 73, 125);">I’m sorry – I think I wasn’t clear.</span></p>
<p class="MsoNormal" style=""><span style="font-size: 11pt; color: rgb(31, 73, 125);"> </span></p>
<p class="MsoNormal" style=""><span style="font-size: 11pt; color: rgb(31, 73, 125);">The problem is that a 3.1.5 client used to write a file to GlusterFS native mount point on a server running 3.1.3 hangs.</span></p>
<p class="MsoNormal" style=""><span style="font-size: 11pt; color: rgb(31, 73, 125);"> </span></p>
<p class="MsoNormal" style=""><span style="font-size: 11pt; color: rgb(31, 73, 125);">Are you saying that the clients are known to not be backward compatible within the 3.1.x series?</span></p>
<div>
<p class="MsoNormal" style=""><span style="font-size: 11pt; color: rgb(31, 73, 125);"> </span></p>
<p class="MsoNormal" style=""><span style="font-size: 9pt; color: rgb(0, 61, 121);">James Burnash</span></p>
<p class="MsoNormal" style=""><span style="font-size: 9pt; color: rgb(0, 61, 121);">Unix Engineer</span></p>
<p class="MsoNormal" style=""><span style="font-size: 8pt; color: rgb(0, 61, 121);">Knight Capital Group</span></p>
<p class="MsoNormal" style=""><span style="font-size: 11pt; color: rgb(31, 73, 125);"> </span></p>
</div>
<p class="MsoNormal" style=""><b><span style="font-size: 10pt;">From:</span></b><span style="font-size: 10pt;"> Anand Avati [mailto:<a href="mailto:anand.avati@gmail.com" target="_blank">anand.avati@gmail.com</a>]
<br>
<b>Sent:</b> Wednesday, June 29, 2011 10:46 AM<br>
<b>To:</b> Burnash, James<br>
<b>Cc:</b> <a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a><br>
<b>Subject:</b> Re: [Gluster-users] Possible new bug in 3.1.5 discovered</span></p>
<div>
<div>
<p class="MsoNormal" style=""> </p>
<p class="MsoNormal" style="">James,</p>
<div>
<p class="MsoNormal" style=""> Both in 3.1.5 and 3.2.1 there were necessary locks hang fixes which went in and as a side effect clients and servers result in a hang when used across versions. Please upgrade your clients to 3.1.5 as well. This is a known, and
hard to fix compatibility issue.</p>
</div>
<div>
<p class="MsoNormal" style=""> </p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom: 12pt;">Avati</p>
<div>
<p class="MsoNormal" style="">On Wed, Jun 29, 2011 at 8:05 PM, Burnash, James <<a href="mailto:jburnash@knight.com" target="_blank">jburnash@knight.com</a>> wrote:</p>
<div>
<div>
<p class="MsoNormal" style=""><span style="font-size: 11pt; color: rgb(31, 73, 125);">“May you live in interesting times”</span></p>
<p class="MsoNormal" style=""><span style="font-size: 11pt; color: rgb(31, 73, 125);"> </span></p>
<p class="MsoNormal" style=""><span style="font-size: 11pt; color: rgb(31, 73, 125);">Is this a curse or a blessing?
</span><span style="font-size: 11pt; font-family: Wingdings; color: rgb(31, 73, 125);">J</span></p>
<p class="MsoNormal" style=""><span style="font-size: 11pt; color: rgb(31, 73, 125);"> </span></p>
<p class="MsoNormal" style=""><span style="font-size: 11pt; color: rgb(31, 73, 125);">I’ve just tested a 3.1.5 GlusterFS native client against a 3.1.3 storage pool using this volume:</span></p>
<p class="MsoNormal" style=""><span style="font-size: 11pt; color: rgb(31, 73, 125);"> </span></p>
<p class="MsoNormal" style=""><span style="font-size: 11pt; color: rgb(31, 73, 125);">Volume Name: pfs-rw1</span></p>
<p class="MsoNormal" style=""><span style="font-size: 11pt; color: rgb(31, 73, 125);">Type: Distributed-Replicate</span></p>
<p class="MsoNormal" style=""><span style="font-size: 11pt; color: rgb(31, 73, 125);">Status: Started</span></p>
<p class="MsoNormal" style=""><span style="font-size: 11pt; color: rgb(31, 73, 125);">Number of Bricks: 2 x 2 = 4</span></p>
<p class="MsoNormal" style=""><span style="font-size: 11pt; color: rgb(31, 73, 125);">Transport-type: tcp</span></p>
<p class="MsoNormal" style=""><span style="font-size: 11pt; color: rgb(31, 73, 125);">Bricks:</span></p>
<p class="MsoNormal" style=""><span style="font-size: 11pt; color: rgb(31, 73, 125);">Brick1: jc1letgfs16-pfs1:/export/read-write/g01</span></p>
<p class="MsoNormal" style=""><span style="font-size: 11pt; color: rgb(31, 73, 125);">Brick2: jc1letgfs13-pfs1:/export/read-write/g01</span></p>
<p class="MsoNormal" style=""><span style="font-size: 11pt; color: rgb(31, 73, 125);">Brick3: jc1letgfs16-pfs1:/export/read-write/g02</span></p>
<p class="MsoNormal" style=""><span style="font-size: 11pt; color: rgb(31, 73, 125);">Brick4: jc1letgfs13-pfs1:/export/read-write/g02</span></p>
<p class="MsoNormal" style=""><span style="font-size: 11pt; color: rgb(31, 73, 125);">Options Reconfigured:</span></p>
<p class="MsoNormal" style=""><span style="font-size: 11pt; color: rgb(31, 73, 125);">performance.cache-size: 2GB</span></p>
<p class="MsoNormal" style=""><span style="font-size: 11pt; color: rgb(31, 73, 125);">performance.stat-prefetch: 0</span></p>
<p class="MsoNormal" style=""><span style="font-size: 11pt; color: rgb(31, 73, 125);">network.ping-timeout: 10</span></p>
<p class="MsoNormal" style=""><span style="font-size: 11pt; color: rgb(31, 73, 125);">diagnostics.client-log-level: ERROR</span></p>
<p class="MsoNormal" style=""><span style="font-size: 11pt; color: rgb(31, 73, 125);"> </span></p>
<p class="MsoNormal" style=""><span style="font-size: 11pt; color: rgb(31, 73, 125);">Any attempt to write to that volume mounted on a native client using version 3.1.5 results in a hang at the command line, which I can only break out of by killing my ssh session
into the client. Upon logging back into the same client, I see a zombie process from the attempt to write:</span></p>
<p class="MsoNormal" style=""><span style="font-size: 11pt; color: rgb(31, 73, 125);"> </span></p>
<p class="MsoNormal" style=""><span style="font-size: 11pt; color: rgb(31, 73, 125);">21172 ? D 0:00 touch /pfs1/test/junk1</span></p>
<p class="MsoNormal" style=""><span style="font-size: 11pt; color: rgb(31, 73, 125);"> </span></p>
<p class="MsoNormal" style=""><span style="font-size: 11pt; color: rgb(31, 73, 125);">Anybody else run into this situation?</span></p>
<p class="MsoNormal" style=""><span style="font-size: 11pt; color: rgb(31, 73, 125);"> </span></p>
<p class="MsoNormal" style=""><span style="font-size: 11pt; color: rgb(31, 73, 125);">Client mount log (/var/log/glusterfs/pfs2.log) below:</span></p>
<p class="MsoNormal" style=""><span style="font-size: 11pt; color: rgb(31, 73, 125);"> </span></p>
<p class="MsoNormal" style=""><span style="font-size: 11pt; color: rgb(31, 73, 125);">[2011-06-29 10:28:07.860519] E [afr-self-heal-metadata.c:522:afr_sh_metadata_fix] 0-pfs-ro1-replicate-6: Unable to self-heal permissions/ownership of '/' (possible split-brain).
Please fix the file on all backend volumes</span></p>
<p class="MsoNormal" style=""><span style="font-size: 11pt; color: rgb(31, 73, 125);">[2011-06-29 10:28:07.860668] E [afr-self-heal-metadata.c:522:afr_sh_metadata_fix] 0-pfs-ro1-replicate-1: Unable to self-heal permissions/ownership of '/' (possible split-brain).
Please fix the file on all backend volumes</span></p>
<p class="MsoNormal" style=""><span style="font-size: 11pt; color: rgb(31, 73, 125);">s/ownership of '/' (possible split-brain). Please fix the file on all backend volumes</span></p>
<p class="MsoNormal" style=""><span style="font-size: 11pt; color: rgb(31, 73, 125);">s/ownership of '/' (possible split-brain). Please fix the file on all backend volumes</span></p>
<p class="MsoNormal" style=""><span style="font-size: 11pt; color: rgb(31, 73, 125);">s/ownership of '/' (possible split-brain). Please fix the file on all backend volumes</span></p>
<p class="MsoNormal" style=""><span style="font-size: 11pt; color: rgb(31, 73, 125);">s/ownership of '/' (possible split-brain). Please fix the file on all backend volumes</span></p>
<p class="MsoNormal" style=""><span style="font-size: 11pt; color: rgb(31, 73, 125);">ns/ownership of '/' (possible split-brain). Please fix the file on all backend volumes</span></p>
<p class="MsoNormal" style=""><span style="font-size: 11pt; color: rgb(31, 73, 125);">data self-heal failed on /</span></p>
<p class="MsoNormal" style=""><span style="font-size: 11pt; color: rgb(31, 73, 125);">data self-heal failed on /</span></p>
<p class="MsoNormal" style=""><span style="font-size: 11pt; color: rgb(31, 73, 125);">s/ownership of '/' (possible split-brain). Please fix the file on all backend volumes</span></p>
<p class="MsoNormal" style=""><span style="font-size: 11pt; color: rgb(31, 73, 125);">ns/ownership of '/' (possible split-brain). Please fix the file on all backend volumes</span></p>
<p class="MsoNormal" style=""><span style="font-size: 11pt; color: rgb(31, 73, 125);">data self-heal failed on /</span></p>
<p class="MsoNormal" style=""><span style="font-size: 11pt; color: rgb(31, 73, 125);">data self-heal failed on /</span></p>
<p class="MsoNormal" style=""><span style="font-size: 11pt; color: rgb(31, 73, 125);">data self-heal failed on /</span></p>
<p class="MsoNormal" style=""><span style="font-size: 11pt; color: rgb(31, 73, 125);">s/ownership of '/' (possible split-brain). Please fix the file on all backend volumes</span></p>
<p class="MsoNormal" style=""><span style="font-size: 11pt; color: rgb(31, 73, 125);">s/ownership of '/' (possible split-brain). Please fix the file on all backend volumes</span></p>
<p class="MsoNormal" style=""><span style="font-size: 11pt; color: rgb(31, 73, 125);">data self-heal failed on /</span></p>
<p class="MsoNormal" style=""><span style="font-size: 11pt; color: rgb(31, 73, 125);">data self-heal failed on /</span></p>
<p class="MsoNormal" style=""><span style="font-size: 11pt; color: rgb(31, 73, 125);">data self-heal failed on /</span></p>
<p class="MsoNormal" style=""><span style="font-size: 11pt; color: rgb(31, 73, 125);">s/ownership of '/' (possible split-brain). Please fix the file on all backend volumes</span></p>
<p class="MsoNormal" style=""><span style="font-size: 11pt; color: rgb(31, 73, 125);">data self-heal failed on /</span></p>
<p class="MsoNormal" style=""><span style="font-size: 11pt; color: rgb(31, 73, 125);">data self-heal failed on /</span></p>
<p class="MsoNormal" style=""><span style="font-size: 11pt; color: rgb(31, 73, 125);">data self-heal failed on /</span></p>
<p class="MsoNormal" style=""><span style="font-size: 11pt; color: rgb(31, 73, 125);">data self-heal failed on /</span></p>
<p class="MsoNormal" style=""><span style="font-size: 11pt; color: rgb(31, 73, 125);"> </span></p>
<div>
<p class="MsoNormal" style=""><span style="font-size: 9pt; color: rgb(0, 61, 121);">James Burnash</span></p>
<p class="MsoNormal" style=""><span style="font-size: 9pt; color: rgb(0, 61, 121);">Unix Engineer</span></p>
<p class="MsoNormal" style=""><span style="font-size: 8pt; color: rgb(0, 61, 121);">Knight Capital Group</span></p>
</div>
<p class="MsoNormal" style=""><span style="font-size: 11pt; color: rgb(31, 73, 125);"> </span></p>
</div>
<p class="MsoNormal" style="margin-bottom: 12pt;"> </p>
<p><span style="font-size: 7.5pt; color: navy;">DISCLAIMER:<br>
This e-mail, and any attachments thereto, is intended only for use by the addressee(s)named herein and<br>
may contain legally privileged and/or confidential information. If you are not the intended recipient of this<br>
e-mail, you are hereby notified that any dissemination, distribution or copying of this e-mail and any attachments</span><span style="color: navy;"><br>
</span><span style="font-size: 7.5pt; color: navy;">thereto, is strictly prohibited. If you have received this in error, please immediately notify me and permanently<br>
delete the original and any printout thereof.</span><span style="font-size: 7.5pt;">
<span style="color: navy;">E-mail transmission cannot be guaranteed to be secure or error-free.<br>
The sender therefore does not accept liability for any errors or omissions in the contents of this message which<br>
arise as a result of e-mail transmission.<br>
NOTICE REGARDING PRIVACY AND CONFIDENTIALITY<br>
Knight Capital Group may, at its discretion, monitor and review the content of all e-mail communications.</span></span></p>
<p><a href="http://www.knight.com/" target="_blank"><span style="font-size: 7.5pt; color: navy;">http://www.knight.com</span></a></p>
<p class="MsoNormal" style=""> </p>
</div>
<p class="MsoNormal" style="margin-bottom: 12pt;"><br>
_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="http://gluster.org/cgi-bin/mailman/listinfo/gluster-users" target="_blank">http://gluster.org/cgi-bin/mailman/listinfo/gluster-users</a></p>
</div>
<p class="MsoNormal" style=""> </p>
</div>
</div>
</div>
</div>
</div>
</div>
<p class="MsoNormal"> </p>
</div>
</div>
</div>
</div>
</div>
</body>
</html>