<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=ISO-8859-1">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<br>
<div class="moz-forward-container"><br>
<br>
-------- Original Message --------
<table class="moz-email-headers-table" border="0" cellpadding="0"
cellspacing="0">
<tbody>
<tr>
<th align="RIGHT" nowrap="nowrap" valign="BASELINE">Subject:
</th>
<td>Re: [Gluster-users] Fencing FOPs on data-split-brained
files</td>
</tr>
<tr>
<th align="RIGHT" nowrap="nowrap" valign="BASELINE">Date: </th>
<td>Tue, 19 Nov 2013 16:03:14 +0530</td>
</tr>
<tr>
<th align="RIGHT" nowrap="nowrap" valign="BASELINE">From: </th>
<td>Ravishankar N <a class="moz-txt-link-rfc2396E" href="mailto:ravishankar@redhat.com"><ravishankar@redhat.com></a></td>
</tr>
<tr>
<th align="RIGHT" nowrap="nowrap" valign="BASELINE">To: </th>
<td>Anand Avati <a class="moz-txt-link-rfc2396E" href="mailto:avati@gluster.org"><avati@gluster.org></a></td>
</tr>
<tr>
<th align="RIGHT" nowrap="nowrap" valign="BASELINE">CC: </th>
<td>Gluster Devel <a class="moz-txt-link-rfc2396E" href="mailto:gluster-devel@nongnu.org"><gluster-devel@nongnu.org></a>,
<a class="moz-txt-link-rfc2396E" href="mailto:gluster-users@gluster.org">"gluster-users@gluster.org"</a>
<a class="moz-txt-link-rfc2396E" href="mailto:gluster-users@gluster.org"><gluster-users@gluster.org></a></td>
</tr>
</tbody>
</table>
<br>
<br>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
<div class="moz-cite-prefix">On 11/16/2013 01:42 AM, Anand Avati
wrote:<br>
</div>
<blockquote
cite="mid:CAFboF2wZneo=nov=aBsP3257s-+43PfFDGS1rm1-_vkaaLHEPg@mail.gmail.com"
type="cite">
<div dir="ltr">
<div>Ravi,</div>
<div>We should not mix up data and entry operation domains, if
a file is in data split brain that should not stop a user
from rename/link/unlink operations on the file.</div>
<div><br>
</div>
<div> Regarding your concern about complications while healing
- we should change our "manual fixing" instructions to:</div>
<div><br>
</div>
<div>- go to backend, access through gfid path or normal path</div>
<div>- rmxattr the afr changelogs</div>
<div>- truncate the file to 0 bytes (like "> filename")</div>
<div><br>
</div>
<div>Accessing the path through gfid and truncating to 0 bytes
addresses your concerns about hardlinks/renames.</div>
<div><br>
</div>
<div>Avati</div>
</div>
<div class="gmail_extra"><br>
</div>
</blockquote>
<br>
<br>
<meta http-equiv="CONTENT-TYPE" content="text/html;
charset=ISO-8859-1">
<p class="western" style="margin-bottom: 0in"><i>Resending the
mail again as there was no response<br>
-Ravi<br>
</i></p>
<p class="western" style="margin-bottom: 0in">All,<br>
<br>
I have tabulated what operations must/ mustn't be permitted in
case of different split brains. Some of the columns are '?' as I
am not sure what the expected behaviour should be. Could we have
this validated? <br>
</p>
<p class="western" style="margin-bottom: 0in"><br>
</p>
<meta http-equiv="CONTENT-TYPE" content="text/html;
charset=ISO-8859-1">
<table cols="5" border="0" cellspacing="0">
<colgroup width="139"></colgroup> <colgroup width="119"></colgroup>
<colgroup width="180"></colgroup> <colgroup width="224"></colgroup>
<colgroup width="192"></colgroup> <tbody>
<tr>
<td style="border-top: 1px solid #000000; border-bottom: 1px
solid #000000; border-left: 1px solid #000000;
border-right: 1px solid #000000" rowspan="3" height="53"
align="CENTER" bgcolor="#CFE7F5" valign="MIDDLE"><big><b>File
Operation permitted</b></big></td>
<td style="border-top: 1px solid #000000; border-bottom: 1px
solid #000000; border-left: 1px solid #000000;
border-right: 1px solid #000000" colspan="4"
align="CENTER" bgcolor="#00CCCC" valign="MIDDLE"><big><b>Type
of Split Brain</b></big></td>
</tr>
<tr>
<td style="border-top: 1px solid #000000; border-bottom: 1px
solid #000000; border-left: 1px solid #000000;
border-right: 1px solid #000000" align="CENTER"><big><b>Data
SB</b></big></td>
<td style="border-top: 1px solid #000000; border-bottom: 1px
solid #000000; border-left: 1px solid #000000;
border-right: 1px solid #000000" align="CENTER"><big><b>Metadata
SB</b></big></td>
<td style="border-top: 1px solid #000000; border-bottom: 1px
solid #000000; border-left: 1px solid #000000;
border-right: 1px solid #000000" colspan="2"
align="CENTER" valign="MIDDLE"><big><b>Entry SB</b></big></td>
</tr>
<tr>
<td style="border-top: 1px solid #000000; border-bottom: 1px
solid #000000; border-left: 1px solid #000000;
border-right: 1px solid #000000" align="CENTER"><big><b><br>
</b></big></td>
<td style="border-top: 1px solid #000000; border-bottom: 1px
solid #000000; border-left: 1px solid #000000;
border-right: 1px solid #000000" align="CENTER"><big><b><br>
</b></big></td>
<td style="border-top: 1px solid #000000; border-bottom: 1px
solid #000000; border-left: 1px solid #000000;
border-right: 1px solid #000000" align="CENTER"><big><b>Same
entry gfid mismatch SB</b></big></td>
<td style="border-top: 1px solid #000000; border-bottom: 1px
solid #000000; border-left: 1px solid #000000;
border-right: 1px solid #000000" align="CENTER"><big><b>Different
entries</b></big></td>
</tr>
<tr>
<td style="border-top: 1px solid #000000; border-bottom: 1px
solid #000000; border-left: 1px solid #000000;
border-right: 1px solid #000000" height="17" align="LEFT"><big>write</big></td>
<td style="border-top: 1px solid #000000; border-bottom: 1px
solid #000000; border-left: 1px solid #000000;
border-right: 1px solid #000000" align="CENTER"><big>No</big></td>
<td style="border-top: 1px solid #000000; border-bottom: 1px
solid #000000; border-left: 1px solid #000000;
border-right: 1px solid #000000" align="CENTER"><big>Yes
(currently no)</big></td>
<td style="border-top: 1px solid #000000; border-bottom: 1px
solid #000000; border-left: 1px solid #000000;
border-right: 1px solid #000000" align="CENTER"><big>No</big></td>
<td style="border-top: 1px solid #000000; border-bottom: 1px
solid #000000; border-left: 1px solid #000000;
border-right: 1px solid #000000" align="CENTER"><big>Yes</big></td>
</tr>
<tr>
<td style="border-top: 1px solid #000000; border-bottom: 1px
solid #000000; border-left: 1px solid #000000;
border-right: 1px solid #000000" height="17" align="LEFT"><big>read</big></td>
<td style="border-top: 1px solid #000000; border-bottom: 1px
solid #000000; border-left: 1px solid #000000;
border-right: 1px solid #000000" align="CENTER"><big>No</big></td>
<td style="border-top: 1px solid #000000; border-bottom: 1px
solid #000000; border-left: 1px solid #000000;
border-right: 1px solid #000000" align="CENTER"><big>Yes
(currently no)</big></td>
<td style="border-top: 1px solid #000000; border-bottom: 1px
solid #000000; border-left: 1px solid #000000;
border-right: 1px solid #000000" align="CENTER"><big>No</big></td>
<td style="border-top: 1px solid #000000; border-bottom: 1px
solid #000000; border-left: 1px solid #000000;
border-right: 1px solid #000000" align="CENTER"><big>Yes</big></td>
</tr>
<tr>
<td style="border-top: 1px solid #000000; border-bottom: 1px
solid #000000; border-left: 1px solid #000000;
border-right: 1px solid #000000" height="18" align="LEFT"><big>getfattr</big></td>
<td style="border-top: 1px solid #000000; border-bottom: 1px
solid #000000; border-left: 1px solid #000000;
border-right: 1px solid #000000" align="CENTER"><big>Yes</big></td>
<td style="border-top: 1px solid #000000; border-bottom: 1px
solid #000000; border-left: 1px solid #000000;
border-right: 1px solid #000000" align="CENTER"><big>No</big></td>
<td style="border-top: 1px solid #000000; border-bottom: 1px
solid #000000; border-left: 1px solid #000000;
border-right: 1px solid #000000" align="CENTER"><big>No</big></td>
<td style="border-top: 1px solid #000000; border-bottom: 1px
solid #000000; border-left: 1px solid #000000;
border-right: 1px solid #000000" align="CENTER"><big>Yes</big></td>
</tr>
<tr>
<td style="border-top: 1px solid #000000; border-bottom: 1px
solid #000000; border-left: 1px solid #000000;
border-right: 1px solid #000000" height="17" align="LEFT"><big>lookup</big></td>
<td style="border-top: 1px solid #000000; border-bottom: 1px
solid #000000; border-left: 1px solid #000000;
border-right: 1px solid #000000" align="CENTER"><big>?</big></td>
<td style="border-top: 1px solid #000000; border-bottom: 1px
solid #000000; border-left: 1px solid #000000;
border-right: 1px solid #000000" align="CENTER"><big>?</big></td>
<td style="border-top: 1px solid #000000; border-bottom: 1px
solid #000000; border-left: 1px solid #000000;
border-right: 1px solid #000000" align="CENTER"><big>No</big></td>
<td style="border-top: 1px solid #000000; border-bottom: 1px
solid #000000; border-left: 1px solid #000000;
border-right: 1px solid #000000" align="CENTER"><big>Yes</big></td>
</tr>
<tr>
<td style="border-top: 1px solid #000000; border-bottom: 1px
solid #000000; border-left: 1px solid #000000;
border-right: 1px solid #000000" height="18" align="LEFT"><big>stat/fstat</big></td>
<td style="border-top: 1px solid #000000; border-bottom: 1px
solid #000000; border-left: 1px solid #000000;
border-right: 1px solid #000000" align="CENTER"><big>?</big></td>
<td style="border-top: 1px solid #000000; border-bottom: 1px
solid #000000; border-left: 1px solid #000000;
border-right: 1px solid #000000" align="CENTER"><big>?</big></td>
<td style="border-top: 1px solid #000000; border-bottom: 1px
solid #000000; border-left: 1px solid #000000;
border-right: 1px solid #000000" align="CENTER"><big>No</big></td>
<td style="border-top: 1px solid #000000; border-bottom: 1px
solid #000000; border-left: 1px solid #000000;
border-right: 1px solid #000000" align="CENTER"><big>Yes</big></td>
</tr>
<tr>
<td style="border-top: 1px solid #000000; border-bottom: 1px
solid #000000; border-left: 1px solid #000000;
border-right: 1px solid #000000" height="18" align="LEFT"><big>setfattr</big></td>
<td style="border-top: 1px solid #000000; border-bottom: 1px
solid #000000; border-left: 1px solid #000000;
border-right: 1px solid #000000" align="CENTER"><big>Yes</big></td>
<td style="border-top: 1px solid #000000; border-bottom: 1px
solid #000000; border-left: 1px solid #000000;
border-right: 1px solid #000000" align="CENTER"><big>No</big></td>
<td style="border-top: 1px solid #000000; border-bottom: 1px
solid #000000; border-left: 1px solid #000000;
border-right: 1px solid #000000" align="CENTER"><big>No</big></td>
<td style="border-top: 1px solid #000000; border-bottom: 1px
solid #000000; border-left: 1px solid #000000;
border-right: 1px solid #000000" align="CENTER"><big>Yes</big></td>
</tr>
<tr>
<td style="border-top: 1px solid #000000; border-bottom: 1px
solid #000000; border-left: 1px solid #000000;
border-right: 1px solid #000000" height="17" align="LEFT"><big>touch</big></td>
<td style="border-top: 1px solid #000000; border-bottom: 1px
solid #000000; border-left: 1px solid #000000;
border-right: 1px solid #000000" align="CENTER"><big>Yes</big></td>
<td style="border-top: 1px solid #000000; border-bottom: 1px
solid #000000; border-left: 1px solid #000000;
border-right: 1px solid #000000" align="CENTER"><big>Yes</big></td>
<td style="border-top: 1px solid #000000; border-bottom: 1px
solid #000000; border-left: 1px solid #000000;
border-right: 1px solid #000000" align="CENTER"><big>No</big></td>
<td style="border-top: 1px solid #000000; border-bottom: 1px
solid #000000; border-left: 1px solid #000000;
border-right: 1px solid #000000" align="CENTER"><big>Yes</big></td>
</tr>
<tr>
<td style="border-top: 1px solid #000000; border-bottom: 1px
solid #000000; border-left: 1px solid #000000;
border-right: 1px solid #000000" height="17" align="LEFT"><big>hard
link creation</big></td>
<td style="border-top: 1px solid #000000; border-bottom: 1px
solid #000000; border-left: 1px solid #000000;
border-right: 1px solid #000000" align="CENTER"><big>Yes</big></td>
<td style="border-top: 1px solid #000000; border-bottom: 1px
solid #000000; border-left: 1px solid #000000;
border-right: 1px solid #000000" align="CENTER"><big>Yes</big></td>
<td style="border-top: 1px solid #000000; border-bottom: 1px
solid #000000; border-left: 1px solid #000000;
border-right: 1px solid #000000" align="CENTER"><big>No</big></td>
<td style="border-top: 1px solid #000000; border-bottom: 1px
solid #000000; border-left: 1px solid #000000;
border-right: 1px solid #000000" align="CENTER"><big>Yes</big></td>
</tr>
<tr>
<td style="border-top: 1px solid #000000; border-bottom: 1px
solid #000000; border-left: 1px solid #000000;
border-right: 1px solid #000000" height="17" align="LEFT"><big>soft
link creation</big></td>
<td style="border-top: 1px solid #000000; border-bottom: 1px
solid #000000; border-left: 1px solid #000000;
border-right: 1px solid #000000" align="CENTER"><big>Yes</big></td>
<td style="border-top: 1px solid #000000; border-bottom: 1px
solid #000000; border-left: 1px solid #000000;
border-right: 1px solid #000000" align="CENTER"><big>Yes</big></td>
<td style="border-top: 1px solid #000000; border-bottom: 1px
solid #000000; border-left: 1px solid #000000;
border-right: 1px solid #000000" align="CENTER"><big>Yes</big></td>
<td style="border-top: 1px solid #000000; border-bottom: 1px
solid #000000; border-left: 1px solid #000000;
border-right: 1px solid #000000" align="CENTER"><big>Yes</big></td>
</tr>
<tr>
<td style="border-top: 1px solid #000000; border-bottom: 1px
solid #000000; border-left: 1px solid #000000;
border-right: 1px solid #000000" height="17" align="LEFT"><big>rename</big></td>
<td style="border-top: 1px solid #000000; border-bottom: 1px
solid #000000; border-left: 1px solid #000000;
border-right: 1px solid #000000" align="CENTER"><big>Yes</big></td>
<td style="border-top: 1px solid #000000; border-bottom: 1px
solid #000000; border-left: 1px solid #000000;
border-right: 1px solid #000000" align="CENTER"><big>Yes</big></td>
<td style="border-top: 1px solid #000000; border-bottom: 1px
solid #000000; border-left: 1px solid #000000;
border-right: 1px solid #000000" align="CENTER"><big>no</big></td>
<td style="border-top: 1px solid #000000; border-bottom: 1px
solid #000000; border-left: 1px solid #000000;
border-right: 1px solid #000000" align="CENTER"><big>Yes</big></td>
</tr>
<tr>
<td style="border-top: 1px solid #000000; border-bottom: 1px
solid #000000; border-left: 1px solid #000000;
border-right: 1px solid #000000" height="18" align="LEFT"><big>chown</big></td>
<td style="border-top: 1px solid #000000; border-bottom: 1px
solid #000000; border-left: 1px solid #000000;
border-right: 1px solid #000000" align="CENTER"><big>Yes</big></td>
<td style="border-top: 1px solid #000000; border-bottom: 1px
solid #000000; border-left: 1px solid #000000;
border-right: 1px solid #000000" align="CENTER"><big>Yes</big></td>
<td style="border-top: 1px solid #000000; border-bottom: 1px
solid #000000; border-left: 1px solid #000000;
border-right: 1px solid #000000" align="CENTER"><big>Currently
No</big></td>
<td style="border-top: 1px solid #000000; border-bottom: 1px
solid #000000; border-left: 1px solid #000000;
border-right: 1px solid #000000" align="CENTER"><big>Yes</big></td>
</tr>
<tr>
<td style="border-top: 1px solid #000000; border-bottom: 1px
solid #000000; border-left: 1px solid #000000;
border-right: 1px solid #000000" height="18" align="LEFT"><big>chmod</big></td>
<td style="border-top: 1px solid #000000; border-bottom: 1px
solid #000000; border-left: 1px solid #000000;
border-right: 1px solid #000000" align="CENTER"><big>Yes</big></td>
<td style="border-top: 1px solid #000000; border-bottom: 1px
solid #000000; border-left: 1px solid #000000;
border-right: 1px solid #000000" align="CENTER"><big>Yes</big></td>
<td style="border-top: 1px solid #000000; border-bottom: 1px
solid #000000; border-left: 1px solid #000000;
border-right: 1px solid #000000" align="CENTER"><big>Currently
No</big></td>
<td style="border-top: 1px solid #000000; border-bottom: 1px
solid #000000; border-left: 1px solid #000000;
border-right: 1px solid #000000" align="CENTER"><big>Yes</big></td>
</tr>
<tr>
<td style="border-top: 1px solid #000000; border-bottom: 1px
solid #000000; border-left: 1px solid #000000;
border-right: 1px solid #000000" height="17" align="LEFT"><big>unlink</big></td>
<td style="border-top: 1px solid #000000; border-bottom: 1px
solid #000000; border-left: 1px solid #000000;
border-right: 1px solid #000000" align="CENTER"><big>Yes</big></td>
<td style="border-top: 1px solid #000000; border-bottom: 1px
solid #000000; border-left: 1px solid #000000;
border-right: 1px solid #000000" align="CENTER"><big>Yes</big></td>
<td style="border-top: 1px solid #000000; border-bottom: 1px
solid #000000; border-left: 1px solid #000000;
border-right: 1px solid #000000" align="CENTER"><big>Currently
No</big></td>
<td style="border-top: 1px solid #000000; border-bottom: 1px
solid #000000; border-left: 1px solid #000000;
border-right: 1px solid #000000" align="CENTER"><big>Yes</big></td>
</tr>
<tr>
<td style="border-top: 1px solid #000000; border-bottom: 1px
solid #000000; border-left: 1px solid #000000;
border-right: 1px solid #000000" height="18" align="LEFT"><big>readdir</big></td>
<td style="border-top: 1px solid #000000; border-bottom: 1px
solid #000000; border-left: 1px solid #000000;
border-right: 1px solid #000000" align="CENTER"><big>N/A</big></td>
<td style="border-top: 1px solid #000000; border-bottom: 1px
solid #000000; border-left: 1px solid #000000;
border-right: 1px solid #000000" align="CENTER"><big>N/A</big></td>
<td style="border-top: 1px solid #000000; border-bottom: 1px
solid #000000; border-left: 1px solid #000000;
border-right: 1px solid #000000" align="CENTER"><big>?</big></td>
<td style="border-top: 1px solid #000000; border-bottom: 1px
solid #000000; border-left: 1px solid #000000;
border-right: 1px solid #000000" align="CENTER"><big>?</big></td>
</tr>
</tbody>
</table>
<title></title>
<meta name="GENERATOR" content="LibreOffice 3.6 (Linux)">
<style>
<!--
BODY,DIV,TABLE,THEAD,TBODY,TFOOT,TR,TH,TD,P { font-family:"Arial"; font-size:x-small }
-->
</style>
<title></title>
<meta name="GENERATOR" content="LibreOffice 3.6 (Linux)">
<style type="text/css">
<!--
@page { margin: 0.79in }
P { margin-bottom: 0.08in }
P.western { font-size: 12pt }
-->
</style><big><big><br>
- stat() also reports the file size. If a data split-brained
file has different sizes, should stat succeed?<br>
- Likewise if metadata split brain is due to different access
permissions, say one brick has file chmod'ed with 777 and the
other brick has it with 744, should we allow read/write if the
corresponding permission bits are *not* conflciting ? ( as of
today they aren't allowed)<br>
</big><br>
<big>Also,In the table above, Entry Split brain has 2 cases- <br>
i) where same entry has different gfids<br>
ii) each brick has different entries for the same directory
(which can cause deleted files to appear in case of
conservative merge). <br>
Should we allow readdir in either case?<br>
<br>
Thanks,<br>
Ravi<br>
</big> </big><br>
<blockquote
cite="mid:CAFboF2wZneo=nov=aBsP3257s-+43PfFDGS1rm1-_vkaaLHEPg@mail.gmail.com"
type="cite">
<div class="gmail_extra">
<div class="gmail_quote">On Wed, Nov 13, 2013 at 3:01 AM,
Ravishankar N <span dir="ltr"><<a moz-do-not-send="true"
href="mailto:ravishankar@redhat.com" target="_blank">ravishankar@redhat.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">Hi,<br>
<br>
Currenly in glusterfs, when there is a data splt-brain
(only) on a file, we disallow the following operations
from the mount-point by returning EIO to the application:<br>
- Writes to the file (truncate, dd, echo, cp etc)<br>
- Reads to the file (cat)<br>
- Reading extended attributes (getfattr) [1]<br>
<br>
However we do permit the following operations:<br>
-creating hardlinks<br>
-creating symlinks<br>
-mv<br>
-setattr<br>
-chmod<br>
-chown<br>
--touch<br>
-ls<br>
-stat<br>
<br>
While it makes sense to allow `ls` and `stat`, is it okay
to add checks in the FOPS to disallow the other
operations? Allowing creation of links and changing file
attributes only seems to complicate things before the
admin can go to the backend bricks and resolve the
splitbrain (by deleteing all but the healthy copy of the
file including hardlinks). More so if the file is renamed
before addressing the split-brain.<br>
Please share your thoughs.<br>
<br>
Thanks,<br>
Ravi<br>
<br>
[1] <a moz-do-not-send="true"
href="http://review.gluster.org/#/c/5988/"
target="_blank">http://review.gluster.org/#/c/5988/</a><br>
_______________________________________________<br>
Gluster-users mailing list<br>
<a moz-do-not-send="true"
href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a moz-do-not-send="true"
href="http://supercolony.gluster.org/mailman/listinfo/gluster-users"
target="_blank">http://supercolony.gluster.org/mailman/listinfo/gluster-users</a><br>
</blockquote>
</div>
<br>
</div>
</blockquote>
<br>
<br>
</div>
<br>
</body>
</html>