<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<br>
<div class="moz-cite-prefix">On 09/11/2014 11:38 PM, mike wrote:<br>
</div>
<blockquote
cite="mid:C841C65B-1CF3-44BC-8C97-2F182B0E053D@luminatewireless.com"
type="cite">
<meta http-equiv="Content-Type" content="text/html;
charset=ISO-8859-1">
Any more to this thread? I don't mean to nag, but this seems like
a pretty serious issue.</blockquote>
Most probably the issue is in write-behind according to my tests.
The people who know that xlator are Avati/Raghavendra G/Niels<br>
CCed all of them<br>
<br>
Pranith<br>
<blockquote
cite="mid:C841C65B-1CF3-44BC-8C97-2F182B0E053D@luminatewireless.com"
type="cite">
<div><br>
</div>
<div>How can I help?</div>
<div><br>
<div>
<div>On Sep 7, 2014, at 9:51 AM, mike <<a
moz-do-not-send="true"
href="mailto:mike@luminatewireless.com">mike@luminatewireless.com</a>>
wrote:</div>
<br class="Apple-interchange-newline">
<blockquote type="cite">
<meta http-equiv="Content-Type" content="text/html;
charset=ISO-8859-1">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space;
-webkit-line-break: after-white-space;">I don't think I
have these enabled. How can I confirm that?
<div><br>
<div>
<div>On Sep 7, 2014, at 12:57 AM, Anand Avati <<a
moz-do-not-send="true"
href="mailto:avati@gluster.org">avati@gluster.org</a>>
wrote:</div>
<br class="Apple-interchange-newline">
<blockquote type="cite">
<div dir="ltr">The only reason O_APPEND gets
stripped on the server side, is because of one of
the following xlators:
<div><br>
</div>
<div>- stripe</div>
<div>- quiesce</div>
<div>- crypt</div>
<div><br>
</div>
<div>If you have any of these, please try
unloading/reconfiguring without these features
and try again.</div>
<div><br>
</div>
<div>Thanks</div>
</div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On Sat, Sep 6, 2014 at
3:31 PM, mike <span dir="ltr"><<a
moz-do-not-send="true"
href="mailto:mike@luminatewireless.com"
target="_blank">mike@luminatewireless.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0
0 0 .8ex;border-left:1px #ccc
solid;padding-left:1ex">I was able to narrow
it down to smallish python script.<br>
<br>
I've attached that to the bug.<br>
<br>
<a moz-do-not-send="true"
href="https://bugzilla.redhat.com/show_bug.cgi?id=1138970"
target="_blank">https://bugzilla.redhat.com/show_bug.cgi?id=1138970</a><br>
<div class="HOEnZb">
<div class="h5"><br>
<br>
On Sep 6, 2014, at 1:05 PM, Justin Clift
<<a moz-do-not-send="true"
href="mailto:justin@gluster.org">justin@gluster.org</a>>
wrote:<br>
<br>
> Thanks Mike, this is good stuff. :)<br>
><br>
> + Justin<br>
><br>
><br>
> On 06/09/2014, at 8:19 PM, mike
wrote:<br>
>> I upgraded the client to Gluster
3.5.2, but there is no difference.<br>
>><br>
>> The bug is almost certainly in
the Fuse client. If I remount the
filesystem with NFS, the problem is no
longer observable.<br>
>><br>
>> I spent a little time looking
through the xlator/fuse-bridge to see
where the offsets are coming from, but I'm
really not familiar enough with the code,
so it is slow going.<br>
>><br>
>> Unfortunately, I'm still having
trouble reproducing this in a python
script that could be readily attached to a
bug report.<br>
>><br>
>> I'll take a crack at that again,
but I will a file a bug anyway for
completeness.<br>
>><br>
>> On Sep 5, 2014, at 7:10 PM, mike
<<a moz-do-not-send="true"
href="mailto:mike@luminatewireless.com">mike@luminatewireless.com</a>>
wrote:<br>
>><br>
>>> I have narrowed down the
source of the bug.<br>
>>><br>
>>> Here is an strace of
glusterfsd <a moz-do-not-send="true"
href="http://fpaste.org/131455/40996378/"
target="_blank">http://fpaste.org/131455/40996378/</a><br>
>>><br>
>>> The first line represents a
write that does *not* make it into the
underlying file.<br>
>>><br>
>>> The last line is the write
that stomps the earlier write.<br>
>>><br>
>>> As I said, the client file is
opened in O_APPEND mode, but on the
glusterfsd side, the file is just
O_CREAT|O_WRONLY. The means the offsets to
pwrite() need to be valid.<br>
>>><br>
>>> I correlated this to a
tcpdump I took and I can see that in fact,
the RPCs being sent have the wrong
offset. Interestingly,
glusterfs.write-is-append = 0, which I
wouldn't have expected.<br>
>>><br>
>>> I think the bug lies in the
glusterfs fuse client.<br>
>>><br>
>>> As to your question about
Gluster 3.5.2, I may be able to do that if
I am unable to find the bug in the source.<br>
>>><br>
>>> -Mike<br>
>>><br>
>>> On Sep 5, 2014, at 6:16 PM,
Justin Clift <<a moz-do-not-send="true"
href="mailto:justin@gluster.org">justin@gluster.org</a>>
wrote:<br>
>>><br>
>>>> On 06/09/2014, at 12:10
AM, mike wrote:<br>
>>>>> I have found that the
O_APPEND flag is key to this failure - I
had overlooked that flag when reading the
strace and trying to cobble up a minimal
reproduction.<br>
>>>>><br>
>>>>> I now have a small
pair of python scripts that can reliably
reproduce this failure.<br>
>>>><br>
>>>><br>
>>>> As a thought, is there a
reasonable way you can test this on
GlusterFS 3.5.2?<br>
>>>><br>
>>>> There were some important
bug fixes in 3.5.2 (from 3.5.1).<br>
>>>><br>
>>>> Note I'm not saying yours
is one of them, I'm just asking if it's<br>
>>>> easy to test and find
out. :)<br>
>>>><br>
>>>> Regards and best wishes,<br>
>>>><br>
>>>> Justin Clift<br>
>>>><br>
>>>> --<br>
>>>> GlusterFS - <a
moz-do-not-send="true"
href="http://www.gluster.org/"
target="_blank">http://www.gluster.org</a><br>
>>>><br>
>>>> An open source,
distributed file system scaling to several<br>
>>>> petabytes, and handling
thousands of clients.<br>
>>>><br>
>>>> My personal twitter: <a
moz-do-not-send="true"
href="http://twitter.com/realjustinclift"
target="_blank">twitter.com/realjustinclift</a><br>
>>>><br>
>>><br>
>><br>
><br>
> --<br>
> GlusterFS - <a
moz-do-not-send="true"
href="http://www.gluster.org/"
target="_blank">http://www.gluster.org</a><br>
><br>
> An open source, distributed file
system scaling to several<br>
> petabytes, and handling thousands of
clients.<br>
><br>
> My personal twitter: <a
moz-do-not-send="true"
href="http://twitter.com/realjustinclift"
target="_blank">twitter.com/realjustinclift</a><br>
><br>
<br>
_______________________________________________<br>
Gluster-users mailing list<br>
<a moz-do-not-send="true"
href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
<a moz-do-not-send="true"
href="http://supercolony.gluster.org/mailman/listinfo/gluster-users"
target="_blank">http://supercolony.gluster.org/mailman/listinfo/gluster-users</a><br>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
</div>
<br>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
Gluster-users mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a>
<a class="moz-txt-link-freetext" href="http://supercolony.gluster.org/mailman/listinfo/gluster-users">http://supercolony.gluster.org/mailman/listinfo/gluster-users</a></pre>
</blockquote>
<br>
</body>
</html>