<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#FFFFFF">
Interestingly disabling write-behind prevents the bug.<br>
CC Raghavendra G<br>
<br>
Pranith<br>
<div class="moz-cite-prefix">On 09/07/2014 01:27 PM, Anand Avati
wrote:<br>
</div>
<blockquote
cite="mid:CAFboF2wtW4AkypnO7nD1twwAqWQ0GJMi3e1LQV3a2FaVJvyX+A@mail.gmail.com"
type="cite">
<div dir="ltr">The only reason O_APPEND gets stripped on the
server side, is because of one of the following xlators:
<div><br>
</div>
<div>- stripe</div>
<div>- quiesce</div>
<div>- crypt</div>
<div><br>
</div>
<div>If you have any of these, please try
unloading/reconfiguring without these features and try again.</div>
<div><br>
</div>
<div>Thanks</div>
</div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On Sat, Sep 6, 2014 at 3:31 PM, mike <span
dir="ltr"><<a moz-do-not-send="true"
href="mailto:mike@luminatewireless.com" target="_blank">mike@luminatewireless.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">I was able
to narrow it down to smallish python script.<br>
<br>
I've attached that to the bug.<br>
<br>
<a moz-do-not-send="true"
href="https://bugzilla.redhat.com/show_bug.cgi?id=1138970"
target="_blank">https://bugzilla.redhat.com/show_bug.cgi?id=1138970</a><br>
<div class="HOEnZb">
<div class="h5"><br>
<br>
On Sep 6, 2014, at 1:05 PM, Justin Clift <<a
moz-do-not-send="true"
href="mailto:justin@gluster.org">justin@gluster.org</a>>
wrote:<br>
<br>
> Thanks Mike, this is good stuff. :)<br>
><br>
> + Justin<br>
><br>
><br>
> On 06/09/2014, at 8:19 PM, mike wrote:<br>
>> I upgraded the client to Gluster 3.5.2, but
there is no difference.<br>
>><br>
>> The bug is almost certainly in the Fuse client.
If I remount the filesystem with NFS, the problem is no
longer observable.<br>
>><br>
>> I spent a little time looking through the
xlator/fuse-bridge to see where the offsets are coming
from, but I'm really not familiar enough with the code,
so it is slow going.<br>
>><br>
>> Unfortunately, I'm still having trouble
reproducing this in a python script that could be
readily attached to a bug report.<br>
>><br>
>> I'll take a crack at that again, but I will a
file a bug anyway for completeness.<br>
>><br>
>> On Sep 5, 2014, at 7:10 PM, mike <<a
moz-do-not-send="true"
href="mailto:mike@luminatewireless.com">mike@luminatewireless.com</a>>
wrote:<br>
>><br>
>>> I have narrowed down the source of the bug.<br>
>>><br>
>>> Here is an strace of glusterfsd <a
moz-do-not-send="true"
href="http://fpaste.org/131455/40996378/"
target="_blank">http://fpaste.org/131455/40996378/</a><br>
>>><br>
>>> The first line represents a write that does
*not* make it into the underlying file.<br>
>>><br>
>>> The last line is the write that stomps the
earlier write.<br>
>>><br>
>>> As I said, the client file is opened in
O_APPEND mode, but on the glusterfsd side, the file is
just O_CREAT|O_WRONLY. The means the offsets to pwrite()
need to be valid.<br>
>>><br>
>>> I correlated this to a tcpdump I took and I
can see that in fact, the RPCs being sent have the wrong
offset. Interestingly, glusterfs.write-is-append = 0,
which I wouldn't have expected.<br>
>>><br>
>>> I think the bug lies in the glusterfs fuse
client.<br>
>>><br>
>>> As to your question about Gluster 3.5.2, I
may be able to do that if I am unable to find the bug in
the source.<br>
>>><br>
>>> -Mike<br>
>>><br>
>>> On Sep 5, 2014, at 6:16 PM, Justin Clift
<<a moz-do-not-send="true"
href="mailto:justin@gluster.org">justin@gluster.org</a>>
wrote:<br>
>>><br>
>>>> On 06/09/2014, at 12:10 AM, mike wrote:<br>
>>>>> I have found that the O_APPEND flag
is key to this failure - I had overlooked that flag when
reading the strace and trying to cobble up a minimal
reproduction.<br>
>>>>><br>
>>>>> I now have a small pair of python
scripts that can reliably reproduce this failure.<br>
>>>><br>
>>>><br>
>>>> As a thought, is there a reasonable way
you can test this on GlusterFS 3.5.2?<br>
>>>><br>
>>>> There were some important bug fixes in
3.5.2 (from 3.5.1).<br>
>>>><br>
>>>> Note I'm not saying yours is one of
them, I'm just asking if it's<br>
>>>> easy to test and find out. :)<br>
>>>><br>
>>>> Regards and best wishes,<br>
>>>><br>
>>>> Justin Clift<br>
>>>><br>
>>>> --<br>
>>>> GlusterFS - <a moz-do-not-send="true"
href="http://www.gluster.org" target="_blank">http://www.gluster.org</a><br>
>>>><br>
>>>> An open source, distributed file system
scaling to several<br>
>>>> petabytes, and handling thousands of
clients.<br>
>>>><br>
>>>> My personal twitter: <a
moz-do-not-send="true"
href="http://twitter.com/realjustinclift"
target="_blank">twitter.com/realjustinclift</a><br>
>>>><br>
>>><br>
>><br>
><br>
> --<br>
> GlusterFS - <a moz-do-not-send="true"
href="http://www.gluster.org" target="_blank">http://www.gluster.org</a><br>
><br>
> An open source, distributed file system scaling to
several<br>
> petabytes, and handling thousands of clients.<br>
><br>
> My personal twitter: <a moz-do-not-send="true"
href="http://twitter.com/realjustinclift"
target="_blank">twitter.com/realjustinclift</a><br>
><br>
<br>
_______________________________________________<br>
Gluster-users mailing list<br>
<a moz-do-not-send="true"
href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
<a moz-do-not-send="true"
href="http://supercolony.gluster.org/mailman/listinfo/gluster-users"
target="_blank">http://supercolony.gluster.org/mailman/listinfo/gluster-users</a><br>
</div>
</div>
</blockquote>
</div>
<br>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
Gluster-users mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a>
<a class="moz-txt-link-freetext" href="http://supercolony.gluster.org/mailman/listinfo/gluster-users">http://supercolony.gluster.org/mailman/listinfo/gluster-users</a></pre>
</blockquote>
<br>
</body>
</html>