<html><head><meta http-equiv="Content-Type" content="text/html charset=iso-8859-1"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;"><br><div><div>On Sep 12, 2014, at 9:09 AM, Pranith Kumar Karampuri <<a href="mailto:pkarampu@redhat.com">pkarampu@redhat.com</a>> wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div class="moz-cite-prefix" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><br class="Apple-interchange-newline">On 09/11/2014 11:38 PM, mike wrote:<br></div><blockquote cite="mid:C841C65B-1CF3-44BC-8C97-2F182B0E053D@luminatewireless.com" type="cite" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;">Any more to this thread? I don't mean to nag, but this seems like a pretty serious issue.</blockquote><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255); float: none; display: inline !important;">Did you get a chance to try this test case when write-behind translator is disabled?</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255); float: none; display: inline !important;">gluster volume set <volname> performance.write-behind off</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"></blockquote><div><br></div>I just ran the test and it seems that disabling write-behind makes the problem go away.</div><div><br></div><div>I'll update the bug, thanks.</div><div><br></div><div><blockquote type="cite"><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255); float: none; display: inline !important;">Pranith</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><blockquote cite="mid:C841C65B-1CF3-44BC-8C97-2F182B0E053D@luminatewireless.com" type="cite" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><div><br></div><div>How can I help?</div><div><br><div><div>On Sep 7, 2014, at 9:51 AM, mike <<a moz-do-not-send="true" href="mailto:mike@luminatewireless.com">mike@luminatewireless.com</a>> wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">I don't think I have these enabled. How can I confirm that?<div><br><div><div>On Sep 7, 2014, at 12:57 AM, Anand Avati <<a moz-do-not-send="true" href="mailto:avati@gluster.org">avati@gluster.org</a>> wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div dir="ltr">The only reason O_APPEND gets stripped on the server side, is because of one of the following xlators:<div><br></div><div>- stripe</div><div>- quiesce</div><div>- crypt</div><div><br></div><div>If you have any of these, please try unloading/reconfiguring without these features and try again.</div><div><br></div><div>Thanks</div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Sat, Sep 6, 2014 at 3:31 PM, mike<span class="Apple-converted-space"> </span><span dir="ltr"><<a moz-do-not-send="true" href="mailto:mike@luminatewireless.com" target="_blank">mike@luminatewireless.com</a>></span><span class="Apple-converted-space"> </span>wrote:<br><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex;">I was able to narrow it down to smallish python script.<br><br>I've attached that to the bug.<br><br><a moz-do-not-send="true" href="https://bugzilla.redhat.com/show_bug.cgi?id=1138970" target="_blank">https://bugzilla.redhat.com/show_bug.cgi?id=1138970</a><br><div class="HOEnZb"><div class="h5"><br><br>On Sep 6, 2014, at 1:05 PM, Justin Clift <<a moz-do-not-send="true" href="mailto:justin@gluster.org">justin@gluster.org</a>> wrote:<br><br>> Thanks Mike, this is good stuff. :)<br>><br>> + Justin<br>><br>><br>> On 06/09/2014, at 8:19 PM, mike wrote:<br>>> I upgraded the client to Gluster 3.5.2, but there is no difference.<br>>><br>>> The bug is almost certainly in the Fuse client. If I remount the filesystem with NFS, the problem is no longer observable.<br>>><br>>> I spent a little time looking through the xlator/fuse-bridge to see where the offsets are coming from, but I'm really not familiar enough with the code, so it is slow going.<br>>><br>>> Unfortunately, I'm still having trouble reproducing this in a python script that could be readily attached to a bug report.<br>>><br>>> I'll take a crack at that again, but I will a file a bug anyway for completeness.<br>>><br>>> On Sep 5, 2014, at 7:10 PM, mike <<a moz-do-not-send="true" href="mailto:mike@luminatewireless.com">mike@luminatewireless.com</a>> wrote:<br>>><br>>>> I have narrowed down the source of the bug.<br>>>><br>>>> Here is an strace of glusterfsd<span class="Apple-converted-space"> </span><a moz-do-not-send="true" href="http://fpaste.org/131455/40996378/" target="_blank">http://fpaste.org/131455/40996378/</a><br>>>><br>>>> The first line represents a write that does *not* make it into the underlying file.<br>>>><br>>>> The last line is the write that stomps the earlier write.<br>>>><br>>>> As I said, the client file is opened in O_APPEND mode, but on the glusterfsd side, the file is just O_CREAT|O_WRONLY. The means the offsets to pwrite() need to be valid.<br>>>><br>>>> I correlated this to a tcpdump I took and I can see that in fact, the RPCs being sent have the wrong offset. Interestingly, glusterfs.write-is-append = 0, which I wouldn't have expected.<br>>>><br>>>> I think the bug lies in the glusterfs fuse client.<br>>>><br>>>> As to your question about Gluster 3.5.2, I may be able to do that if I am unable to find the bug in the source.<br>>>><br>>>> -Mike<br>>>><br>>>> On Sep 5, 2014, at 6:16 PM, Justin Clift <<a moz-do-not-send="true" href="mailto:justin@gluster.org">justin@gluster.org</a>> wrote:<br>>>><br>>>>> On 06/09/2014, at 12:10 AM, mike wrote:<br>>>>>> I have found that the O_APPEND flag is key to this failure - I had overlooked that flag when reading the strace and trying to cobble up a minimal reproduction.<br>>>>>><br>>>>>> I now have a small pair of python scripts that can reliably reproduce this failure.<br>>>>><br>>>>><br>>>>> As a thought, is there a reasonable way you can test this on GlusterFS 3.5.2?<br>>>>><br>>>>> There were some important bug fixes in 3.5.2 (from 3.5.1).<br>>>>><br>>>>> Note I'm not saying yours is one of them, I'm just asking if it's<br>>>>> easy to test and find out. :)<br>>>>><br>>>>> Regards and best wishes,<br>>>>><br>>>>> Justin Clift<br>>>>><br>>>>> --<br>>>>> GlusterFS -<span class="Apple-converted-space"> </span><a moz-do-not-send="true" href="http://www.gluster.org/" target="_blank">http://www.gluster.org</a><br>>>>><br>>>>> An open source, distributed file system scaling to several<br>>>>> petabytes, and handling thousands of clients.<br>>>>><br>>>>> My personal twitter:<span class="Apple-converted-space"> </span><a moz-do-not-send="true" href="http://twitter.com/realjustinclift" target="_blank">twitter.com/realjustinclift</a><br>>>>><br>>>><br>>><br>><br>> --<br>> GlusterFS -<span class="Apple-converted-space"> </span><a moz-do-not-send="true" href="http://www.gluster.org/" target="_blank">http://www.gluster.org</a><br>><br>> An open source, distributed file system scaling to several<br>> petabytes, and handling thousands of clients.<br>><br>> My personal twitter:<span class="Apple-converted-space"> </span><a moz-do-not-send="true" href="http://twitter.com/realjustinclift" target="_blank">twitter.com/realjustinclift</a><br>><br><br>_______________________________________________<br>Gluster-users mailing list<br><a moz-do-not-send="true" href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br><a moz-do-not-send="true" href="http://supercolony.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://supercolony.gluster.org/mailman/listinfo/gluster-users</a><br></div></div></blockquote></div><br></div></blockquote></div><br></div></div></blockquote></div><br></div><br><fieldset class="mimeAttachmentHeader"></fieldset><br><pre wrap="">_______________________________________________
Gluster-users mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a>
<a class="moz-txt-link-freetext" href="http://supercolony.gluster.org/mailman/listinfo/gluster-users">http://supercolony.gluster.org/mailman/listinfo/gluster-users</a></pre></blockquote><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><br class="Apple-interchange-newline"></blockquote></div><br></body></html>