<font size=2 face="sans-serif"><br>John,</font><br><br><font size=2 face="sans-serif">We've experienced the same problem too
it appears. I've ran an strace on the file when it returns "structure
needs cleaning" after trying to do an 'ls' . I get wind of this
from my developers when they make modifcations to their content and want
to see it served up from apache.</font><br><br><font size=2 face="sans-serif">It will stay like that for infinite
it seems like. The band-aid I found was if I did a 'ls -l' of the
directory the file lives in, it will clear up and then I am able to do
an 'ls -l' on the file that had the "structure needs cleaning"
message. My setup is a 2x2 distri/rep bricks.....</font><br><br><font size=2 face="sans-serif">From the #gluster IRC room, it was suggested
that I do an xfs_repair on the bricks even though it doesn't log any issues
with the underlying xfs filesystem. I've ran that 2 days ago off
hours, but it appears this morning the issue has come back to haunt me.</font><br><font size=2 face="sans-serif">Unfortunately my brick logs have rolled,
and I am not able to show you because it logs so much logrotate rolls it
off.</font><br><br><br><font size=2 face="sans-serif">fuse client log file</font><br><a href=http://fpaste.org/61559/69524561/><font size=2 face="sans-serif">http://fpaste.org/61559/69524561/</font></a><br><br><font size=2 face="sans-serif">dirty strace of a file</font><br><a href=http://fpaste.org/61556/95199613/><font size=2 face="sans-serif">http://fpaste.org/61556/95199613/</font></a><br><br><font size=2 face="sans-serif">listing of directory where dirty file
was issued</font><br><a href=http://fpaste.org/61557/86952173/><font size=2 face="sans-serif">http://fpaste.org/61557/86952173/</font></a><br><br><font size=2 face="sans-serif">cleaned and stat-able file</font><br><a href=http://fpaste.org/61558/86952224/><font size=2 face="sans-serif">http://fpaste.org/61558/86952224/</font></a><font size=2 face="sans-serif"><br></font><br><br><br><br><font size=1 color=#5f5f5f face="sans-serif">From:
</font><font size=1 face="sans-serif">gluster-users-request@gluster.org</font><br><font size=1 color=#5f5f5f face="sans-serif">To:
</font><font size=1 face="sans-serif">gluster-users@gluster.org</font><br><font size=1 color=#5f5f5f face="sans-serif">Date:
</font><font size=1 face="sans-serif">12/13/2013 05:58 AM</font><br><font size=1 color=#5f5f5f face="sans-serif">Subject:
</font><font size=1 face="sans-serif">Gluster-users
Digest, Vol 68, Issue 14</font><br><font size=1 color=#5f5f5f face="sans-serif">Sent by:
</font><font size=1 face="sans-serif">gluster-users-bounces@gluster.org</font><br><hr noshade><br><br><br><tt><font size=2>Send Gluster-users mailing list submissions to<br>
gluster-users@gluster.org<br><br>To subscribe or unsubscribe via the World Wide Web, visit<br>
</font></tt><a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users"><tt><font size=2>http://supercolony.gluster.org/mailman/listinfo/gluster-users</font></tt></a><tt><font size=2><br>or, via email, send a message with subject or body 'help' to<br>
gluster-users-request@gluster.org<br><br>You can reach the person managing the list at<br>
gluster-users-owner@gluster.org<br><br>When replying, please edit your Subject line so it is more specific<br>than "Re: Contents of Gluster-users digest..."<br><br><br>Today's Topics:<br><br> 1. Re: Structure needs cleaning on some files (Johan Huysmans)<br> 2. Re: Structure needs cleaning on some files (Johan Huysmans)<br> 3. Re: Gluster Community Weekly Meeting (Vijay Bellur)<br> 4. Re: Gluster Community Weekly Meeting (James)<br> 5. Re: Gluster Community Weekly Meeting (Vijay Bellur)<br> 6. Re: Structure needs cleaning on some files (Maik Kulbe)<br> 7. Re: Structure needs cleaning on some files (Anand Avati)<br> 8. Re: Structure needs cleaning on some files (Anand Avati)<br> 9. Gerrit doesn't use HTTPS (James)<br> 10. gluster fails under heavy array job load load (harry mangalam)<br> 11. qemu remote insecure connections (Joe Topjian)<br> 12. Documentation hackathon for 3.5 (Vijay Bellur)<br> 13. Re: gluster fails under heavy array job load load (Anand Avati)<br> 14. Re: Gluster Community Weekly Meeting (Niels de Vos)<br><br><br>----------------------------------------------------------------------<br><br>Message: 1<br>Date: Thu, 12 Dec 2013 14:40:37 +0100<br>From: Johan Huysmans <johan.huysmans@inuits.be><br>To: "gluster-users@gluster.org" <gluster-users@gluster.org><br>Subject: Re: [Gluster-users] Structure needs cleaning on some files<br>Message-ID: <52A9BCD5.9020905@inuits.be><br>Content-Type: text/plain; charset="iso-8859-1"; Format="flowed"<br><br>I created a bug for this issue:<br><br></font></tt><a href="https://bugzilla.redhat.com/show_bug.cgi?id=1041109"><tt><font size=2>https://bugzilla.redhat.com/show_bug.cgi?id=1041109</font></tt></a><tt><font size=2><br><br>gr.<br>Johan<br><br>On 10-12-13 12:52, Johan Huysmans wrote:<br>> Hi All,<br>><br>> It seems I can easily reproduce the problem.<br>><br>> * on node 1 create a file (touch , cat , ...).<br>> * on node 2 take md5sum of direct file (md5sum /path/to/file)<br>> * on node 1 move file to other name (mv file file1)<br>> * on node 2 take md5sum of direct file (md5sum /path/to/file), this
is <br>> still working although the file is not really there<br>> * on node 1 change file content<br>> * on node 2 take md5sum of direct file (md5sum /path/to/file), this
is <br>> still working and has a changed md5sum<br>><br>> This is really strange behaviour.<br>> Is this normal, can this be altered with a a setting?<br>><br>> Thanks for any info,<br>> gr.<br>> Johan<br>><br>> On 10-12-13 10:02, Johan Huysmans wrote:<br>>> I could reproduce this problem with while my mount point is running
<br>>> in debug mode.<br>>> logfile is attached.<br>>><br>>> gr.<br>>> Johan Huysmans<br>>><br>>> On 10-12-13 09:30, Johan Huysmans wrote:<br>>>> Hi All,<br>>>><br>>>> When reading some files we get this error:<br>>>> md5sum: /path/to/file.xml: Structure needs cleaning<br>>>><br>>>> in /var/log/glusterfs/mnt-sharedfs.log we see these errors:<br>>>> [2013-12-10 08:07:32.256910] W <br>>>> [client-rpc-fops.c:526:client3_3_stat_cbk] 1-testvolume-client-0:
<br>>>> remote operation failed: No such file or directory<br>>>> [2013-12-10 08:07:32.257436] W <br>>>> [client-rpc-fops.c:526:client3_3_stat_cbk] 1-testvolume-client-1:
<br>>>> remote operation failed: No such file or directory<br>>>> [2013-12-10 08:07:32.259356] W [fuse-bridge.c:705:fuse_attr_cbk]
<br>>>> 0-glusterfs-fuse: 8230: STAT() /path/to/file.xml => -1
(Structure <br>>>> needs cleaning)<br>>>><br>>>> We are using gluster 3.4.1-3 on CentOS6.<br>>>> Our servers are 64-bit, our clients 32-bit (we are already
using <br>>>> --enable-ino32 on the mountpoint)<br>>>><br>>>> This is my gluster configuration:<br>>>> Volume Name: testvolume<br>>>> Type: Replicate<br>>>> Volume ID: ca9c2f87-5d5b-4439-ac32-b7c138916df7<br>>>> Status: Started<br>>>> Number of Bricks: 1 x 2 = 2<br>>>> Transport-type: tcp<br>>>> Bricks:<br>>>> Brick1: SRV-1:/gluster/brick1<br>>>> Brick2: SRV-2:/gluster/brick2<br>>>> Options Reconfigured:<br>>>> performance.force-readdirp: on<br>>>> performance.stat-prefetch: off<br>>>> network.ping-timeout: 5<br>>>><br>>>> And this is how the applications work:<br>>>> We have 2 client nodes who both have a fuse.glusterfs mountpoint.<br>>>> On 1 client node we have a application which writes files.<br>>>> On the other client node we have a application which reads
these files.<br>>>> On the node where the files are written we don't see any problem,
<br>>>> and can read that file without problems.<br>>>> On the other node we have problems (error messages above)
reading <br>>>> that file.<br>>>> The problem occurs when we perform a md5sum on the exact file,
when <br>>>> perform a md5sum on all files in that directory there is no
problem.<br>>>><br>>>><br>>>> How can we solve this problem as this is annoying.<br>>>> The problem occurs after some time (can be days), an umount
and <br>>>> mount of the mountpoint solves it for some days.<br>>>> Once it occurs (and we don't remount) it occurs every time.<br>>>><br>>>><br>>>> I hope someone can help me with this problems.<br>>>><br>>>> Thanks,<br>>>> Johan Huysmans<br>>>> _______________________________________________<br>>>> Gluster-users mailing list<br>>>> Gluster-users@gluster.org<br>>>> </font></tt><a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users"><tt><font size=2>http://supercolony.gluster.org/mailman/listinfo/gluster-users</font></tt></a><tt><font size=2><br>>><br>>><br>>><br>>> _______________________________________________<br>>> Gluster-users mailing list<br>>> Gluster-users@gluster.org<br>>> </font></tt><a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users"><tt><font size=2>http://supercolony.gluster.org/mailman/listinfo/gluster-users</font></tt></a><tt><font size=2><br>><br>><br>><br>> _______________________________________________<br>> Gluster-users mailing list<br>> Gluster-users@gluster.org<br>> </font></tt><a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users"><tt><font size=2>http://supercolony.gluster.org/mailman/listinfo/gluster-users</font></tt></a><tt><font size=2><br><br>-------------- next part --------------<br>An HTML attachment was scrubbed...<br>URL: <</font></tt><a href="http://supercolony.gluster.org/pipermail/gluster-users/attachments/20131212/4152cb13/attachment-0001.html"><tt><font size=2>http://supercolony.gluster.org/pipermail/gluster-users/attachments/20131212/4152cb13/attachment-0001.html</font></tt></a><tt><font size=2>><br><br>------------------------------<br><br>Message: 2<br>Date: Thu, 12 Dec 2013 14:51:35 +0100<br>From: Johan Huysmans <johan.huysmans@inuits.be><br>To: "gluster-users@gluster.org" <gluster-users@gluster.org><br>Subject: Re: [Gluster-users] Structure needs cleaning on some files<br>Message-ID: <52A9BF67.4070403@inuits.be><br>Content-Type: text/plain; charset="iso-8859-1"; Format="flowed"<br><br>I created a bug for this issue:<br><br></font></tt><a href="https://bugzilla.redhat.com/show_bug.cgi?id=1041109"><tt><font size=2>https://bugzilla.redhat.com/show_bug.cgi?id=1041109</font></tt></a><tt><font size=2><br><br>gr.<br>Johan<br><br>On 10-12-13 12:52, Johan Huysmans wrote:<br>> Hi All,<br>><br>> It seems I can easily reproduce the problem.<br>><br>> * on node 1 create a file (touch , cat , ...).<br>> * on node 2 take md5sum of direct file (md5sum /path/to/file)<br>> * on node 1 move file to other name (mv file file1)<br>> * on node 2 take md5sum of direct file (md5sum /path/to/file), this
is <br>> still working although the file is not really there<br>> * on node 1 change file content<br>> * on node 2 take md5sum of direct file (md5sum /path/to/file), this
is <br>> still working and has a changed md5sum<br>><br>> This is really strange behaviour.<br>> Is this normal, can this be altered with a a setting?<br>><br>> Thanks for any info,<br>> gr.<br>> Johan<br>><br>> On 10-12-13 10:02, Johan Huysmans wrote:<br>>> I could reproduce this problem with while my mount point is running
<br>>> in debug mode.<br>>> logfile is attached.<br>>><br>>> gr.<br>>> Johan Huysmans<br>>><br>>> On 10-12-13 09:30, Johan Huysmans wrote:<br>>>> Hi All,<br>>>><br>>>> When reading some files we get this error:<br>>>> md5sum: /path/to/file.xml: Structure needs cleaning<br>>>><br>>>> in /var/log/glusterfs/mnt-sharedfs.log we see these errors:<br>>>> [2013-12-10 08:07:32.256910] W <br>>>> [client-rpc-fops.c:526:client3_3_stat_cbk] 1-testvolume-client-0:
<br>>>> remote operation failed: No such file or directory<br>>>> [2013-12-10 08:07:32.257436] W <br>>>> [client-rpc-fops.c:526:client3_3_stat_cbk] 1-testvolume-client-1:
<br>>>> remote operation failed: No such file or directory<br>>>> [2013-12-10 08:07:32.259356] W [fuse-bridge.c:705:fuse_attr_cbk]
<br>>>> 0-glusterfs-fuse: 8230: STAT() /path/to/file.xml => -1
(Structure <br>>>> needs cleaning)<br>>>><br>>>> We are using gluster 3.4.1-3 on CentOS6.<br>>>> Our servers are 64-bit, our clients 32-bit (we are already
using <br>>>> --enable-ino32 on the mountpoint)<br>>>><br>>>> This is my gluster configuration:<br>>>> Volume Name: testvolume<br>>>> Type: Replicate<br>>>> Volume ID: ca9c2f87-5d5b-4439-ac32-b7c138916df7<br>>>> Status: Started<br>>>> Number of Bricks: 1 x 2 = 2<br>>>> Transport-type: tcp<br>>>> Bricks:<br>>>> Brick1: SRV-1:/gluster/brick1<br>>>> Brick2: SRV-2:/gluster/brick2<br>>>> Options Reconfigured:<br>>>> performance.force-readdirp: on<br>>>> performance.stat-prefetch: off<br>>>> network.ping-timeout: 5<br>>>><br>>>> And this is how the applications work:<br>>>> We have 2 client nodes who both have a fuse.glusterfs mountpoint.<br>>>> On 1 client node we have a application which writes files.<br>>>> On the other client node we have a application which reads
these files.<br>>>> On the node where the files are written we don't see any problem,
<br>>>> and can read that file without problems.<br>>>> On the other node we have problems (error messages above)
reading <br>>>> that file.<br>>>> The problem occurs when we perform a md5sum on the exact file,
when <br>>>> perform a md5sum on all files in that directory there is no
problem.<br>>>><br>>>><br>>>> How can we solve this problem as this is annoying.<br>>>> The problem occurs after some time (can be days), an umount
and <br>>>> mount of the mountpoint solves it for some days.<br>>>> Once it occurs (and we don't remount) it occurs every time.<br>>>><br>>>><br>>>> I hope someone can help me with this problems.<br>>>><br>>>> Thanks,<br>>>> Johan Huysmans<br>>>> _______________________________________________<br>>>> Gluster-users mailing list<br>>>> Gluster-users@gluster.org<br>>>> </font></tt><a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users"><tt><font size=2>http://supercolony.gluster.org/mailman/listinfo/gluster-users</font></tt></a><tt><font size=2><br>>><br>>><br>>><br>>> _______________________________________________<br>>> Gluster-users mailing list<br>>> Gluster-users@gluster.org<br>>> </font></tt><a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users"><tt><font size=2>http://supercolony.gluster.org/mailman/listinfo/gluster-users</font></tt></a><tt><font size=2><br>><br>><br>><br>> _______________________________________________<br>> Gluster-users mailing list<br>> Gluster-users@gluster.org<br>> </font></tt><a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users"><tt><font size=2>http://supercolony.gluster.org/mailman/listinfo/gluster-users</font></tt></a><tt><font size=2><br><br>-------------- next part --------------<br>An HTML attachment was scrubbed...<br>URL: <</font></tt><a href="http://supercolony.gluster.org/pipermail/gluster-users/attachments/20131212/97a7843c/attachment-0001.html"><tt><font size=2>http://supercolony.gluster.org/pipermail/gluster-users/attachments/20131212/97a7843c/attachment-0001.html</font></tt></a><tt><font size=2>><br><br>------------------------------<br><br>Message: 3<br>Date: Fri, 13 Dec 2013 00:13:30 +0530<br>From: Vijay Bellur <vbellur@redhat.com><br>To: James <purpleidea@gmail.com><br>Cc: "gluster-users@gluster.org" <gluster-users@gluster.org>,
Gluster<br>
Devel <gluster-devel@nongnu.org>, Niels de Vos <ndevos@redhat.com><br>Subject: Re: [Gluster-users] Gluster Community Weekly Meeting<br>Message-ID: <52AA03D2.9030708@redhat.com><br>Content-Type: text/plain; charset=UTF-8; format=flowed<br><br>On 12/12/2013 10:42 AM, James wrote:<br>> RE: meeting, sorry I couldn't make it, but I have some comments:<br><br>No problem. It would be really good to have everybody in the meeting, <br>but if you cannot comments are definitely welcome :).<br><br>><br>> 1) About the pre-packaged VM comment's. I've gotten Vagrant working
on<br>> Fedora. I'm using this to rapidly spin up and test GlusterFS.<br>> </font></tt><a href="https://ttboj.wordpress.com/2013/12/09/vagrant-on-fedora-with-libvirt/"><tt><font size=2>https://ttboj.wordpress.com/2013/12/09/vagrant-on-fedora-with-libvirt/</font></tt></a><tt><font size=2><br>> In the coming week or so, I'll be publishing the Vagrant file for
my<br>> GlusterFS setup, but if you really want it now I can send you an early<br>> version. This obviously integrates with Puppet-Gluster, but whether<br>> you use that or not is optional. I think this is the best way to test<br>> GlusterFS. If someone gives me hosting, I could publish "pre-built"<br>> images very easily. Let me know what you think.<br><br>Niels - do you have any thoughts here?<br><br>><br>> 2) I never heard back from any action items from 2 weeks ago. I think<br>> someone was going to connect me with a way to get access to some VM's<br>> for testing stuff !<br><br>I see that there is an ongoing offline thread now. I think that should
<br>result in you getting those VMs.<br><br>><br>> 3) Hagarth: RE: typos, I have at least one spell check patch
against<br>> 3.4.1 I sent it to list before, but someone told me to enroll in the<br>> jenkins thing, which wasn't worth it for a small patch. Let me know
if<br>> you want it.<br><br>There are more typos now. I ran a cursory check with misspell-check [1]
<br>and found quite a few. Having that cleaned up on master and release-3.5
<br>would be great. Since the number is more, I am sure the patch would be
<br>non-trivial and having that routed through gerrit would be great. If you
<br>need a how to on getting to gerrit, it is available at [2].<br><br>><br>> 4a) Someone mentioned documentation. Please feel free to merge in<br>> </font></tt><a href="https://github.com/purpleidea/puppet-gluster/blob/master/DOCUMENTATION.md"><tt><font size=2>https://github.com/purpleidea/puppet-gluster/blob/master/DOCUMENTATION.md</font></tt></a><tt><font size=2><br>> (markdown format). I have gone to great lengths to format this so
that<br>> it displays properly in github markdown, and standard (pandoc)<br>> markdown. This way it works on github, and can also be rendered to
a<br>> pdf easily. Example:<br>> </font></tt><a href="https://github.com/purpleidea/puppet-gluster/raw/master/puppet-gluster-documentation.pdf"><tt><font size=2>https://github.com/purpleidea/puppet-gluster/raw/master/puppet-gluster-documentation.pdf</font></tt></a><tt><font size=2><br>> You can use the file as a template!<br><br>Again having this in gerrit would be useful for merging the puppet <br>documentation.<br><br>><br>> 4b) I think the documentation should be kept in the same repo as<br>> GlusterFS. This way, when you submit a feature branch, it can also<br>> come with documentation. Lots of people work this way. It helps you<br>> get minimal docs there, and/or at least some example code or a few<br>> sentences. Also, looking at the docs, you can see what commits came<br>> with this<br><br>I am with you on this one. After we are done with the planned <br>documentation hackathon, let us open a new thread on this to get more <br>opinions.<br><br>-Vijay<br><br>[1] </font></tt><a href="https://github.com/lyda/misspell-check"><tt><font size=2>https://github.com/lyda/misspell-check</font></tt></a><tt><font size=2><br><br>[2] <br></font></tt><a href=http://www.gluster.org/community/documentation/index.php/Development_Work_Flow><tt><font size=2>http://www.gluster.org/community/documentation/index.php/Development_Work_Flow</font></tt></a><tt><font size=2><br><br>><br>> Thanks!<br>><br>> James<br>><br>><br><br><br><br>------------------------------<br><br>Message: 4<br>Date: Thu, 12 Dec 2013 13:48:31 -0500<br>From: James <purpleidea@gmail.com><br>To: Vijay Bellur <vbellur@redhat.com><br>Cc: "gluster-users@gluster.org" <gluster-users@gluster.org>,
Gluster<br>
Devel <gluster-devel@nongnu.org><br>Subject: Re: [Gluster-users] Gluster Community Weekly Meeting<br>Message-ID:<br>
<CADCaTgqJrJ6uTyGiti+q0SpXxMjE+m-vr6yeohgxP3xEdYyt5w@mail.gmail.com><br>Content-Type: text/plain; charset=UTF-8<br><br>On Thu, Dec 12, 2013 at 1:43 PM, Vijay Bellur <vbellur@redhat.com>
wrote:<br>>> 4a) Someone mentioned documentation. Please feel free to merge
in<br>>> </font></tt><a href="https://github.com/purpleidea/puppet-gluster/blob/master/DOCUMENTATION.md"><tt><font size=2>https://github.com/purpleidea/puppet-gluster/blob/master/DOCUMENTATION.md</font></tt></a><tt><font size=2><br>>> (markdown format). I have gone to great lengths to format this
so that<br>>> it displays properly in github markdown, and standard (pandoc)<br>>> markdown. This way it works on github, and can also be rendered
to a<br>>> pdf easily. Example:<br>>><br>>> </font></tt><a href="https://github.com/purpleidea/puppet-gluster/raw/master/puppet-gluster-documentation.pdf"><tt><font size=2>https://github.com/purpleidea/puppet-gluster/raw/master/puppet-gluster-documentation.pdf</font></tt></a><tt><font size=2><br>>> You can use the file as a template!<br>><br>><br>> Again having this in gerrit would be useful for merging the puppet<br>> documentation.<br><br><br>Okay, I'll try to look into Gerrit and maybe submit a fake patch for testing.<br>When and where (in the tree) would be a good time to submit a doc<br>patch? It's probably best to wait until after your docs hackathon,<br>right?<br><br><br>------------------------------<br><br>Message: 5<br>Date: Fri, 13 Dec 2013 00:30:09 +0530<br>From: Vijay Bellur <vbellur@redhat.com><br>To: James <purpleidea@gmail.com><br>Cc: "gluster-users@gluster.org" <gluster-users@gluster.org>,
Gluster<br>
Devel <gluster-devel@nongnu.org><br>Subject: Re: [Gluster-users] Gluster Community Weekly Meeting<br>Message-ID: <52AA07B9.9030807@redhat.com><br>Content-Type: text/plain; charset=UTF-8; format=flowed<br><br>On 12/13/2013 12:18 AM, James wrote:<br>> On Thu, Dec 12, 2013 at 1:43 PM, Vijay Bellur <vbellur@redhat.com>
wrote:<br>>>> 4a) Someone mentioned documentation. Please feel free to merge
in<br>>>> </font></tt><a href="https://github.com/purpleidea/puppet-gluster/blob/master/DOCUMENTATION.md"><tt><font size=2>https://github.com/purpleidea/puppet-gluster/blob/master/DOCUMENTATION.md</font></tt></a><tt><font size=2><br>>>> (markdown format). I have gone to great lengths to format
this so that<br>>>> it displays properly in github markdown, and standard (pandoc)<br>>>> markdown. This way it works on github, and can also be rendered
to a<br>>>> pdf easily. Example:<br>>>><br>>>> </font></tt><a href="https://github.com/purpleidea/puppet-gluster/raw/master/puppet-gluster-documentation.pdf"><tt><font size=2>https://github.com/purpleidea/puppet-gluster/raw/master/puppet-gluster-documentation.pdf</font></tt></a><tt><font size=2><br>>>> You can use the file as a template!<br>>><br>>><br>>> Again having this in gerrit would be useful for merging the puppet<br>>> documentation.<br>><br>><br>> Okay, I'll try to look into Gerrit and maybe submit a fake patch for
testing.<br>> When and where (in the tree) would be a good time to submit a doc<br>> patch? It's probably best to wait until after your docs hackathon,<br>> right?<br>><br><br>Just added a page in preparation for the documentation hackathon:<br><br></font></tt><a href=http://www.gluster.org/community/documentation/index.php/Submitting_Documentation_Patches><tt><font size=2>http://www.gluster.org/community/documentation/index.php/Submitting_Documentation_Patches</font></tt></a><tt><font size=2><br><br>I think the puppet guide can be under a new hierarchy located at <br>doc/deploy-guide/markdown/en-US/. You can certainly submit the puppet <br>doc patch as part of the hackathon.<br><br>-Vijay<br><br><br><br>------------------------------<br><br>Message: 6<br>Date: Thu, 12 Dec 2013 21:46:12 +0100<br>From: "Maik Kulbe" <info@linux-web-development.de><br>To: "Johan Huysmans" <johan.huysmans@inuits.be>,<br>
"gluster-users@gluster.org" <gluster-users@gluster.org><br>Subject: Re: [Gluster-users] Structure needs cleaning on some files<br>Message-ID:<br>
<026e5e3d74c6956119dcde7b393990c4@linux-web-development.de><br>Content-Type: text/plain; charset="utf-8"; Format="flowed"<br><br>How do you mount your Client? FUSE? I had similar problems when playing
around with the timeout options for the FUSE mount. If they are too high
they cache the metadata for too long. When you move the file the inode
should stay the same and on the second node the path should stay in cache
for a while so it still knows the inode for that moved files old path thus
can act on the file without knowing it's path.<br><br>The problems kick in when you delete a file and recreate it - the cache
tries to access the old inode, which was deleted, thus throwing errors.
If I recall correctly the "structure needs cleaning" is one of
two error messages I got, depending on which of the timeout mount options
was set to a higher value.<br><br>-----Original Mail-----<br>From: Johan Huysmans [johan.huysmans@inuits.be]<br>Sent: 12.12.13 - 14:51:35<br>To: gluster-users@gluster.org [gluster-users@gluster.org]<br><br>Subject: Re: [Gluster-users] Structure needs cleaning on some files<br><br>> I created a bug for this issue:<br>><br>> </font></tt><a href="https://bugzilla.redhat.com/show_bug.cgi?id=1041109"><tt><font size=2>https://bugzilla.redhat.com/show_bug.cgi?id=1041109</font></tt></a><tt><font size=2><br>><br>> gr.<br>> Johan<br>><br>> On 10-12-13 12:52, Johan Huysmans wrote:<br>><br>> Hi All,<br>><br>> It seems I can easily reproduce the problem.<br>><br>> * on node 1 create a file (touch , cat , ...).<br>> * on node 2 take md5sum of direct file (md5sum /path/to/file)<br>> * on node 1 move file to other name (mv file file1)<br>> * on node 2 take md5sum of direct file (md5sum /path/to/file), this
is<br>> still working although the file is not really there<br>> * on node 1 change file content<br>> * on node 2 take md5sum of direct file (md5sum /path/to/file), this
is<br>> still working and has a changed md5sum<br>><br>> This is really strange behaviour.<br>> Is this normal, can this be altered with a a setting?<br>><br>> Thanks for any info,<br>> gr.<br>> Johan<br>><br>> On 10-12-13 10:02, Johan Huysmans wrote:<br>><br>> I could reproduce this problem with while my mount point is running
in<br>> debug mode.<br>> logfile is attached.<br>><br>> gr.<br>> Johan Huysmans<br>><br>> On 10-12-13 09:30, Johan Huysmans wrote:<br>><br>> Hi All,<br>><br>> When reading some files we get this error:<br>> md5sum: /path/to/file.xml: Structure needs cleaning<br>><br>> in /var/log/glusterfs/mnt-sharedfs.log we see these errors:<br>> [2013-12-10 08:07:32.256910] W<br>> [client-rpc-fops.c:526:client3_3_stat_cbk] 1-testvolume-client-0:<br>> remote operation failed: No such file or directory<br>> [2013-12-10 08:07:32.257436] W<br>> [client-rpc-fops.c:526:client3_3_stat_cbk] 1-testvolume-client-1:<br>> remote operation failed: No such file or directory<br>> [2013-12-10 08:07:32.259356] W [fuse-bridge.c:705:fuse_attr_cbk]<br>> 0-glusterfs-fuse: 8230: STAT() /path/to/file.xml => -1 (Structure<br>> needs cleaning)<br>><br>> We are using gluster 3.4.1-3 on CentOS6.<br>> Our servers are 64-bit, our clients 32-bit (we are already using<br>> --enable-ino32 on the mountpoint)<br>><br>> This is my gluster configuration:<br>> Volume Name: testvolume<br>> Type: Replicate<br>> Volume ID: ca9c2f87-5d5b-4439-ac32-b7c138916df7<br>> Status: Started<br>> Number of Bricks: 1 x 2 = 2<br>> Transport-type: tcp<br>> Bricks:<br>> Brick1: SRV-1:/gluster/brick1<br>> Brick2: SRV-2:/gluster/brick2<br>> Options Reconfigured:<br>> performance.force-readdirp: on<br>> performance.stat-prefetch: off<br>> network.ping-timeout: 5<br>><br>> And this is how the applications work:<br>> We have 2 client nodes who both have a fuse.glusterfs mountpoint.<br>> On 1 client node we have a application which writes files.<br>> On the other client node we have a application which reads these<br>> files.<br>> On the node where the files are written we don't see any problem,<br>> and can read that file without problems.<br>> On the other node we have problems (error messages above) reading<br>> that file.<br>> The problem occurs when we perform a md5sum on the exact file, when<br>> perform a md5sum on all files in that directory there is no problem.<br>><br>> How can we solve this problem as this is annoying.<br>> The problem occurs after some time (can be days), an umount and<br>> mount of the mountpoint solves it for some days.<br>> Once it occurs (and we don't remount) it occurs every time.<br>><br>> I hope someone can help me with this problems.<br>><br>> Thanks,<br>> Johan Huysmans<br>> _______________________________________________<br>> Gluster-users mailing list<br>> Gluster-users@gluster.org<br>> </font></tt><a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users"><tt><font size=2>http://supercolony.gluster.org/mailman/listinfo/gluster-users</font></tt></a><tt><font size=2><br>><br>> _______________________________________________<br>> Gluster-users mailing list<br>> Gluster-users@gluster.org<br>> </font></tt><a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users"><tt><font size=2>http://supercolony.gluster.org/mailman/listinfo/gluster-users</font></tt></a><tt><font size=2><br>><br>> _______________________________________________<br>> Gluster-users mailing list<br>> Gluster-users@gluster.org<br>> </font></tt><a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users"><tt><font size=2>http://supercolony.gluster.org/mailman/listinfo/gluster-users</font></tt></a><tt><font size=2><br>-------------- next part --------------<br>A non-text attachment was scrubbed...<br>Name: smime.p7s<br>Type: application/x-pkcs7-signature<br>Size: 2332 bytes<br>Desc: not available</font></tt><br><tt><font size=2>URL: <</font></tt><a href="http://supercolony.gluster.org/pipermail/gluster-users/attachments/20131212/99d59ead/attachment-0001.bin"><tt><font size=2>http://supercolony.gluster.org/pipermail/gluster-users/attachments/20131212/99d59ead/attachment-0001.bin</font></tt></a><tt><font size=2>><br><br>------------------------------<br><br>Message: 7<br>Date: Thu, 12 Dec 2013 13:26:56 -0800<br>From: Anand Avati <avati@gluster.org><br>To: Maik Kulbe <info@linux-web-development.de><br>Cc: "gluster-users@gluster.org" <gluster-users@gluster.org><br>Subject: Re: [Gluster-users] Structure needs cleaning on some files<br>Message-ID:<br>
<CAFboF2x1CraXbYSokGt1jhOhBCny+9LRPzASt-Wk8LYqx6nFdw@mail.gmail.com><br>Content-Type: text/plain; charset="iso-8859-1"<br><br>I have the same question. Do you have excessively high --entry-timeout<br>parameter to your FUSE mount? In any case, "Structure needs cleaning"
error<br>should not surface up to FUSE and that is still a bug.<br><br><br>On Thu, Dec 12, 2013 at 12:46 PM, Maik Kulbe<br><info@linux-web-development.de>wrote:<br><br>> How do you mount your Client? FUSE? I had similar problems when playing<br>> around with the timeout options for the FUSE mount. If they are too
high<br>> they cache the metadata for too long. When you move the file the inode<br>> should stay the same and on the second node the path should stay in
cache<br>> for a while so it still knows the inode for that moved files old path
thus<br>> can act on the file without knowing it's path.<br>><br>> The problems kick in when you delete a file and recreate it - the
cache<br>> tries to access the old inode, which was deleted, thus throwing errors.
If<br>> I recall correctly the "structure needs cleaning" is one
of two error<br>> messages I got, depending on which of the timeout mount options was
set to<br>> a higher value.<br>><br>> -----Original Mail-----<br>> From: Johan Huysmans [johan.huysmans@inuits.be]<br>> Sent: 12.12.13 - 14:51:35<br>> To: gluster-users@gluster.org [gluster-users@gluster.org]<br>><br>> Subject: Re: [Gluster-users] Structure needs cleaning on some files<br>><br>><br>> I created a bug for this issue:<br>>><br>>> </font></tt><a href="https://bugzilla.redhat.com/show_bug.cgi?id=1041109"><tt><font size=2>https://bugzilla.redhat.com/show_bug.cgi?id=1041109</font></tt></a><tt><font size=2><br>>><br>>> gr.<br>>> Johan<br>>><br>>> On 10-12-13 12:52, Johan Huysmans wrote:<br>>><br>>> Hi All,<br>>><br>>> It seems I can easily reproduce the problem.<br>>><br>>> * on node 1 create a file (touch , cat , ...).<br>>> * on node 2 take md5sum of direct file (md5sum /path/to/file)<br>>> * on node 1 move file to other name (mv file file1)<br>>> * on node 2 take md5sum of direct file (md5sum /path/to/file),
this is<br>>> still working although the file is not really there<br>>> * on node 1 change file content<br>>> * on node 2 take md5sum of direct file (md5sum /path/to/file),
this is<br>>> still working and has a changed md5sum<br>>><br>>> This is really strange behaviour.<br>>> Is this normal, can this be altered with a a setting?<br>>><br>>> Thanks for any info,<br>>> gr.<br>>> Johan<br>>><br>>> On 10-12-13 10:02, Johan Huysmans wrote:<br>>><br>>> I could reproduce this problem with while my mount point is running
in<br>>> debug mode.<br>>> logfile is attached.<br>>><br>>> gr.<br>>> Johan Huysmans<br>>><br>>> On 10-12-13 09:30, Johan Huysmans wrote:<br>>><br>>> Hi All,<br>>><br>>> When reading some files we get this error:<br>>> md5sum: /path/to/file.xml: Structure needs cleaning<br>>><br>>> in /var/log/glusterfs/mnt-sharedfs.log we see these errors:<br>>> [2013-12-10 08:07:32.256910] W<br>>> [client-rpc-fops.c:526:client3_3_stat_cbk] 1-testvolume-client-0:<br>>> remote operation failed: No such file or directory<br>>> [2013-12-10 08:07:32.257436] W<br>>> [client-rpc-fops.c:526:client3_3_stat_cbk] 1-testvolume-client-1:<br>>> remote operation failed: No such file or directory<br>>> [2013-12-10 08:07:32.259356] W [fuse-bridge.c:705:fuse_attr_cbk]<br>>> 0-glusterfs-fuse: 8230: STAT() /path/to/file.xml => -1 (Structure<br>>> needs cleaning)<br>>><br>>> We are using gluster 3.4.1-3 on CentOS6.<br>>> Our servers are 64-bit, our clients 32-bit (we are already using<br>>> --enable-ino32 on the mountpoint)<br>>><br>>> This is my gluster configuration:<br>>> Volume Name: testvolume<br>>> Type: Replicate<br>>> Volume ID: ca9c2f87-5d5b-4439-ac32-b7c138916df7<br>>> Status: Started<br>>> Number of Bricks: 1 x 2 = 2<br>>> Transport-type: tcp<br>>> Bricks:<br>>> Brick1: SRV-1:/gluster/brick1<br>>> Brick2: SRV-2:/gluster/brick2<br>>> Options Reconfigured:<br>>> performance.force-readdirp: on<br>>> performance.stat-prefetch: off<br>>> network.ping-timeout: 5<br>>><br>>> And this is how the applications work:<br>>> We have 2 client nodes who both have a fuse.glusterfs mountpoint.<br>>> On 1 client node we have a application which writes files.<br>>> On the other client node we have a application which reads these<br>>> files.<br>>> On the node where the files are written we don't see any problem,<br>>> and can read that file without problems.<br>>> On the other node we have problems (error messages above) reading<br>>> that file.<br>>> The problem occurs when we perform a md5sum on the exact file,
when<br>>> perform a md5sum on all files in that directory there is no problem.<br>>><br>>> How can we solve this problem as this is annoying.<br>>> The problem occurs after some time (can be days), an umount and<br>>> mount of the mountpoint solves it for some days.<br>>> Once it occurs (and we don't remount) it occurs every time.<br>>><br>>> I hope someone can help me with this problems.<br>>><br>>> Thanks,<br>>> Johan Huysmans<br>>> _______________________________________________<br>>> Gluster-users mailing list<br>>> Gluster-users@gluster.org<br>>> </font></tt><a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users"><tt><font size=2>http://supercolony.gluster.org/mailman/listinfo/gluster-users</font></tt></a><tt><font size=2><br>>><br>>> _______________________________________________<br>>> Gluster-users mailing list<br>>> Gluster-users@gluster.org<br>>> </font></tt><a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users"><tt><font size=2>http://supercolony.gluster.org/mailman/listinfo/gluster-users</font></tt></a><tt><font size=2><br>>><br>>> _______________________________________________<br>>> Gluster-users mailing list<br>>> Gluster-users@gluster.org<br>>> </font></tt><a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users"><tt><font size=2>http://supercolony.gluster.org/mailman/listinfo/gluster-users</font></tt></a><tt><font size=2><br>>><br>><br>> _______________________________________________<br>> Gluster-users mailing list<br>> Gluster-users@gluster.org<br>> </font></tt><a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users"><tt><font size=2>http://supercolony.gluster.org/mailman/listinfo/gluster-users</font></tt></a><tt><font size=2><br>><br>-------------- next part --------------<br>An HTML attachment was scrubbed...<br>URL: <</font></tt><a href="http://supercolony.gluster.org/pipermail/gluster-users/attachments/20131212/730b11a2/attachment-0001.html"><tt><font size=2>http://supercolony.gluster.org/pipermail/gluster-users/attachments/20131212/730b11a2/attachment-0001.html</font></tt></a><tt><font size=2>><br><br>------------------------------<br><br>Message: 8<br>Date: Thu, 12 Dec 2013 13:38:18 -0800<br>From: Anand Avati <avati@gluster.org><br>To: Maik Kulbe <info@linux-web-development.de><br>Cc: "gluster-users@gluster.org" <gluster-users@gluster.org><br>Subject: Re: [Gluster-users] Structure needs cleaning on some files<br>Message-ID:<br>
<CAFboF2zNOFFbuM9_ayrw6Wv+DdXVOr+D=9Az0cxNx+gHiPEpaw@mail.gmail.com><br>Content-Type: text/plain; charset="iso-8859-1"<br><br>Looks like your issue was fixed by patch </font></tt><a href=http://review.gluster.org/4989/><tt><font size=2>http://review.gluster.org/4989/</font></tt></a><tt><font size=2>in<br>master branch. Backporting this to release-3.4 now.<br><br>Thanks!<br>Avati<br><br><br>On Thu, Dec 12, 2013 at 1:26 PM, Anand Avati <avati@gluster.org>
wrote:<br><br>> I have the same question. Do you have excessively high --entry-timeout<br>> parameter to your FUSE mount? In any case, "Structure needs cleaning"
error<br>> should not surface up to FUSE and that is still a bug.<br>><br>><br>> On Thu, Dec 12, 2013 at 12:46 PM, Maik Kulbe <<br>> info@linux-web-development.de> wrote:<br>><br>>> How do you mount your Client? FUSE? I had similar problems when
playing<br>>> around with the timeout options for the FUSE mount. If they are
too high<br>>> they cache the metadata for too long. When you move the file the
inode<br>>> should stay the same and on the second node the path should stay
in cache<br>>> for a while so it still knows the inode for that moved files old
path thus<br>>> can act on the file without knowing it's path.<br>>><br>>> The problems kick in when you delete a file and recreate it -
the cache<br>>> tries to access the old inode, which was deleted, thus throwing
errors. If<br>>> I recall correctly the "structure needs cleaning" is
one of two error<br>>> messages I got, depending on which of the timeout mount options
was set to<br>>> a higher value.<br>>><br>>> -----Original Mail-----<br>>> From: Johan Huysmans [johan.huysmans@inuits.be]<br>>> Sent: 12.12.13 - 14:51:35<br>>> To: gluster-users@gluster.org [gluster-users@gluster.org]<br>>><br>>> Subject: Re: [Gluster-users] Structure needs cleaning on some
files<br>>><br>>><br>>> I created a bug for this issue:<br>>>><br>>>> </font></tt><a href="https://bugzilla.redhat.com/show_bug.cgi?id=1041109"><tt><font size=2>https://bugzilla.redhat.com/show_bug.cgi?id=1041109</font></tt></a><tt><font size=2><br>>>><br>>>> gr.<br>>>> Johan<br>>>><br>>>> On 10-12-13 12:52, Johan Huysmans wrote:<br>>>><br>>>> Hi All,<br>>>><br>>>> It seems I can easily reproduce the problem.<br>>>><br>>>> * on node 1 create a file (touch , cat , ...).<br>>>> * on node 2 take md5sum of direct file (md5sum /path/to/file)<br>>>> * on node 1 move file to other name (mv file file1)<br>>>> * on node 2 take md5sum of direct file (md5sum /path/to/file),
this is<br>>>> still working although the file is not really there<br>>>> * on node 1 change file content<br>>>> * on node 2 take md5sum of direct file (md5sum /path/to/file),
this is<br>>>> still working and has a changed md5sum<br>>>><br>>>> This is really strange behaviour.<br>>>> Is this normal, can this be altered with a a setting?<br>>>><br>>>> Thanks for any info,<br>>>> gr.<br>>>> Johan<br>>>><br>>>> On 10-12-13 10:02, Johan Huysmans wrote:<br>>>><br>>>> I could reproduce this problem with while my mount point is
running in<br>>>> debug mode.<br>>>> logfile is attached.<br>>>><br>>>> gr.<br>>>> Johan Huysmans<br>>>><br>>>> On 10-12-13 09:30, Johan Huysmans wrote:<br>>>><br>>>> Hi All,<br>>>><br>>>> When reading some files we get this error:<br>>>> md5sum: /path/to/file.xml: Structure needs cleaning<br>>>><br>>>> in /var/log/glusterfs/mnt-sharedfs.log we see these errors:<br>>>> [2013-12-10 08:07:32.256910] W<br>>>> [client-rpc-fops.c:526:client3_3_stat_cbk] 1-testvolume-client-0:<br>>>> remote operation failed: No such file or directory<br>>>> [2013-12-10 08:07:32.257436] W<br>>>> [client-rpc-fops.c:526:client3_3_stat_cbk] 1-testvolume-client-1:<br>>>> remote operation failed: No such file or directory<br>>>> [2013-12-10 08:07:32.259356] W [fuse-bridge.c:705:fuse_attr_cbk]<br>>>> 0-glusterfs-fuse: 8230: STAT() /path/to/file.xml => -1
(Structure<br>>>> needs cleaning)<br>>>><br>>>> We are using gluster 3.4.1-3 on CentOS6.<br>>>> Our servers are 64-bit, our clients 32-bit (we are already
using<br>>>> --enable-ino32 on the mountpoint)<br>>>><br>>>> This is my gluster configuration:<br>>>> Volume Name: testvolume<br>>>> Type: Replicate<br>>>> Volume ID: ca9c2f87-5d5b-4439-ac32-b7c138916df7<br>>>> Status: Started<br>>>> Number of Bricks: 1 x 2 = 2<br>>>> Transport-type: tcp<br>>>> Bricks:<br>>>> Brick1: SRV-1:/gluster/brick1<br>>>> Brick2: SRV-2:/gluster/brick2<br>>>> Options Reconfigured:<br>>>> performance.force-readdirp: on<br>>>> performance.stat-prefetch: off<br>>>> network.ping-timeout: 5<br>>>><br>>>> And this is how the applications work:<br>>>> We have 2 client nodes who both have a fuse.glusterfs mountpoint.<br>>>> On 1 client node we have a application which writes files.<br>>>> On the other client node we have a application which reads
these<br>>>> files.<br>>>> On the node where the files are written we don't see any problem,<br>>>> and can read that file without problems.<br>>>> On the other node we have problems (error messages above)
reading<br>>>> that file.<br>>>> The problem occurs when we perform a md5sum on the exact file,
when<br>>>> perform a md5sum on all files in that directory there is no
problem.<br>>>><br>>>> How can we solve this problem as this is annoying.<br>>>> The problem occurs after some time (can be days), an umount
and<br>>>> mount of the mountpoint solves it for some days.<br>>>> Once it occurs (and we don't remount) it occurs every time.<br>>>><br>>>> I hope someone can help me with this problems.<br>>>><br>>>> Thanks,<br>>>> Johan Huysmans<br>>>> _______________________________________________<br>>>> Gluster-users mailing list<br>>>> Gluster-users@gluster.org<br>>>> </font></tt><a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users"><tt><font size=2>http://supercolony.gluster.org/mailman/listinfo/gluster-users</font></tt></a><tt><font size=2><br>>>><br>>>> _______________________________________________<br>>>> Gluster-users mailing list<br>>>> Gluster-users@gluster.org<br>>>> </font></tt><a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users"><tt><font size=2>http://supercolony.gluster.org/mailman/listinfo/gluster-users</font></tt></a><tt><font size=2><br>>>><br>>>> _______________________________________________<br>>>> Gluster-users mailing list<br>>>> Gluster-users@gluster.org<br>>>> </font></tt><a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users"><tt><font size=2>http://supercolony.gluster.org/mailman/listinfo/gluster-users</font></tt></a><tt><font size=2><br>>>><br>>><br>>> _______________________________________________<br>>> Gluster-users mailing list<br>>> Gluster-users@gluster.org<br>>> </font></tt><a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users"><tt><font size=2>http://supercolony.gluster.org/mailman/listinfo/gluster-users</font></tt></a><tt><font size=2><br>>><br>><br>><br>-------------- next part --------------<br>An HTML attachment was scrubbed...<br>URL: <</font></tt><a href="http://supercolony.gluster.org/pipermail/gluster-users/attachments/20131212/f3934ac1/attachment-0001.html"><tt><font size=2>http://supercolony.gluster.org/pipermail/gluster-users/attachments/20131212/f3934ac1/attachment-0001.html</font></tt></a><tt><font size=2>><br><br>------------------------------<br><br>Message: 9<br>Date: Thu, 12 Dec 2013 17:35:02 -0500<br>From: James <purpleidea@gmail.com><br>To: "gluster-users@gluster.org" <gluster-users@gluster.org>,
Gluster<br>
Devel <gluster-devel@nongnu.org><br>Subject: [Gluster-users] Gerrit doesn't use HTTPS<br>Message-ID:<br>
<CADCaTgrmcoJuNL4=vSCDGjjSSND8_eojSEjgUiocHV_cVskKxA@mail.gmail.com><br>Content-Type: text/plain; charset=UTF-8<br><br>I just noticed that the Gluster Gerrit [1] doesn't use HTTPS!<br><br>Can this be fixed ASAP?<br><br>Cheers,<br>James<br><br>[1] </font></tt><a href=http://review.gluster.org/><tt><font size=2>http://review.gluster.org/</font></tt></a><tt><font size=2><br><br><br>------------------------------<br><br>Message: 10<br>Date: Thu, 12 Dec 2013 17:03:12 -0800<br>From: harry mangalam <harry.mangalam@uci.edu><br>To: "gluster-users@gluster.org" <gluster-users@gluster.org><br>Subject: [Gluster-users] gluster fails under heavy array job load load<br>Message-ID: <2288372.PhVIIXonbv@stunted><br>Content-Type: text/plain; charset="us-ascii"<br><br>Hi All,<br>(Gluster Volume Details at bottom)<br><br>I've posted some of this previously, but even after various upgrades, <br>attempted fixes, etc, it remains a problem.<br><br><br>Short version: Our gluster fs (~340TB) provides scratch space for
a ~5000core <br>academic compute cluster. <br>Much of our load is streaming IO, doing a lot of genomics work, and that
is <br>the load under which we saw this latest failure.<br>Under heavy batch load, especially array jobs, where there might be several
<br>64core nodes doing I/O on the 4servers/8bricks, we often get job failures
that <br>have the following profile:<br><br>Client POV:<br>Here is a sampling of the client logs (/var/log/glusterfs/gl.log) for all
<br>compute nodes that indicated interaction with the user's files<br><</font></tt><a href=http://pastie.org/8548781><tt><font size=2>http://pastie.org/8548781</font></tt></a><tt><font size=2>><br><br>Here are some client Info logs that seem fairly serious:<br><</font></tt><a href=http://pastie.org/8548785><tt><font size=2>http://pastie.org/8548785</font></tt></a><tt><font size=2>><br><br>The errors that referenced this user were gathered from all the nodes that
<br>were running his code (in compute*) and agglomerated with:<br><br>cut -f2,3 -d']' compute* |cut -f1 -dP | sort | uniq -c | sort -gr <br><br>and placed here to show the profile of errors that his run generated.<br><</font></tt><a href=http://pastie.org/8548796><tt><font size=2>http://pastie.org/8548796</font></tt></a><tt><font size=2>><br><br>so 71 of them were:<br> W [client-rpc-fops.c:2624:client3_3_lookup_cbk] 0-gl-client-7: remote
<br>operation failed: Transport endpoint is not connected. <br>etc<br><br>We've seen this before and previously discounted it bc it seems to have
been <br>related to the problem of spurious NFS-related bugs, but now I'm wondering
<br>whether it's a real problem. <br>Also the 'remote operation failed: Stale file handle. ' warnings.<br><br>There were no Errors logged per se, tho some of the W's looked fairly nasty,
<br>like the 'dht_layout_dir_mismatch'<br><br>>From the server side, however, during the same period, there were:<br>0 Warnings about this user's files<br>0 Errors <br>458 Info lines<br>of which only 1 line was not a 'cleanup' line like this:<br>---<br>10.2.7.11:[2013-12-12 21:22:01.064289] I [server-helpers.c:460:do_fd_cleanup]
<br>0-gl-server: fd cleanup on /path/to/file<br>---<br>it was:<br>---<br>10.2.7.14:[2013-12-12 21:00:35.209015] I [server-rpc-<br>fops.c:898:_gf_server_log_setxattr_failure] 0-gl-server: 113697332: SETXATTR
<br>/bio/tdlong/RNAseqIII/ckpt.1084030 (c9488341-c063-4175-8492-75e2e282f690)
==> <br>trusted.glusterfs.dht<br>---<br><br>We're losing about 10% of these kinds of array jobs bc of this, which is
just <br>not supportable.<br><br><br><br>Gluster details<br><br>servers and clients running gluster 3.4.0-8.el6 over QDR IB, IPoIB, thru
2 <br>Mellanox, 1 Voltaire switches, Mellanox cards, CentOS 6.4<br><br>$ gluster volume info<br> <br>Volume Name: gl<br>Type: Distribute<br>Volume ID: 21f480f7-fc5a-4fd8-a084-3964634a9332<br>Status: Started<br>Number of Bricks: 8<br>Transport-type: tcp,rdma<br>Bricks:<br>Brick1: bs2:/raid1<br>Brick2: bs2:/raid2<br>Brick3: bs3:/raid1<br>Brick4: bs3:/raid2<br>Brick5: bs4:/raid1<br>Brick6: bs4:/raid2<br>Brick7: bs1:/raid1<br>Brick8: bs1:/raid2<br>Options Reconfigured:<br>performance.write-behind-window-size: 1024MB<br>performance.flush-behind: on<br>performance.cache-size: 268435456<br>nfs.disable: on<br>performance.io-cache: on<br>performance.quick-read: on<br>performance.io-thread-count: 64<br>auth.allow: 10.2.*.*,10.1.*.*<br><br><br>'gluster volume status gl detail': <br><</font></tt><a href=http://pastie.org/8548826><tt><font size=2>http://pastie.org/8548826</font></tt></a><tt><font size=2>><br><br>---<br>Harry Mangalam - Research Computing, OIT, Rm 225 MSTB, UC Irvine<br>[m/c 2225] / 92697 Google Voice Multiplexer: (949) 478-4487<br>415 South Circle View Dr, Irvine, CA, 92697 [shipping]<br>MSTB Lat/Long: (33.642025,-117.844414) (paste into Google Maps)<br>---<br>-------------- next part --------------<br>An HTML attachment was scrubbed...<br>URL: <</font></tt><a href="http://supercolony.gluster.org/pipermail/gluster-users/attachments/20131212/f804e657/attachment-0001.html"><tt><font size=2>http://supercolony.gluster.org/pipermail/gluster-users/attachments/20131212/f804e657/attachment-0001.html</font></tt></a><tt><font size=2>><br><br>------------------------------<br><br>Message: 11<br>Date: Thu, 12 Dec 2013 22:28:57 -0700<br>From: Joe Topjian <joe@topjian.net><br>To: gluster-users@gluster.org<br>Subject: [Gluster-users] qemu remote insecure connections<br>Message-ID:<br>
<CA+y7hvj8LR2mQ9-AbroXLOCb9jQu+M6GLPAduVfs5a3UGcWBEw@mail.gmail.com><br>Content-Type: text/plain; charset="iso-8859-1"<br><br>Hello,<br><br>I'm having a problem getting remote servers to connect to Gluster with
qemu.<br><br>I have 5 servers, 4 of which run Gluster and host a volume. The qemu user<br>on all 5 servers has the same uid.<br><br>storage.owner-uid and storage.owner-gid is set to that user.<br><br>In addition, server.allow-insecure is on and is also set in the<br>glusterd.vol file. glusterd has also been restarted (numerous times).<br><br>When attempting to create a qemu file by connecting to the same server,<br>everything works:<br><br>qemu@192.168.1.11> qemu-img create gluster://192.168.1.11/volumes/v.img
1M<br>Formatting 'gluster://192.168.1.11/volumes/v.img', fmt=raw size=1048576<br>qemu@192.168.1.11><br><br>But when trying to do it remotely, the command hangs indefinitely:<br><br>qemu@192.168.1.12> qemu-img create gluster://192.168.1.11/volumes/v.img
1M<br>Formatting 'gluster://192.168.1.11/volumes/v.img', fmt=raw size=1048576<br>^C<br><br>Yet when 192.168.1.12 connects to gluster://192.168.1.12, the command works<br>and the file shows up in the distributed volume.<br><br>Further, when turning server.allow-insecure off, I get an immediate error<br>no matter what the source and destination connection is:<br><br>qemu@192.168.1.12> qemu-img create gluster://192.168.1.11/volumes/v.img
1M<br>Formatting 'gluster://192.168.1.11/volumes/v.img', fmt=raw size=1048576<br>qemu-img: Gluster connection failed for server=192.168.1.11 port=0<br>volume=volumes image=v.img transport=tcp<br>qemu-img: gluster://192.168.1.11/volumes/v.img: error while creating raw:<br>No data available<br><br>Does anyone have any ideas how I can have an unprivileged user connect
to<br>remote gluster servers?<br><br>Thanks,<br>Joe<br>-------------- next part --------------<br>An HTML attachment was scrubbed...<br>URL: <</font></tt><a href="http://supercolony.gluster.org/pipermail/gluster-users/attachments/20131212/86b86bc1/attachment-0001.html"><tt><font size=2>http://supercolony.gluster.org/pipermail/gluster-users/attachments/20131212/86b86bc1/attachment-0001.html</font></tt></a><tt><font size=2>><br><br>------------------------------<br><br>Message: 12<br>Date: Fri, 13 Dec 2013 11:43:42 +0530<br>From: Vijay Bellur <vbellur@redhat.com><br>To: "'gluster-devel@nongnu.org'" <gluster-devel@nongnu.org><br>Cc: gluster-users Discussion List <Gluster-users@gluster.org><br>Subject: [Gluster-users] Documentation hackathon for 3.5<br>Message-ID: <52AAA596.6020908@redhat.com><br>Content-Type: text/plain; charset=ISO-8859-1; format=flowed<br><br>Hi All,<br><br>The documentation hackathon for 3.5 is underway. You can find more <br>details here [1].<br><br>Anybody who submits a documentation patch that gets accepted between now
<br>and next week will stand a chance to get some swag :).<br><br>Keep your patches coming!<br><br>Cheers,<br>Vijay<br><br>[1] <br></font></tt><a href=http://www.gluster.org/community/documentation/index.php/3.5_Documentation_Hackathon><tt><font size=2>http://www.gluster.org/community/documentation/index.php/3.5_Documentation_Hackathon</font></tt></a><tt><font size=2><br><br><br>------------------------------<br><br>Message: 13<br>Date: Thu, 12 Dec 2013 23:46:03 -0800<br>From: Anand Avati <avati@gluster.org><br>To: harry mangalam <harry.mangalam@uci.edu><br>Cc: "gluster-users@gluster.org" <gluster-users@gluster.org><br>Subject: Re: [Gluster-users] gluster fails under heavy array job load<br>
load<br>Message-ID:<br>
<CAFboF2y8HpFB3xo1Mh43VOiv7jf=g22rHnpxzatGEvtHQgYQ6A@mail.gmail.com><br>Content-Type: text/plain; charset="iso-8859-1"<br><br>Please provide the full client and server logs (in a bug report). The<br>snippets give some hints, but are not very meaningful without the full<br>context/history since mount time (they have after-the-fact symptoms, but<br>not the part which show the reason why disconnects happened).<br><br>Even before looking into the full logs here are some quick observations:<br><br>- write-behind-window-size = 1024MB seems *excessively* high. Please set<br>this to 1MB (default) and check if the stability improves.<br><br>- I see RDMA is enabled on the volume. Are you mounting clients through<br>RDMA? If so, for the purpose of diagnostics can you mount through TCP and<br>check the stability improves? If you are using RDMA with such a high<br>write-behind-window-size, spurious ping-timeouts are an almost certainty<br>during heavy writes. The RDMA driver has limited flow control, and setting<br>such a high window-size can easily congest all the RDMA buffers resulting<br>in spurious ping-timeouts and disconnections.<br><br>Avati<br></font></tt><br><tt><font size=2><br>On Thu, Dec 12, 2013 at 5:03 PM, harry mangalam <harry.mangalam@uci.edu>wrote:<br><br>> Hi All,<br>><br>> (Gluster Volume Details at bottom)<br>><br>><br>><br>> I've posted some of this previously, but even after various upgrades,<br>> attempted fixes, etc, it remains a problem.<br>><br>><br>><br>><br>><br>> Short version: Our gluster fs (~340TB) provides scratch space for
a<br>> ~5000core academic compute cluster.<br>><br>> Much of our load is streaming IO, doing a lot of genomics work, and
that<br>> is the load under which we saw this latest failure.<br>><br>> Under heavy batch load, especially array jobs, where there might be<br>> several 64core nodes doing I/O on the 4servers/8bricks, we often get
job<br>> failures that have the following profile:<br>><br>><br>><br>> Client POV:<br>><br>> Here is a sampling of the client logs (/var/log/glusterfs/gl.log)
for all<br>> compute nodes that indicated interaction with the user's files<br>><br>> <</font></tt><a href=http://pastie.org/8548781><tt><font size=2>http://pastie.org/8548781</font></tt></a><tt><font size=2>><br>><br>><br>><br>> Here are some client Info logs that seem fairly serious:<br>><br>> <</font></tt><a href=http://pastie.org/8548785><tt><font size=2>http://pastie.org/8548785</font></tt></a><tt><font size=2>><br>><br>><br>><br>> The errors that referenced this user were gathered from all the nodes
that<br>> were running his code (in compute*) and agglomerated with:<br>><br>><br>><br>> cut -f2,3 -d']' compute* |cut -f1 -dP | sort | uniq -c | sort -gr<br>><br>><br>><br>> and placed here to show the profile of errors that his run generated.<br>><br>> <</font></tt><a href=http://pastie.org/8548796><tt><font size=2>http://pastie.org/8548796</font></tt></a><tt><font size=2>><br>><br>><br>><br>> so 71 of them were:<br>><br>> W [client-rpc-fops.c:2624:client3_3_lookup_cbk] 0-gl-client-7: remote<br>> operation failed: Transport endpoint is not connected.<br>><br>> etc<br>><br>><br>><br>> We've seen this before and previously discounted it bc it seems to
have<br>> been related to the problem of spurious NFS-related bugs, but now
I'm<br>> wondering whether it's a real problem.<br>><br>> Also the 'remote operation failed: Stale file handle. ' warnings.<br>><br>><br>><br>> There were no Errors logged per se, tho some of the W's looked fairly<br>> nasty, like the 'dht_layout_dir_mismatch'<br>><br>><br>><br>> From the server side, however, during the same period, there were:<br>><br>> 0 Warnings about this user's files<br>><br>> 0 Errors<br>><br>> 458 Info lines<br>><br>> of which only 1 line was not a 'cleanup' line like this:<br>><br>> ---<br>><br>> 10.2.7.11:[2013-12-12 21:22:01.064289] I<br>> [server-helpers.c:460:do_fd_cleanup] 0-gl-server: fd cleanup on<br>> /path/to/file<br>><br>> ---<br>><br>> it was:<br>><br>> ---<br>><br>> 10.2.7.14:[2013-12-12 21:00:35.209015] I<br>> [server-rpc-fops.c:898:_gf_server_log_setxattr_failure] 0-gl-server:<br>> 113697332: SETXATTR /bio/tdlong/RNAseqIII/ckpt.1084030<br>> (c9488341-c063-4175-8492-75e2e282f690) ==> trusted.glusterfs.dht<br>><br>> ---<br>><br>><br>><br>> We're losing about 10% of these kinds of array jobs bc of this, which
is<br>> just not supportable.<br>><br>><br>><br>><br>><br>><br>><br>> Gluster details<br>><br>><br>><br>> servers and clients running gluster 3.4.0-8.el6 over QDR IB, IPoIB,
thru 2<br>> Mellanox, 1 Voltaire switches, Mellanox cards, CentOS 6.4<br>><br>><br>><br>> $ gluster volume info<br>><br>> Volume Name: gl<br>><br>> Type: Distribute<br>><br>> Volume ID: 21f480f7-fc5a-4fd8-a084-3964634a9332<br>><br>> Status: Started<br>><br>> Number of Bricks: 8<br>><br>> Transport-type: tcp,rdma<br>><br>> Bricks:<br>><br>> Brick1: bs2:/raid1<br>><br>> Brick2: bs2:/raid2<br>><br>> Brick3: bs3:/raid1<br>><br>> Brick4: bs3:/raid2<br>><br>> Brick5: bs4:/raid1<br>><br>> Brick6: bs4:/raid2<br>><br>> Brick7: bs1:/raid1<br>><br>> Brick8: bs1:/raid2<br>><br>> Options Reconfigured:<br>><br>> performance.write-behind-window-size: 1024MB<br>><br>> performance.flush-behind: on<br>><br>> performance.cache-size: 268435456<br>><br>> nfs.disable: on<br>><br>> performance.io-cache: on<br>><br>> performance.quick-read: on<br>><br>> performance.io-thread-count: 64<br>><br>> auth.allow: 10.2.*.*,10.1.*.*<br>><br>><br>><br>><br>><br>> 'gluster volume status gl detail':<br>><br>> <</font></tt><a href=http://pastie.org/8548826><tt><font size=2>http://pastie.org/8548826</font></tt></a><tt><font size=2>><br>><br>><br>><br>> ---<br>><br>> Harry Mangalam - Research Computing, OIT, Rm 225 MSTB, UC Irvine<br>><br>> [m/c 2225] / 92697 Google Voice Multiplexer: (949) 478-4487<br>><br>> 415 South Circle View Dr, Irvine, CA, 92697 [shipping]<br>><br>> MSTB Lat/Long: (33.642025,-117.844414) (paste into Google Maps)<br>><br>> ---<br>><br>><br>><br>> _______________________________________________<br>> Gluster-users mailing list<br>> Gluster-users@gluster.org<br>> </font></tt><a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users"><tt><font size=2>http://supercolony.gluster.org/mailman/listinfo/gluster-users</font></tt></a><tt><font size=2><br>><br>-------------- next part --------------<br>An HTML attachment was scrubbed...<br>URL: <</font></tt><a href="http://supercolony.gluster.org/pipermail/gluster-users/attachments/20131212/4fea4498/attachment-0001.html"><tt><font size=2>http://supercolony.gluster.org/pipermail/gluster-users/attachments/20131212/4fea4498/attachment-0001.html</font></tt></a><tt><font size=2>><br><br>------------------------------<br><br>Message: 14<br>Date: Fri, 13 Dec 2013 09:30:17 +0100<br>From: Niels de Vos <ndevos@redhat.com><br>To: Vijay Bellur <vbellur@redhat.com><br>Cc: "gluster-users@gluster.org" <gluster-users@gluster.org>,
Gluster<br>
Devel <gluster-devel@nongnu.org><br>Subject: Re: [Gluster-users] Gluster Community Weekly Meeting<br>Message-ID: <20131213083017.GQ11941@ndevos-laptop.usersys.redhat.com><br>Content-Type: text/plain; charset=us-ascii<br><br>On Fri, Dec 13, 2013 at 12:13:30AM +0530, Vijay Bellur wrote:<br>> On 12/12/2013 10:42 AM, James wrote:<br>> >RE: meeting, sorry I couldn't make it, but I have some comments:<br>> <br>> No problem. It would be really good to have everybody in the<br>> meeting, but if you cannot comments are definitely welcome :).<br>> <br>> ><br>> >1) About the pre-packaged VM comment's. I've gotten Vagrant working
on<br>> >Fedora. I'm using this to rapidly spin up and test GlusterFS.<br>> ></font></tt><a href="https://ttboj.wordpress.com/2013/12/09/vagrant-on-fedora-with-libvirt/"><tt><font size=2>https://ttboj.wordpress.com/2013/12/09/vagrant-on-fedora-with-libvirt/</font></tt></a><tt><font size=2><br>> >In the coming week or so, I'll be publishing the Vagrant file
for my<br>> >GlusterFS setup, but if you really want it now I can send you
an early<br>> >version. This obviously integrates with Puppet-Gluster, but whether<br>> >you use that or not is optional. I think this is the best way
to test<br>> >GlusterFS. If someone gives me hosting, I could publish "pre-built"<br>> >images very easily. Let me know what you think.<br>> <br>> Niels - do you have any thoughts here?<br><br>I was thinking of using virt-builder[1] which is already part of Fedora.
<br>Personally I would stick with the Fedora tools, and not use yet <br>something else again. However, I'm more than happy if James builds and
<br>publishes one or more VMs for the test days, of course he is free to use
<br>whatever tools he likes :-)<br><br>The images should be minimal, and I do not expect them to be bigger than
<br>512MB when compressed. Best would be to host them on the <br>download.gluster.org server, at the same location of the packages.<br><br>Niels<br><br>1. </font></tt><a href="http://libguestfs.org/virt-builder.1.html"><tt><font size=2>http://libguestfs.org/virt-builder.1.html</font></tt></a><tt><font size=2><br><br><br>------------------------------<br><br>_______________________________________________<br>Gluster-users mailing list<br>Gluster-users@gluster.org<br></font></tt><a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users"><tt><font size=2>http://supercolony.gluster.org/mailman/listinfo/gluster-users</font></tt></a><tt><font size=2><br><br>End of Gluster-users Digest, Vol 68, Issue 14<br>*********************************************<br></font></tt><br><BR>
**<BR>
<BR>
This email and any attachments may contain information that is confidential and/or privileged for the sole use of the intended recipient. Any use, review, disclosure, copying, distribution or reliance by others, and any forwarding of this email or its contents, without the express permission of the sender is strictly prohibited by law. If you are not the intended recipient, please contact the sender immediately, delete the e-mail and destroy all copies.<BR>
**<BR>