<div dir="ltr">Hi list<div><br></div><div style>Recently I've experienced more and more input/output errors from my most write heavy gluster filesystem.</div><div style>The logfile on the gluster servers show nothing, but the client(s) that get the input/output errors (and timeouts) will as far as I can tell get errors such as :</div>
<div style>[2013-06-14 15:55:56] W [fuse-bridge.c:493:fuse_entry_cbk] glusterfs-fuse: LOOKUP(/369/60702093) inode (ptr=0x1efd440, ino=865790953174, gen=5881526961019962283) found conflict (ptr=0x21a8980, ino=865790953174, gen=5881526961019964013)<br>
</div><div style><br></div><div style>I've enabled trace logging on a few clients and the attached file has an excerpt from one of the clients from when one of the found conflict errors occur. I wasn't sure which parts were relevant so there are logmessages from more than the directory which is causing the problem.</div>
<div style>It seems to be path /369 and /369/60702093 which is causing the problem.</div><div style><br></div><div style>We're currently using version 3.0.5 and I know people will say the solution is to upgrade. At this point in time this is not something I can do easily as I've got almost 6 PB of data spread across 90 storage nodes with roughly 1100 bricks on 21 gluster filesystems. This again has caused me problems on the client nodes with running out of ports below 1024, hence we had to enable user/pass authentication to allow clients to connect from ports above 1024. I'm mentioning this in case it is relevant for the connection/timeout/conflict problem we're experiencing.</div>
<div style><br></div><div style>Excerpt from the included file which might be the most relevant parts:</div><div style><div>[2013-06-14 15:55:54] T [fuse-bridge.c:596:fuse_lookup] glusterfs-fuse: 3642552: LOOKUP /369/60702093</div>
<div>[2013-06-14 15:55:54] T [dht-layout.c:306:dht_disk_layout_merge] distribute: merged to layout: 1610612730 - 1700091214 (type 0) from dn-083-1</div><div>[2013-06-14 15:55:54] T [dht-layout.c:306:dht_disk_layout_merge] distribute: merged to layout: 894784850 - 984263334 (type 0) from dn-082-1</div>
<div>.</div><div><div>[2013-06-14 15:55:54] D [dht-layout.c:576:dht_layout_normalize] distribute: found anomalies in /369/60702093. holes=1 overlaps=1</div><div>[2013-06-14 15:55:54] D [dht-common.c:164:dht_lookup_dir_cbk] distribute: fixing assignment on /369/60702093</div>
<div>[2013-06-14 15:55:54] T [fuse-bridge.c:1964:fuse_write] glusterfs-fuse: 3642624: WRITE (0x18f5d80, size=131072, offset=2962411520)</div><div>[2013-06-14 15:55:54] T [fuse-bridge.c:1912:fuse_writev_cbk] glusterfs-fuse: 3642624: WRITE => 131072/131072,2962411520/0</div>
<div>[2013-06-14 15:55:54] T [dht-selfheal.c:352:dht_selfheal_layout_new_directory] distribute: gave fix: 0 - 89478484 on dn-086-7 for /369/60702093</div><div>[2013-06-14 15:55:54] T [io-cache.c:133:ioc_inode_flush] iocache: locked inode(0x7f6628614170)</div>
<div>[2013-06-14 15:55:54] T [dht-selfheal.c:352:dht_selfheal_layout_new_directory] distribute: gave fix: 89478485 - 178956969 on dn-086-8 for /369/60702093</div></div><div>.</div><div><div>[2013-06-14 15:55:54] T [dht-selfheal.c:174:dht_selfheal_dir_xattr] distribute: 48 subvolumes missing xattr for /369/60702093</div>
</div><div><div>[2013-06-14 15:55:54] T [fuse-bridge.c:1964:fuse_write] glusterfs-fuse: 3642628: WRITE (0x18f5d80, size=131072, offset=2962935808)</div><div>[2013-06-14 15:55:54] T [dht-selfheal.c:124:dht_selfheal_dir_xattr_persubvol] distribute: setting hash range 894784850 - 984263334 (type 0) on subvolume dn-082-1 for /369/60702093</div>
<div>[2013-06-14 15:55:54] T [fuse-bridge.c:1912:fuse_writev_cbk] glusterfs-fuse: 3642628: WRITE => 131072/131072,2962935808/0</div><div>[2013-06-14 15:55:54] T [dht-selfheal.c:124:dht_selfheal_dir_xattr_persubvol] distribute: setting hash range 984263335 - 1073741819 (type 0) on subvolume dn-082-2 for /369/60702093</div>
<div>[2013-06-14 15:55:54] T [io-cache.c:133:ioc_inode_flush] iocache: locked inode(0x7f6628614170)</div></div><div>.</div><div><div>[2013-06-14 15:55:55] T [dht-layout.c:349:dht_layout_merge] distribute: missing disk layout on dn-083-1. err = -1</div>
<div>[2013-06-14 15:55:55] T [dht-selfheal.c:352:dht_selfheal_layout_new_directory] distribute: gave fix: 0 - 91382281 on dn-083-8 for /369/60702093/344</div><div>[2013-06-14 15:55:55] T [dht-selfheal.c:352:dht_selfheal_layout_new_directory] distribute: gave fix: 91382282 - 182764563 on dn-084-1 for /369/60702093/344</div>
<div>[2013-06-14 15:55:55] T [dht-selfheal.c:352:dht_selfheal_layout_new_directory] distribute: gave fix: 182764564 - 274146845 on dn-084-2 for /369/60702093/344</div></div><div><br></div><div style>etc</div><div style><br>
</div><div style>I hope someone can help shed some light over what might be the cause of the problem here.</div><div style>Thanks in advance</div><div style><br></div><div style>Ingard</div></div></div>