<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">Dear All,<div>we are facing a problem in our computer room, we have 6 servers that act like bricks for GlusterFS, the servers are configured in the following way:</div><div><br></div><div>OS: Centos 6.2 x86_64</div><div>Kernel: 2.6.32-220.4.2.el6.x86_64</div><div><br></div><div>Gluster RPM packages:</div><div><div>glusterfs-core-3.2.5-2.el6.x86_64</div><div>glusterfs-rdma-3.2.5-2.el6.x86_64</div><div>glusterfs-geo-replication-3.2.5-2.el6.x86_64</div><div>glusterfs-fuse-3.2.5-2.el6.x86_64</div></div><div><br></div><div>Each one is contributing a XFS filesystem to the global volume, the transport mechanism is RDMA:</div><div><br></div><div><code class="java plain">gluster volume create HPC_data transport rdma pleiades01:/data pleiades02:/data pleiades03:/data pleiades04:/data
pleiades05:/data pleiades06:/data</code></div><div><font class="Apple-style-span" face="monospace"><br></font></div><div>Each server mounts, using the fuse driver, the volume on a dedicated mount point according to the following fstab:</div><div><br></div><div>pleiades01:/HPC_data /HPCdata glusterfs defaults,_netdev 0 0</div><div><br></div><div>We are running mongodb on top of the Gluster volume for performance testing and speed is definitely high. Unfortunately when we run a large mongoimport job after short time from the beginning the GlusterFS volume hangs completely and is inaccessible from any node. The following error is logged after some time in /var/log/messages:</div><div><br></div><div>Mar 8 08:16:03 pleiades03 kernel: INFO: task mongod:5508 blocked for more than 120 seconds.</div><div>Mar 8 08:16:03 pleiades03 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.</div><div>Mar 8 08:16:03 pleiades03 kernel: mongod D 0000000000000007 0 5508 1 0x00000000</div><div>Mar 8 08:16:03 pleiades03 kernel: ffff881709b95de8 0000000000000086 0000000000000000 0000000000000008</div><div>Mar 8 08:16:03 pleiades03 kernel: ffff881709b95d68 ffffffff81090a7f ffff8816b6974cc0 0000000000000000</div><div>Mar 8 08:16:03 pleiades03 kernel: ffff8817fdd81af8 ffff881709b95fd8 000000000000f4e8 ffff8817fdd81af8</div><div>Mar 8 08:16:03 pleiades03 kernel: Call Trace:</div><div>Mar 8 08:16:03 pleiades03 kernel: [<ffffffff81090a7f>] ? wake_up_bit+0x2f/0x40</div><div>Mar 8 08:16:03 pleiades03 kernel: [<ffffffff81090d7e>] ? prepare_to_wait+0x4e/0x80</div><div>Mar 8 08:16:03 pleiades03 kernel: [<ffffffffa112c6b5>] fuse_set_nowrite+0xa5/0xe0 [fuse]</div><div>Mar 8 08:16:03 pleiades03 kernel: [<ffffffff81090a90>] ? autoremove_wake_function+0x0/0x40</div><div>Mar 8 08:16:03 pleiades03 kernel: [<ffffffffa112fd48>] fuse_fsync_common+0xa8/0x180 [fuse]</div><div>Mar 8 08:16:03 pleiades03 kernel: [<ffffffffa112fe30>] fuse_fsync+0x10/0x20 [fuse]</div><div>Mar 8 08:16:03 pleiades03 kernel: [<ffffffff811a52d1>] vfs_fsync_range+0xa1/0xe0</div><div>Mar 8 08:16:03 pleiades03 kernel: [<ffffffff811a537d>] vfs_fsync+0x1d/0x20</div><div>Mar 8 08:16:03 pleiades03 kernel: [<ffffffff81144421>] sys_msync+0x151/0x1e0</div><div>Mar 8 08:16:03 pleiades03 kernel: [<ffffffff8100b0f2>] system_call_fastpath+0x16/0x1b</div><div><br></div><div>Any attempt to access the volume from any node is fruitless until the mongodb process is killed, the sessions accessing the /HPCdata path gets freezed on any node. </div><div>Anyway a complete stop (force) and start of the volume is needed to have it back operational.</div><div>The situation can be reproduced at will.</div><div>Is there anybody able to help us? Could we collect more pieces of information to help diagnosing the problem?</div><div><br></div><div>Thanks a lot</div><div>Alessio <br>
<br></div></body></html>