Is this a new setup and used to work before? How is the CPU, memory etc? Also, what do you see in gluster nodes?<br><br>
<div class="gmail_quote">On Wed, Mar 14, 2012 at 7:33 PM, Alessio Checcucci <span dir="ltr"><<a href="mailto:alessio.checcucci@gmail.com">alessio.checcucci@gmail.com</a>></span> wrote:<br>
<blockquote style="BORDER-LEFT:#ccc 1px solid;MARGIN:0px 0px 0px 0.8ex;PADDING-LEFT:1ex" class="gmail_quote">
<div style="WORD-WRAP:break-word">Dear All,
<div>we are facing a problem in our computer room, we have 6 servers that act like bricks for GlusterFS, the servers are configured in the following way:</div>
<div><br></div>
<div>OS: Centos 6.2 x86_64</div>
<div>Kernel: 2.6.32-220.4.2.el6.x86_64</div>
<div><br></div>
<div>Gluster RPM packages:</div>
<div>
<div>glusterfs-core-3.2.5-2.el6.x86_64</div>
<div>glusterfs-rdma-3.2.5-2.el6.x86_64</div>
<div>glusterfs-geo-replication-3.2.5-2.el6.x86_64</div>
<div>glusterfs-fuse-3.2.5-2.el6.x86_64</div></div>
<div><br></div>
<div>Each one is contributing a XFS filesystem to the global volume, the transport mechanism is RDMA:</div>
<div><br></div>
<div><code>gluster volume create HPC_data transport rdma pleiades01:/data pleiades02:/data pleiades03:/data pleiades04:/data pleiades05:/data pleiades06:/data</code></div>
<div><font face="monospace"><br></font></div>
<div>Each server mounts, using the fuse driver, the volume on a dedicated mount point according to the following fstab:</div>
<div><br></div>
<div>pleiades01:/HPC_data /HPCdata glusterfs defaults,_netdev 0 0</div>
<div><br></div>
<div>We are running mongodb on top of the Gluster volume for performance testing and speed is definitely high. Unfortunately when we run a large mongoimport job after short time from the beginning the GlusterFS volume hangs completely and is inaccessible from any node. The following error is logged after some time in /var/log/messages:</div>
<div><br></div>
<div>Mar 8 08:16:03 pleiades03 kernel: INFO: task mongod:5508 blocked for more than 120 seconds.</div>
<div>Mar 8 08:16:03 pleiades03 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.</div>
<div>Mar 8 08:16:03 pleiades03 kernel: mongod D 0000000000000007 0 5508 1 0x00000000</div>
<div>Mar 8 08:16:03 pleiades03 kernel: ffff881709b95de8 0000000000000086 0000000000000000 0000000000000008</div>
<div>Mar 8 08:16:03 pleiades03 kernel: ffff881709b95d68 ffffffff81090a7f ffff8816b6974cc0 0000000000000000</div>
<div>Mar 8 08:16:03 pleiades03 kernel: ffff8817fdd81af8 ffff881709b95fd8 000000000000f4e8 ffff8817fdd81af8</div>
<div>Mar 8 08:16:03 pleiades03 kernel: Call Trace:</div>
<div>Mar 8 08:16:03 pleiades03 kernel: [<ffffffff81090a7f>] ? wake_up_bit+0x2f/0x40</div>
<div>Mar 8 08:16:03 pleiades03 kernel: [<ffffffff81090d7e>] ? prepare_to_wait+0x4e/0x80</div>
<div>Mar 8 08:16:03 pleiades03 kernel: [<ffffffffa112c6b5>] fuse_set_nowrite+0xa5/0xe0 [fuse]</div>
<div>Mar 8 08:16:03 pleiades03 kernel: [<ffffffff81090a90>] ? autoremove_wake_function+0x0/0x40</div>
<div>Mar 8 08:16:03 pleiades03 kernel: [<ffffffffa112fd48>] fuse_fsync_common+0xa8/0x180 [fuse]</div>
<div>Mar 8 08:16:03 pleiades03 kernel: [<ffffffffa112fe30>] fuse_fsync+0x10/0x20 [fuse]</div>
<div>Mar 8 08:16:03 pleiades03 kernel: [<ffffffff811a52d1>] vfs_fsync_range+0xa1/0xe0</div>
<div>Mar 8 08:16:03 pleiades03 kernel: [<ffffffff811a537d>] vfs_fsync+0x1d/0x20</div>
<div>Mar 8 08:16:03 pleiades03 kernel: [<ffffffff81144421>] sys_msync+0x151/0x1e0</div>
<div>Mar 8 08:16:03 pleiades03 kernel: [<ffffffff8100b0f2>] system_call_fastpath+0x16/0x1b</div>
<div><br></div>
<div>Any attempt to access the volume from any node is fruitless until the mongodb process is killed, the sessions accessing the /HPCdata path gets freezed on any node. </div>
<div>Anyway a complete stop (force) and start of the volume is needed to have it back operational.</div>
<div>The situation can be reproduced at will.</div>
<div>Is there anybody able to help us? Could we collect more pieces of information to help diagnosing the problem?</div>
<div><br></div>
<div>Thanks a lot</div>
<div>Alessio <font color="#888888"><br><br></font></div></div><br>_______________________________________________<br>Gluster-users mailing list<br><a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
<a href="http://gluster.org/cgi-bin/mailman/listinfo/gluster-users" target="_blank">http://gluster.org/cgi-bin/mailman/listinfo/gluster-users</a><br><br></blockquote></div><br>