Hello, <div>some weeks ago, i send report to tell you that's glusterfs 3.x reboot our system when we are testing some ha ( desactivate network interface : ifconfig eth0 down).</div><div>You cannot reproduce into your systems.</div>
<div><br></div><div>Reboot of our system is due to : hung_task_panic and hung_task_timeout_secs , when a task is blocking during 120 s , linux kernel does panic.</div><div>so set ung_task_panic to 0 or hung_task_timeout_secs > 600 to let some time.</div>
<div><br></div><div><br></div><meta http-equiv="content-type" content="text/html; charset=utf-8"><meta http-equiv="content-type" content="text/html; charset=utf-8"><meta http-equiv="content-type" content="text/html; charset=utf-8"><div>
<div>1 - two server / client in replicate mode</div><div>2 - First server 10.98.98.1 is configuration server </div><div>3 - run gluster on two servers as :</div><div><div> /usr/local/sbin/glusterfsd --log-level=DEBUG --log-file=/tmpsafe/server.log -N -f /etc/glusterfs/glusterfs-server.vol</div>
<div>/usr/local/sbin/glusterfs --log-level=DEBUG --log-file=/tmpsafe/client.log -N -s 10.98.98.1 /mnt/vdisk/</div><div><br></div><div>4 - now on 10.98.98.1, do a ifconfig eth0 down.</div><div>5 - on 10.98.98.10, after a little timeout, ls /mnt/vdisk comes back ( using 10.98.98.10 as server )</div>
<div>6 - on 10.98.98.1 , ls /mnt/vdisk hangs forever</div><div>7 - on 10.98.98.1 , kill glusterfs client, rerun glusterfs , then ls /mnt/vdisk reworks again ( using 10.98.98.1 as server )</div></div><div><br></div><div>during 6 , there's no log on server and client on 10.98.98.1</div>
<div><br></div><div>show log, </div><div>Regards, </div><div>Nicolas Prochazka.</div><div><br></div><div>-----------------------------------------------</div><div><br></div><div><br></div><div><br></div><div><div>#This file is auto generated, not edit ( Nicolas Prochazka Sep 2009)</div>
<div># ------------- Create Brick blade definition</div><div>volume 10.98.98.1</div><div>type protocol/client</div><div>option transport-type tcp/client</div><div>option remote-host 10.98.98.1</div><div>option transport.socket.nodelay on</div>
<div>option remote-subvolume brick</div><div>end-volume</div><div><br></div><div><br></div><div>volume 10.98.98.10</div><div>type protocol/client</div><div>option transport-type tcp/client</div><div>option remote-host 10.98.98.10</div>
<div>option transport.socket.nodelay on</div><div>option remote-subvolume brick</div><div>end-volume</div><div><br></div><div><br></div><div># ------------- Create Brick Replicate definition</div><div># ------------- Create Distribute definition</div>
<div>volume last</div><div>type cluster/distribute</div><div>subvolumes 10.98.98.1 10.98.98.10</div><div>end-volume</div><div><br></div><div><br></div><div><br></div><div>volume iothreads</div><div>type performance/io-threads</div>
<div>option thread-count 8</div><div>subvolumes last</div><div>end-volume</div><div><br></div><div>volume io-cache</div><div>type performance/io-cache</div><div>option cache-size 2GB # default is 32MB</div><div>
option cache-timeout 5 # default is 1</div><div>subvolumes iothreads</div><div>end-volume</div><div><br></div><div>volume writebehind</div><div>type performance/write-behind</div><div>option cache-size 4MB</div><div>subvolumes io-cache</div>
<div>end-volume</div><div><br></div></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><div>DEV-10.98.98.1:~# cat /etc/glusterfs/glusterfs-server.vol </div><div>volume brickless</div><div>
type storage/posix</div><div>option directory /mnt/disks/export</div><div>end-volume</div><div><br></div><div>volume brickthread</div><div>type features/locks</div><div>subvolumes brickless</div><div>end-volume</div><div>
<br></div><div>volume brickcache</div><div>type performance/io-cache</div><div>option cache-size 2GB # default is 32MB</div><div>option cache-timeout 2 # default is 1</div><div>subvolumes brickthread</div><div>
end-volume</div><div><br></div><div><br></div><div>volume brick</div><div>type performance/io-threads</div><div>option thread-count 8</div><div>subvolumes brickcache</div><div>end-volume</div><div><br></div><div><br></div>
<div><br></div><div>volume server</div><div>type protocol/server</div><div>subvolumes brick</div><div>option client-volume-filename /etc/glusterfs/Gglusterfs-client.vol</div><div>option transport-type tcp</div><div>option transport.socket.nodelay on</div>
<div>option verify-volfile-checksum no</div><div>option auth.addr.brick.allow 10.98.98.*</div><div>end-volume</div><div><br></div><div><br></div><div>Log of client on 10.98.98.10 , all seems to be ok. </div><div><br></div>
</div><div><div>[2010-03-29 12:48:04] E [client-protocol.c:415:client_ping_timer_expired] <a href="http://10.98.98.1">10.98.98.1</a>: Server <a href="http://10.98.98.1:6996">10.98.98.1:6996</a> has not responded in the last 42 seconds, disconnecting.</div>
<div>[2010-03-29 12:48:04] E [saved-frames.c:165:saved_frames_unwind] <a href="http://10.98.98.1">10.98.98.1</a>: forced unwinding frame type(1) op(STATFS)</div><div>[2010-03-29 12:48:04] E [saved-frames.c:165:saved_frames_unwind] <a href="http://10.98.98.1">10.98.98.1</a>: forced unwinding frame type(2) op(PING)</div>
<div>[2010-03-29 12:48:04] D [client-protocol.c:537:client_ping_cbk] <a href="http://10.98.98.1">10.98.98.1</a>: timer must have expired</div><div>[2010-03-29 12:48:04] N [client-protocol.c:6994:notify] <a href="http://10.98.98.1">10.98.98.1</a>: disconnected</div>
<div>[2010-03-29 12:48:06] E [socket.c:762:socket_connect_finish] <a href="http://10.98.98.1">10.98.98.1</a>: connection to <a href="http://10.98.98.1:6996">10.98.98.1:6996</a> failed (No route to host)</div><div>[2010-03-29 12:48:09] E [socket.c:762:socket_connect_finish] <a href="http://10.98.98.1">10.98.98.1</a>: connection to <a href="http://10.98.98.1:6996">10.98.98.1:6996</a> failed (No route to host)</div>
</div><div><br></div><div><br></div><div>log on 10.98.98.1</div><div><br></div><div><br><div><div><div><div><div>[2010-03-29 16:30:17] D [dht-diskusage.c:71:dht_du_info_cbk] last: on subvolume '10.98.98.1': avail_percent is: 99.00 and avail_space is: 15069396992</div>
<div>[2010-03-29 16:30:17] N [client-protocol.c:6246:client_setvolume_cbk] <a href="http://10.98.98.1">10.98.98.1</a>: Connected to <a href="http://10.98.98.1:6996">10.98.98.1:6996</a>, attached to remote volume 'brick'.</div>
<div>[2010-03-29 16:30:17] N [client-protocol.c:6246:client_setvolume_cbk] <a href="http://10.98.98.10">10.98.98.10</a>: Connected to <a href="http://10.98.98.10:6996">10.98.98.10:6996</a>, attached to remote volume 'brick'.</div>
<div>[2010-03-29 16:30:17] N [client-protocol.c:6246:client_setvolume_cbk] <a href="http://10.98.98.10">10.98.98.10</a>: Connected to <a href="http://10.98.98.10:6996">10.98.98.10:6996</a>, attached to remote volume 'brick'.</div>
<div>[2010-03-29 16:30:17] D [dht-diskusage.c:71:dht_du_info_cbk] last: on subvolume '10.98.98.1': avail_percent is: 99.00 and avail_space is: 15069396992</div><div>[2010-03-29 16:30:17] D [dht-diskusage.c:71:dht_du_info_cbk] last: on subvolume '10.98.98.10': avail_percent is: 99.00 and avail_space is: 88316628992</div>
<div>[2010-03-29 16:30:17] D [dht-diskusage.c:71:dht_du_info_cbk] last: on subvolume '10.98.98.10': avail_percent is: 99.00 and avail_space is: 88316628992</div><div>[2010-03-29 16:30:21] D [dht-layout.c:576:dht_layout_normalize] last: found anomalies in /iso. holes=1 overlaps=0</div>
<div>[2010-03-29 16:30:21] D [dht-common.c:164:dht_lookup_dir_cbk] last: fixing assignment on /iso</div><div>[2010-03-29 16:30:21] D [dht-layout.c:576:dht_layout_normalize] last: found anomalies in /ha. holes=1 overlaps=0</div>
<div>[2010-03-29 16:30:21] D [dht-common.c:164:dht_lookup_dir_cbk] last: fixing assignment on /ha</div><div>[2010-03-29 16:30:21] D [dht-layout.c:576:dht_layout_normalize] last: found anomalies in /monitoring. holes=1 overlaps=0</div>
<div>[2010-03-29 16:30:21] D [dht-common.c:164:dht_lookup_dir_cbk] last: fixing assignment on /monitoring</div><div><br></div><div>nothing during hang</div><div>restart</div><div><br></div><div>[2010-03-29 16:58:26] E [saved-frames.c:165:saved_frames_unwind] <a href="http://10.98.98.10">10.98.98.10</a>: forced unwinding frame type(1) op(LOOKUP)</div>
<div>[2010-03-29 16:58:26] E [saved-frames.c:165:saved_frames_unwind] <a href="http://10.98.98.10">10.98.98.10</a>: forced unwinding frame type(1) op(STATFS)</div><div>[2010-03-29 16:58:26] E [saved-frames.c:165:saved_frames_unwind] <a href="http://10.98.98.10">10.98.98.10</a>: forced unwinding frame type(1) op(LOOKUP)</div>
<div>[2010-03-29 16:58:26] E [saved-frames.c:165:saved_frames_unwind] <a href="http://10.98.98.10">10.98.98.10</a>: forced unwinding frame type(1) op(STATFS)</div><div>[2010-03-29 16:58:26] E [saved-frames.c:165:saved_frames_unwind] <a href="http://10.98.98.10">10.98.98.10</a>: forced unwinding frame type(1) op(STATFS)</div>
<div>[2010-03-29 16:58:26] E [saved-frames.c:165:saved_frames_unwind] <a href="http://10.98.98.10">10.98.98.10</a>: forced unwinding frame type(1) op(STATFS)</div><div>[2010-03-29 16:58:26] E [saved-frames.c:165:saved_frames_unwind] <a href="http://10.98.98.10">10.98.98.10</a>: forced unwinding frame type(1) op(LOOKUP)</div>
<div>[2010-03-29 16:58:26] E [saved-frames.c:165:saved_frames_unwind] <a href="http://10.98.98.10">10.98.98.10</a>: forced unwinding frame type(1) op(STATFS)</div><div>[2010-03-29 16:58:26] E [saved-frames.c:165:saved_frames_unwind] <a href="http://10.98.98.10">10.98.98.10</a>: forced unwinding frame type(1) op(LOOKUP)</div>
<div>[2010-03-29 16:58:26] E [saved-frames.c:165:saved_frames_unwind] <a href="http://10.98.98.10">10.98.98.10</a>: forced unwinding frame type(1) op(STATFS)</div><div>[2010-03-29 16:58:26] E [saved-frames.c:165:saved_frames_unwind] <a href="http://10.98.98.10">10.98.98.10</a>: forced unwinding frame type(1) op(LOOKUP)</div>
<div>[2010-03-29 16:58:26] D [socket.c:1326:socket_submit] <a href="http://10.98.98.10">10.98.98.10</a>: not connected (priv->connected = 255)</div><div>[2010-03-29 16:58:26] D [dht-common.c:1590:dht_fd_cbk] last: subvolume 10.98.98.10 returned -1 (Transport endpoint is not connected)</div>
<div>[2010-03-29 16:58:26] E [saved-frames.c:165:saved_frames_unwind] <a href="http://10.98.98.10">10.98.98.10</a>: forced unwinding frame type(1) op(STATFS)</div><div>[2010-03-29 16:58:26] E [saved-frames.c:165:saved_frames_unwind] <a href="http://10.98.98.10">10.98.98.10</a>: forced unwinding frame type(1) op(STATFS)</div>
<div>[2010-03-29 16:58:26] E [saved-frames.c:165:saved_frames_unwind] <a href="http://10.98.98.10">10.98.98.10</a>: forced unwinding frame type(1) op(STATFS)</div><div>[2010-03-29 16:58:26] D [dht-common.c:1590:dht_fd_cbk] last: subvolume 10.98.98.10 returned -1 (Transport endpoint is not connected)</div>
<div>[2010-03-29 16:58:26] E [saved-frames.c:165:saved_frames_unwind] <a href="http://10.98.98.10">10.98.98.10</a>: forced unwinding frame type(1) op(STATFS)</div><div>[2010-03-29 16:58:26] E [saved-frames.c:165:saved_frames_unwind] <a href="http://10.98.98.10">10.98.98.10</a>: forced unwinding frame type(1) op(STATFS)</div>
<div>[2010-03-29 16:58:26] E [saved-frames.c:165:saved_frames_unwind] <a href="http://10.98.98.10">10.98.98.10</a>: forced unwinding frame type(1) op(STATFS)</div><div>[2010-03-29 16:58:26] E [saved-frames.c:165:saved_frames_unwind] <a href="http://10.98.98.10">10.98.98.10</a>: forced unwinding frame type(1) op(STATFS)</div>
<div>[2010-03-29 16:58:26] E [saved-frames.c:165:saved_frames_unwind] <a href="http://10.98.98.10">10.98.98.10</a>: forced unwinding frame type(1) op(STATFS)</div><div>[2010-03-29 16:58:26] E [saved-frames.c:165:saved_frames_unwind] <a href="http://10.98.98.10">10.98.98.10</a>: forced unwinding frame type(1) op(STATFS)</div>
<div>[2010-03-29 16:58:26] E [saved-frames.c:165:saved_frames_unwind] <a href="http://10.98.98.10">10.98.98.10</a>: forced unwinding frame type(1) op(STATFS)</div><div>[2010-03-29 16:58:26] E [saved-frames.c:165:saved_frames_unwind] <a href="http://10.98.98.10">10.98.98.10</a>: forced unwinding frame type(1) op(STATFS)</div>
<div>[2010-03-29 16:58:26] E [saved-frames.c:165:saved_frames_unwind] <a href="http://10.98.98.10">10.98.98.10</a>: forced unwinding frame type(1) op(STATFS)</div><div>[2010-03-29 16:58:26] E [saved-frames.c:165:saved_frames_unwind] <a href="http://10.98.98.10">10.98.98.10</a>: forced unwinding frame type(1) op(STATFS)</div>
<div>[2010-03-29 16:58:26] E [saved-frames.c:165:saved_frames_unwind] <a href="http://10.98.98.10">10.98.98.10</a>: forced unwinding frame type(1) op(STATFS)</div><div>[2010-03-29 16:58:26] E [saved-frames.c:165:saved_frames_unwind] <a href="http://10.98.98.10">10.98.98.10</a>: forced unwinding frame type(1) op(STATFS)</div>
<div>[2010-03-29 16:58:26] E [saved-frames.c:165:saved_frames_unwind] <a href="http://10.98.98.10">10.98.98.10</a>: forced unwinding frame type(1) op(STATFS)</div><div>[2010-03-29 16:58:26] E [saved-frames.c:165:saved_frames_unwind] <a href="http://10.98.98.10">10.98.98.10</a>: forced unwinding frame type(1) op(STATFS)</div>
<div>[2010-03-29 16:58:26] E [saved-frames.c:165:saved_frames_unwind] <a href="http://10.98.98.10">10.98.98.10</a>: forced unwinding frame type(1) op(STATFS)</div><div>[2010-03-29 16:58:26] E [saved-frames.c:165:saved_frames_unwind] <a href="http://10.98.98.10">10.98.98.10</a>: forced unwinding frame type(1) op(STATFS)</div>
<div>[2010-03-29 16:58:26] E [saved-frames.c:165:saved_frames_unwind] <a href="http://10.98.98.10">10.98.98.10</a>: forced unwinding frame type(1) op(STATFS)</div><div>[2010-03-29 16:58:26] E [saved-frames.c:165:saved_frames_unwind] <a href="http://10.98.98.10">10.98.98.10</a>: forced unwinding frame type(1) op(STATFS)</div>
<div>[2010-03-29 16:58:26] E [saved-frames.c:165:saved_frames_unwind] <a href="http://10.98.98.10">10.98.98.10</a>: forced unwinding frame type(1) op(STATFS)</div><div>[2010-03-29 16:58:26] E [saved-frames.c:165:saved_frames_unwind] <a href="http://10.98.98.10">10.98.98.10</a>: forced unwinding frame type(1) op(STATFS)</div>
<div>[2010-03-29 16:58:26] E [saved-frames.c:165:saved_frames_unwind] <a href="http://10.98.98.10">10.98.98.10</a>: forced unwinding frame type(1) op(STATFS)</div><div>[2010-03-29 16:58:26] E [saved-frames.c:165:saved_frames_unwind] <a href="http://10.98.98.10">10.98.98.10</a>: forced unwinding frame type(2) op(PING)</div>
<div>[2010-03-29 16:58:26] D [client-protocol.c:537:client_ping_cbk] <a href="http://10.98.98.10">10.98.98.10</a>: timer must have expired</div><div>[2010-03-29 16:58:29] E [socket.c:762:socket_connect_finish] <a href="http://10.98.98.10">10.98.98.10</a>: connection to <a href="http://10.98.98.10:6996">10.98.98.10:6996</a> failed (No route to host)</div>
<div><br></div></div></div></div></div></div></div>