<div dir="ltr">Apparently the problem isn't fixed... even when qemu doesn't crash, the guest raises many I/O error and turns unusable, just like a real machine would do if you physically remove the hard drive, I guess...<div>
<br></div><div>I'm doing more tests anyway and will post a much more detailed report as soon as I can. Thanks for now.</div><div><br></div><div>Guido</div></div><div class="gmail_extra"><br><br><div class="gmail_quote">
2013/7/28 Anand Avati <span dir="ltr"><<a href="mailto:anand.avati@gmail.com" target="_blank">anand.avati@gmail.com</a>></span><br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">Guido,<div>You need a couple of fixes:</div><div><br></div><div><a href="http://review.gluster.org/5378" target="_blank">http://review.gluster.org/5378</a> - to fix the root cause of the failure (portmap failure)</div>
<div>
<a href="http://review.gluster.org/5407" target="_blank">http://review.gluster.org/5407</a> - to prevent a crash in case of such failures.</div><div><br></div><div>Can you please apply the patches and confirm if they fix your issue?</div>
<div><br></div><div>Thanks,</div><div>Avati</div></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><br><div class="gmail_quote">On Fri, Jul 26, 2013 at 4:02 PM, Guido De Rosa <span dir="ltr"><<a href="mailto:guido.derosa@vemarsas.it" target="_blank">guido.derosa@vemarsas.it</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">2013/7/26 Anand Avati <<a href="mailto:anand.avati@gmail.com" target="_blank">anand.avati@gmail.com</a>>:<br>
<div>> Can you please post the backtrace and logs from the crash?<br>
<br>
</div># gdb<br>
...<br>
(gdb) file /usr/local/bin/qemu-system-x86_64<br>
Reading symbols from /usr/local/bin/qemu-system-x86_64...done.<br>
(gdb) run -uuid "e06dc280-d74a-0130-49e3-003018a4d17c" -name<br>
"deb-on-gluster" -m "2048" -vnc ":1" -k "it" -pidfile<br>
"/var/run/onboard/qemu-e06dc280.pid" -monitor<br>
unix:"/var/run/onboard/qemu-e06dc280.sock",server,nowait -smp 2<br>
-device piix3-usb-uhci,id=piix3-uhci -device usb-ehci,id=ehci -device<br>
usb-tablet,bus=piix3-uhci.0 -drive<br>
serial="QME06DC28000",if="virtio",media="disk",cache="unsafe",file="gluster://localhost/gvtest/QEMU/deb-on-gluster/disk0.qcow2",index=0<br>
-drive serial="QME06DC28001",if="ide",media="cdrom",bus=1,unit=0 -net<br>
nic,vlan=0,model=virtio,macaddr=DE:AD:BE:6C:CF:30 -net<br>
tap,vlan=0,ifname=vDebOnGluster_0,script=no,downscript=no -boot<br>
menu=on,order=dc -cpu host -enable-kvm -runas onboard<br>
<br>
...<br>
<br>
On another terminal:<br>
<br>
# gluster volume add-brick gvtest replica 2<br>
192.168.232.101:/var/export/gluster/gvtest<br>
volume add-brick: success<br>
<br>
And here the crash:<br>
<br>
[Thread debugging using libthread_db enabled]<br>
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".<br>
[New Thread 0x7fffe5dad700 (LWP 8611)]<br>
[New Thread 0x7fffe55ac700 (LWP 8618)]<br>
[New Thread 0x7fffe4548700 (LWP 8621)]<br>
[New Thread 0x7fffe3b33700 (LWP 8622)]<br>
[New Thread 0x7fffdbdfd700 (LWP 8623)]<br>
[New Thread 0x7fffdb5fc700 (LWP 8624)]<br>
[New Thread 0x7fffd9dff700 (LWP 8626)]<br>
[New Thread 0x7fffc7fff700 (LWP 8627)]<br>
[New Thread 0x7fffc63fe700 (LWP 8628)]<br>
<br>
Program received signal SIGSEGV, Segmentation fault.<br>
[Switching to Thread 0x7fffe5dad700 (LWP 8611)]<br>
glfs_subvol_done (fs=0x5555566cfc20, subvol=subvol@entry=0x0) at<br>
glfs-resolve.c:802<br>
802 glfs-resolve.c: No such file or directory.<br>
(gdb)<br>
(gdb) bt<br>
#0 glfs_subvol_done (fs=0x5555566cfc20, subvol=subvol@entry=0x0) at<br>
glfs-resolve.c:802<br>
#1 0x00007ffff70aff50 in glfs_pwritev (glfd=0x5555566d79b0,<br>
iovec=<optimized out>,<br>
iovcnt=<optimized out>, offset=8368975872, flags=0) at glfs-fops.c:761<br>
#2 0x00007ffff70b04e7 in glfs_io_async_task (data=<optimized out>) at<br>
glfs-fops.c:584<br>
#3 0x00007ffff0080b22 in synctask_wrap (old_task=<optimized out>) at<br>
syncop.c:131<br>
#4 0x00007ffff0541710 in ?? () from /lib/x86_64-linux-gnu/libc.so.6<br>
#5 0x0000000000000000 in ?? ()<br>
<br>
<br>
Log: /var/log/glusterfs/etc-glusterfs-glusterd.vol.log<br>
<br>
[2013-07-26 22:27:45.877325] I<br>
[glusterd-brick-ops.c:370:__glusterd_handle_add_brick] 0-management:<br>
Received add brick req<br>
[2013-07-26 22:27:45.877398] I<br>
[glusterd-brick-ops.c:417:__glusterd_handle_add_brick] 0-management:<br>
repl<br>
ica-count is 2<br>
[2013-07-26 22:27:45.877415] I<br>
[glusterd-brick-ops.c:193:gd_addbr_validate_replica_count]<br>
0-management:<br>
Changing the type of volume gvtest from 'distribute' to 'replica'<br>
[2013-07-26 22:27:45.879877] I<br>
[glusterd-brick-ops.c:894:glusterd_op_perform_add_bricks]<br>
0-management: r<br>
eplica-count is set 2<br>
[2013-07-26 22:27:45.879907] I<br>
[glusterd-brick-ops.c:898:glusterd_op_perform_add_bricks]<br>
0-management: t<br>
ype is set 2, need to change it<br>
[2013-07-26 22:27:45.882314] I<br>
[glusterd-utils.c:954:glusterd_volume_brickinfo_get] 0-management:<br>
Found<br>
brick<br>
[2013-07-26 22:27:45.882349] I<br>
[glusterd-utils.c:954:glusterd_volume_brickinfo_get] 0-management:<br>
Found<br>
brick<br>
[2013-07-26 22:27:45.883952] I<br>
[glusterd-utils.c:954:glusterd_volume_brickinfo_get] 0-management:<br>
Found<br>
brick<br>
[2013-07-26 22:27:45.883978] I<br>
[glusterd-utils.c:954:glusterd_volume_brickinfo_get] 0-management:<br>
Found<br>
brick<br>
[2013-07-26 22:27:45.884704] I<br>
[glusterd-utils.c:954:glusterd_volume_brickinfo_get] 0-management:<br>
Found<br>
brick<br>
[2013-07-26 22:27:46.617898] I<br>
[glusterd-utils.c:954:glusterd_volume_brickinfo_get] 0-management:<br>
Found<br>
brick<br>
[2013-07-26 22:27:46.617945] I<br>
[glusterd-utils.c:954:glusterd_volume_brickinfo_get] 0-management:<br>
Found<br>
brick<br>
[2013-07-26 22:27:47.618736] E<br>
[glusterd-utils.c:3627:glusterd_nodesvc_unlink_socket_file]<br>
0-management:<br>
[2013-07-26 22:27:45.877325] I<br>
[glusterd-brick-ops.c:370:__glusterd_handle_add_brick] 0-management:<br>
Rece<br>
ived add brick req<br>
[2013-07-26 22:27:45.877398] I<br>
[glusterd-brick-ops.c:417:__glusterd_handle_add_brick] 0-management:<br>
repl<br>
ica-count is 2<br>
[2013-07-26 22:27:45.877415] I<br>
[glusterd-brick-ops.c:193:gd_addbr_validate_replica_count]<br>
0-management:<br>
Changing the type of volume gvtest from 'distribute' to 'replica'<br>
[2013-07-26 22:27:45.879877] I<br>
[glusterd-brick-ops.c:894:glusterd_op_perform_add_bricks]<br>
0-management: r<br>
eplica-count is set 2<br>
[2013-07-26 22:27:45.879907] I<br>
[glusterd-brick-ops.c:898:glusterd_op_perform_add_bricks]<br>
0-management: t<br>
ype is set 2, need to change it<br>
[2013-07-26 22:27:45.882314] I<br>
[glusterd-utils.c:954:glusterd_volume_brickinfo_get] 0-management:<br>
Found<br>
brick<br>
[2013-07-26 22:27:45.882349] I<br>
[glusterd-utils.c:954:glusterd_volume_brickinfo_get] 0-management:<br>
Found<br>
brick<br>
[2013-07-26 22:27:45.883952] I<br>
[glusterd-utils.c:954:glusterd_volume_brickinfo_get] 0-management:<br>
Found<br>
brick<br>
[2013-07-26 22:27:45.883978] I<br>
[glusterd-utils.c:954:glusterd_volume_brickinfo_get] 0-management:<br>
Found<br>
brick<br>
[2013-07-26 22:27:45.884704] I<br>
[glusterd-utils.c:954:glusterd_volume_brickinfo_get] 0-management:<br>
Found<br>
brick<br>
[2013-07-26 22:27:46.617898] I<br>
[glusterd-utils.c:954:glusterd_volume_brickinfo_get] 0-management:<br>
Found<br>
brick<br>
[2013-07-26 22:27:46.617945] I<br>
[glusterd-utils.c:954:glusterd_volume_brickinfo_get] 0-management:<br>
Found<br>
brick<br>
[2013-07-26 22:27:47.618736] E<br>
[glusterd-utils.c:3627:glusterd_nodesvc_unlink_socket_file]<br>
0-management:<br>
[2013-07-26 22:27:47.619092] I<br>
[glusterd-utils.c:3661:glusterd_nfs_pmap_deregister] 0-: De-registered<br>
MOUNTV3 successfully<br>
[2013-07-26 22:27:47.619386] I<br>
[glusterd-utils.c:3666:glusterd_nfs_pmap_deregister] 0-: De-registered<br>
MOUNTV1 successfully<br>
[2013-07-26 22:27:47.619682] I<br>
[glusterd-utils.c:3671:glusterd_nfs_pmap_deregister] 0-: De-registered<br>
NFSV3 successfully<br>
[2013-07-26 22:27:47.620004] I<br>
[glusterd-utils.c:3676:glusterd_nfs_pmap_deregister] 0-: De-registered<br>
NLM v4 successfully<br>
[2013-07-26 22:27:47.620343] I<br>
[glusterd-utils.c:3681:glusterd_nfs_pmap_deregister] 0-: De-registered<br>
NLM v1 successfully<br>
[2013-07-26 22:27:47.622905] I<br>
[rpc-clnt.c:962:rpc_clnt_connection_init] 0-management: setting<br>
frame-timeout to 600<br>
[2013-07-26 22:27:47.622988] I [socket.c:3480:socket_init]<br>
0-management: SSL support is NOT enabled<br>
[2013-07-26 22:27:47.623003] I [socket.c:3495:socket_init]<br>
0-management: using system polling thread<br>
[2013-07-26 22:27:48.650339] E<br>
[glusterd-utils.c:3627:glusterd_nodesvc_unlink_socket_file]<br>
0-management: Failed to remove<br>
/var/run/52b6ed075a07af2e9235e49dd9d214af.socke<br>
t error: No such file or directory<br>
[2013-07-26 22:27:48.652919] I<br>
[rpc-clnt.c:962:rpc_clnt_connection_init] 0-management: setting<br>
frame-timeout to 600<br>
[2013-07-26 22:27:48.653002] I [socket.c:3480:socket_init]<br>
0-management: SSL support is NOT enabled<br>
[2013-07-26 22:27:48.653017] I [socket.c:3495:socket_init]<br>
0-management: using system polling thread<br>
[2013-07-26 22:27:48.653215] W [socket.c:514:__socket_rwv]<br>
0-management: readv failed (No data available)<br>
[2013-07-26 22:27:48.653333] I [mem-pool.c:541:mem_pool_destroy]<br>
0-management: size=2236 max=0 total=0<br>
[2013-07-26 22:27:48.653354] I [mem-pool.c:541:mem_pool_destroy]<br>
0-management: size=124 max=0 total=0<br>
[2013-07-26 22:27:48.653398] I [socket.c:2236:socket_event_handler]<br>
0-transport: disconnecting now<br>
[2013-07-26 22:27:48.653453] W [socket.c:514:__socket_rwv]<br>
0-management: readv failed (No data available)<br>
[2013-07-26 22:27:48.653478] I [mem-pool.c:541:mem_pool_destroy]<br>
0-management: size=2236 max=0 total=0<br>
[2013-07-26 22:27:48.653493] I [mem-pool.c:541:mem_pool_destroy]<br>
0-management: size=124 max=0 total=0<br>
[2013-07-26 22:27:48.653537] I [socket.c:2236:socket_event_handler]<br>
0-transport: disconnecting now<br>
[2013-07-26 22:27:48.680240] E [rpcsvc.c:519:rpcsvc_handle_rpc_call]<br>
0-glusterd: Request received from non-privileged port. Failing request<br>
[2013-07-26 22:27:52.093567] E [rpcsvc.c:519:rpcsvc_handle_rpc_call]<br>
0-glusterd: Request received from non-privileged port. Failing request<br>
[2013-07-26 22:27:55.094469] E [rpcsvc.c:519:rpcsvc_handle_rpc_call]<br>
0-glusterd: Request received from non-privileged port. Failing request<br>
<br>
--<br>
<br>
G.<br>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div>