Hi Tomoaki,<br><br>This looks like the bug <a href="https://bugzilla.redhat.com/show_bug.cgi?id=767359">https://bugzilla.redhat.com/show_bug.cgi?id=767359</a> <br><br>This is fixed in master branch, and will be backporting the fix to 3.2.x branch soon. Thanks for reporting the issue.<br>
<br>Regards,<br>Amar<br><br><div class="gmail_quote">2012/1/30 Tomoaki Sato <span dir="ltr"><<a href="mailto:tsato@valinux.co.jp">tsato@valinux.co.jp</a>></span><br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi,<br>
<br>
3 gluster processes crashed in my scalable NAS environment.<br>
Any insight into this is much appreciated.<br>
please find attached log files & configureation files.<br>
Thanks,<br>
<br>
tomo sato<br>
--<br>
# gluster volume info<br>
<br>
Volume Name: two<br>
Type: Distribute<br>
Status: Started<br>
Number of Bricks: 2<br>
Transport-type: tcp<br>
Bricks:<br>
Brick1: two-1-private:/mnt/brick<br>
Brick2: two-2-private:/mnt/brick<br>
<br>
1/3:<br>
<br>
# strings /core.5606 | head<br>
CORE<br>
d-ahead<br>
CORE<br>
glusterfs<br>
/opt/glusterfs/3.2.5/sbin/glusterfs -f /etc/glusterd/nfs/nfs-server.vol -p /etc<br>
CORE<br>
CORE<br>
CORE<br>
CORE<br>
CORE<br>
# gdb /opt/glusterfs/3.2.5/sbin/glusterfs /core.5606<br>
GNU gdb (GDB) Red Hat Enterprise Linux (7.0.1-37.el5_7.1)<br>
Copyright (C) 2009 Free Software Foundation, Inc.<br>
License GPLv3+: GNU GPL version 3 or later <<a href="http://gnu.org/licenses/gpl.html" target="_blank">http://gnu.org/licenses/gpl.html</a>><br>
This is free software: you are free to change and redistribute it.<br>
There is NO WARRANTY, to the extent permitted by law. Type "show copying"<br>
and "show warranty" for details.<br>
This GDB was configured as "x86_64-redhat-linux-gnu".<br>
For bug reporting instructions, please see:<br>
<<a href="http://www.gnu.org/software/gdb/bugs/" target="_blank">http://www.gnu.org/software/gdb/bugs/</a>>...<br>
Reading symbols from /opt/glusterfs/3.2.5/sbin/glusterfs...(no debugging symbols found)...done.<br>
[New Thread 5627]<br>
[New Thread 5633]<br>
[New Thread 5607]<br>
[New Thread 5606]<br>
Reading symbols from /opt/glusterfs/3.2.5/lib64/libglusterfs.so.0...done.<br>
Loaded symbols for /opt/glusterfs/3.2.5/lib64/libglusterfs.so.0<br>
Reading symbols from /opt/glusterfs/3.2.5/lib64/libgfrpc.so.0...done.<br>
Loaded symbols for /opt/glusterfs/3.2.5/lib64/libgfrpc.so.0<br>
Reading symbols from /opt/glusterfs/3.2.5/lib64/libgfxdr.so.0...done.<br>
Loaded symbols for /opt/glusterfs/3.2.5/lib64/libgfxdr.so.0<br>
Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.<br>
Loaded symbols for /lib64/libdl.so.2<br>
Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done.<br>
[Thread debugging using libthread_db enabled]<br>
Loaded symbols for /lib64/libpthread.so.0<br>
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.<br>
Loaded symbols for /lib64/libc.so.6<br>
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.<br>
Loaded symbols for /lib64/ld-linux-x86-64.so.2<br>
Reading symbols from /opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/protocol/client.so...done.<br>
Loaded symbols for /opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/protocol/client.so<br>
Reading symbols from /opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/cluster/distribute.so...done.<br>
Loaded symbols for /opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/cluster/distribute.so<br>
Reading symbols from /opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/performance/write-behind.so...done.<br>
Loaded symbols for /opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/performance/write-behind.so<br>
Reading symbols from /opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/performance/read-ahead.so...done.<br>
Loaded symbols for /opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/performance/read-ahead.so<br>
Reading symbols from /opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/performance/io-cache.so...done.<br>
Loaded symbols for /opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/performance/io-cache.so<br>
Reading symbols from /opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/performance/quick-read.so...done.<br>
Loaded symbols for /opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/performance/quick-read.so<br>
Reading symbols from /opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/debug/io-stats.so...done.<br>
Loaded symbols for /opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/debug/io-stats.so<br>
Reading symbols from /opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/nfs/server.so...done.<br>
Loaded symbols for /opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/nfs/server.so<br>
Reading symbols from /opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/rpc-transport/socket.so...done.<br>
Loaded symbols for /opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/rpc-transport/socket.so<br>
Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols found)...done.<br>
Loaded symbols for /lib64/libnss_files.so.2<br>
Reading symbols from /lib64/libnss_dns.so.2...(no debugging symbols found)...done.<br>
Loaded symbols for /lib64/libnss_dns.so.2<br>
Reading symbols from /lib64/libresolv.so.2...(no debugging symbols found)...done.<br>
Loaded symbols for /lib64/libresolv.so.2<br>
Reading symbols from /lib64/libgcc_s.so.1...(no debugging symbols found)...done.<br>
Loaded symbols for /lib64/libgcc_s.so.1<br>
<br>
warning: no loadable sections found in added symbol-file system-supplied DSO at 0x7fff92bfc000<br>
Core was generated by `/opt/glusterfs/3.2.5/sbin/glusterfs -f /etc/glusterd/nfs/nfs-server.vol -p /etc'.<br>
Program terminated with signal 11, Segmentation fault.<br>
#0 ioc_writev_cbk (frame=0x2b1d5583555c, cookie=<value optimized out>, this=0x1eb230b0, op_ret=-1, op_errno=9, prebuf=0x0, postbuf=0x0) at io-cache.c:1193<br>
1193 io-cache.c: No such file or directory.<br>
in io-cache.c<br>
(gdb) bt<br>
#0 ioc_writev_cbk (frame=0x2b1d5583555c, cookie=<value optimized out>, this=0x1eb230b0, op_ret=-1, op_errno=9, prebuf=0x0, postbuf=0x0) at io-cache.c:1193<br>
#1 0x00002aaaab12a0b9 in ra_writev_cbk (frame=0x2b1d55835600, cookie=<value optimized out>, this=<value optimized out>, op_ret=-1, op_errno=9, prebuf=0x0,<br>
postbuf=0x0) at read-ahead.c:685<br>
#2 0x00002aaaaaf19096 in wb_writev_cbk (frame=0x2b1d558356a4, cookie=<value optimized out>, this=<value optimized out>, op_ret=-1, op_errno=116,<br>
prebuf=0x0, postbuf=0x0) at write-behind.c:2055<br>
#3 0x00002aaaaacf744f in dht_writev_cbk (frame=0x2b1d55835748, cookie=<value optimized out>, this=<value optimized out>, op_ret=-1, op_errno=116,<br>
prebuf=0x0, postbuf=0x0) at dht-common.c:2685<br>
#4 0x00002aaaaaac815d in client3_1_writev (frame=0x2b1d558357ec, this=0x1eb1f1b0, data=<value optimized out>) at client3_1-fops.c:3623<br>
#5 0x00002aaaaaab964b in client_writev (frame=0x2b1d558357ec, this=0x1eb1f1b0, fd=0x2aaaad1672ec, vector=0x2aaaad3c9a68, count=1, off=420155392,<br>
iobref=0x2aaab0019da0) at client.c:817<br>
#6 0x00002aaaaacfc690 in dht_writev (frame=0x2b1d55835748, this=<value optimized out>, fd=0x2aaaad1672ec, vector=0x2aaaad3c9a68, count=1, off=420155392,<br>
iobref=0x2aaab0019da0) at dht-common.c:2721<br>
#7 0x00002aaaaaf1f24a in wb_writev (frame=0x2b1d558356a4, this=0x1eb20e90, fd=0x2aaaad1672ec, vector=0x2aaaad3c9a68, count=1, offset=420155392,<br>
iobref=0x2aaab0019da0) at write-behind.c:2134<br>
#8 0x00002aaaab129e57 in ra_writev (frame=0x2b1d55835600, this=0x1eb21f60, fd=0x2aaaad1672ec, vector=0x2aaaad3c9a68, count=1, offset=420155392,<br>
iobref=0x2aaab0019da0) at read-ahead.c:718<br>
#9 0x00002aaaab3364f0 in ioc_writev (frame=0x2b1d5583555c, this=0x1eb230b0, fd=0x2aaaad1672ec, vector=0x2aaaad3c9a68, count=1, offset=420155392,<br>
iobref=0x2aaab0019da0) at io-cache.c:1237<br>
#10 0x00002aaaab54eb53 in qr_writev (frame=0x2b1d558354b8, this=0x1eb24100, fd=0x2aaaad1672ec, vector=0x2aaaad3c9a68, count=1, off=420155392,<br>
iobref=0x2aaab0019da0) at quick-read.c:1504<br>
#11 0x00002aaaab75ed0a in io_stats_writev (frame=0x2b1d55835414, this=0x1eb251d0, fd=0x2aaaad1672ec, vector=0x2aaaad3c9a68, count=1, offset=420155392,<br>
iobref=0x2aaab0019da0) at io-stats.c:1982<br>
#12 0x00002aaaab981dcc in nfs_fop_write (nfsx=<value optimized out>, xl=0x1eb251d0, nfu=<value optimized out>, fd=0x2aaaad1672ec,<br>
srciob=<value optimized out>, vector=0x2aaaad3c9a68, count=1, offset=420155392, cbk=0x2aaaab9956a0 <nfs3svc_write_cbk>, local=0x2aaaad3c9698)<br>
at nfs-fops.c:1260<br>
#13 0x00002aaaab9900c9 in __nfs3_write_resume (cs=0x2aaaad3c9698) at nfs3.c:2025<br>
#14 0x00002aaaab993d95 in nfs3_write_resume (carg=0x2b1d5583555c) at nfs3.c:2045<br>
#15 0x00002aaaab9a4004 in nfs3_file_open_and_resume (cs=0x2aaaad3c9698, resume=<value optimized out>) at nfs3-helpers.c:2136<br>
#16 0x00002aaaab993ccc in nfs3_write_open_resume (carg=0x2b1d5583555c) at nfs3.c:2072<br>
#17 0x00002aaaab99e794 in nfs3_fh_resolve_inode_done (cs=0x2aaaad3c9698, inode=<value optimized out>) at nfs3-helpers.c:2421<br>
#18 0x00002aaaab99e813 in nfs3_fh_resolve_inode (cs=0x2aaaad3c9698) at nfs3-helpers.c:3018<br>
#19 0x00002aaaab99e8e5 in nfs3_fh_resolve_resume (cs=0x2b1d5583555c) at nfs3-helpers.c:3050<br>
#20 0x00002aaaab99e9a8 in nfs3_fh_resolve_root (cs=0x2aaaad3c9698) at nfs3-helpers.c:3104<br>
#21 0x00002aaaab99b506 in nfs3_write (req=0x1ed76ee0, fh=0x40320fc0, offset=420155392, count=65536, stable=UNSTABLE, payload=..., iob=0x2aaab00808c0)<br>
at nfs3.c:2119<br>
#22 0x00002aaaab99b780 in nfs3svc_write_vec (req=0x1ed76ee0, iob=0x2aaab00808c0) at nfs3.c:2207<br>
#23 0x00002aaaab9a9e5a in nfs_rpcsvc_record_vectored_call_actor (conn=0x1eb27bec) at ../../../../xlators/nfs/lib/src/rpcsvc.c:2326<br>
#24 0x00002aaaab9aa72e in nfs_rpcsvc_update_vectored_state (conn=0x1eb27bec) at ../../../../xlators/nfs/lib/src/rpcsvc.c:2385<br>
#25 0x00002aaaab9aab18 in nfs_rpcsvc_record_update_state (conn=0x1eb27bec, dataread=52688) at ../../../../xlators/nfs/lib/src/rpcsvc.c:2481<br>
#26 0x00002aaaab9ab4ff in nfs_rpcsvc_conn_data_handler (fd=<value optimized out>, idx=514994352, data=0x1eb27bec, poll_in=1, poll_out=9, poll_err=0)<br>
at ../../../../xlators/nfs/lib/src/rpcsvc.c:2701<br>
#27 0x00002b1d5491c9a1 in event_dispatch_epoll_handler (event_pool=0x1eb41980) at event.c:794<br>
#28 event_dispatch_epoll (event_pool=0x1eb41980) at event.c:856<br>
#29 0x00002aaaab9a90c2 in nfs_rpcsvc_stage_proc (arg=<value optimized out>) at ../../../../xlators/nfs/lib/src/rpcsvc.c:64<br>
#30 0x00000035e760673d in start_thread () from /lib64/libpthread.so.0<br>
#31 0x00000035e72d40cd in clone () from /lib64/libc.so.6<br>
(gdb)<br>
<br>
2/3:<br>
<br>
# strings /core.5629 | head -20<br>
;Z3+<br>
0;Z3+<br>
p<Z3+<br>
p\Z3+<br>
\Z3+<br>
CORE<br>
\Z3+<br>
0E_a<br>
\Z3+<br>
CORE<br>
glusterd<br>
/opt/glusterfs/3.2.5/sbin/glusterd<br>
CORE<br>
CORE<br>
CORE<br>
\Z3+<br>
CORE<br>
CORE<br>
\Z3+<br>
CORE<br>
# gdb /opt/glusterfs/3.2.5/sbin/glusterd /core.5629<br>
GNU gdb (GDB) Red Hat Enterprise Linux (7.0.1-37.el5_7.1)<br>
Copyright (C) 2009 Free Software Foundation, Inc.<br>
License GPLv3+: GNU GPL version 3 or later <<a href="http://gnu.org/licenses/gpl.html" target="_blank">http://gnu.org/licenses/gpl.html</a>><br>
This is free software: you are free to change and redistribute it.<br>
There is NO WARRANTY, to the extent permitted by law. Type "show copying"<br>
and "show warranty" for details.<br>
This GDB was configured as "x86_64-redhat-linux-gnu".<br>
For bug reporting instructions, please see:<br>
<<a href="http://www.gnu.org/software/gdb/bugs/" target="_blank">http://www.gnu.org/software/gdb/bugs/</a>>...<br>
Reading symbols from /opt/glusterfs/3.2.5/sbin/glusterd...(no debugging symbols found)...done.<br>
[New Thread 5629]<br>
[New Thread 5638]<br>
[New Thread 5630]<br>
Reading symbols from /opt/glusterfs/3.2.5/lib64/libglusterfs.so.0...done.<br>
Loaded symbols for /opt/glusterfs/3.2.5/lib64/libglusterfs.so.0<br>
Reading symbols from /opt/glusterfs/3.2.5/lib64/libgfrpc.so.0...done.<br>
Loaded symbols for /opt/glusterfs/3.2.5/lib64/libgfrpc.so.0<br>
Reading symbols from /opt/glusterfs/3.2.5/lib64/libgfxdr.so.0...done.<br>
Loaded symbols for /opt/glusterfs/3.2.5/lib64/libgfxdr.so.0<br>
Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.<br>
Loaded symbols for /lib64/libdl.so.2<br>
Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done.<br>
[Thread debugging using libthread_db enabled]<br>
Loaded symbols for /lib64/libpthread.so.0<br>
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.<br>
Loaded symbols for /lib64/libc.so.6<br>
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.<br>
Loaded symbols for /lib64/ld-linux-x86-64.so.2<br>
Reading symbols from /opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/mgmt/glusterd.so...done.<br>
Loaded symbols for /opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/mgmt/glusterd.so<br>
Reading symbols from /opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/rpc-transport/socket.so...done.<br>
Loaded symbols for /opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/rpc-transport/socket.so<br>
Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols found)...done.<br>
Loaded symbols for /lib64/libnss_files.so.2<br>
Reading symbols from /lib64/libnss_dns.so.2...(no debugging symbols found)...done.<br>
Loaded symbols for /lib64/libnss_dns.so.2<br>
Reading symbols from /lib64/libresolv.so.2...(no debugging symbols found)...done.<br>
Loaded symbols for /lib64/libresolv.so.2<br>
Reading symbols from /lib64/libgcc_s.so.1...(no debugging symbols found)...done.<br>
Loaded symbols for /lib64/libgcc_s.so.1<br>
<br>
warning: no loadable sections found in added symbol-file system-supplied DSO at 0x7fff617f5000<br>
Core was generated by `/opt/glusterfs/3.2.5/sbin/glusterd'.<br>
Program terminated with signal 11, Segmentation fault.<br>
#0 glusterd_friend_sm () at glusterd-sm.c:934<br>
934 glusterd-sm.c: No such file or directory.<br>
in glusterd-sm.c<br>
(gdb) bt<br>
#0 glusterd_friend_sm () at glusterd-sm.c:934<br>
#1 0x00002aaaaaac7a04 in glusterd_peer_rpc_notify (rpc=0x519dc50, mydata=0x519e4b0, event=RPC_CLNT_DISCONNECT, data=<value optimized out>)<br>
at glusterd-handler.c:3883<br>
#2 0x00002b335a1ab8a7 in rpc_clnt_start (rpc=<value optimized out>) at rpc-clnt.c:1022<br>
#3 0x00002aaaaaac874f in glusterd_rpc_create (rpc=0x7fff615f4630, options=<value optimized out>, notify_fn=0x2aaaaaac7850 <glusterd_peer_rpc_notify>,<br>
notify_data=0x519e4b0) at glusterd-handler.c:3279<br>
#4 0x00002aaaaaac8ad7 in glusterd_friend_add (hoststr=0x519e420 "two-2-private", port=0, state=GD_FRIEND_STATE_BEFRIENDED, uuid=<value optimized out>,<br>
rpc=0x519dc50, friend=0x7fff615f6710, restore=_gf_true, args=0x7fff615f66d0) at glusterd-handler.c:3378<br>
#5 0x00002aaaaab0d4c2 in glusterd_store_retrieve_peers (this=<value optimized out>) at glusterd-store.c:2248<br>
#6 0x00002aaaaab0edd8 in glusterd_restore () at glusterd-store.c:2310<br>
#7 0x00002aaaaaac4312 in init (this=0x5190c20) at glusterd.c:695<br>
#8 0x00002b3359f46488 in __xlator_init (xl=0x5190c20) at xlator.c:1418<br>
#9 xlator_init (xl=0x5190c20) at xlator.c:1441<br>
#10 0x00002b3359f75501 in glusterfs_graph_init (graph=<value optimized out>) at graph.c:345<br>
#11 0x00002b3359f75688 in glusterfs_graph_activate (graph=0x518c900, ctx=0x518b010) at graph.c:524<br>
#12 0x0000000000404033 in glusterfs_process_volfp ()<br>
#13 0x000000000040424b in glusterfs_volumes_init ()<br>
#14 0x0000000000405299 in main ()<br>
(gdb)<br>
<br>
3/3:<br>
<br>
# strings /core.6124 | head<br>
CORE<br>
d-ahead<br>
CORE<br>
glusterfs<br>
/opt/glusterfs/3.2.5/sbin/glusterfs -f /etc/glusterd/nfs/nfs-server.vol -p /etc<br>
CORE<br>
CORE<br>
CORE<br>
PZm<br>
CORE<br>
# gdb /opt/glusterfs/3.2.5/sbin/glusterfs /core.6124<br>
GNU gdb (GDB) Red Hat Enterprise Linux (7.0.1-37.el5_7.1)<br>
Copyright (C) 2009 Free Software Foundation, Inc.<br>
License GPLv3+: GNU GPL version 3 or later <<a href="http://gnu.org/licenses/gpl.html" target="_blank">http://gnu.org/licenses/gpl.html</a>><br>
This is free software: you are free to change and redistribute it.<br>
There is NO WARRANTY, to the extent permitted by law. Type "show copying"<br>
and "show warranty" for details.<br>
This GDB was configured as "x86_64-redhat-linux-gnu".<br>
For bug reporting instructions, please see:<br>
<<a href="http://www.gnu.org/software/gdb/bugs/" target="_blank">http://www.gnu.org/software/gdb/bugs/</a>>...<br>
Reading symbols from /opt/glusterfs/3.2.5/sbin/glusterfs...(no debugging symbols found)...done.<br>
[New Thread 6173]<br>
[New Thread 6230]<br>
[New Thread 6125]<br>
[New Thread 6124]<br>
Reading symbols from /opt/glusterfs/3.2.5/lib64/libglusterfs.so.0...done.<br>
Loaded symbols for /opt/glusterfs/3.2.5/lib64/libglusterfs.so.0<br>
Reading symbols from /opt/glusterfs/3.2.5/lib64/libgfrpc.so.0...done.<br>
Loaded symbols for /opt/glusterfs/3.2.5/lib64/libgfrpc.so.0<br>
Reading symbols from /opt/glusterfs/3.2.5/lib64/libgfxdr.so.0...done.<br>
Loaded symbols for /opt/glusterfs/3.2.5/lib64/libgfxdr.so.0<br>
Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.<br>
Loaded symbols for /lib64/libdl.so.2<br>
Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done.<br>
[Thread debugging using libthread_db enabled]<br>
Loaded symbols for /lib64/libpthread.so.0<br>
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.<br>
Loaded symbols for /lib64/libc.so.6<br>
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.<br>
Loaded symbols for /lib64/ld-linux-x86-64.so.2<br>
Reading symbols from /opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/protocol/client.so...done.<br>
Loaded symbols for /opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/protocol/client.so<br>
Reading symbols from /opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/cluster/distribute.so...done.<br>
Loaded symbols for /opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/cluster/distribute.so<br>
Reading symbols from /opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/performance/write-behind.so...done.<br>
Loaded symbols for /opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/performance/write-behind.so<br>
Reading symbols from /opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/performance/read-ahead.so...done.<br>
Loaded symbols for /opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/performance/read-ahead.so<br>
Reading symbols from /opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/performance/io-cache.so...done.<br>
Loaded symbols for /opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/performance/io-cache.so<br>
Reading symbols from /opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/performance/quick-read.so...done.<br>
Loaded symbols for /opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/performance/quick-read.so<br>
Reading symbols from /opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/debug/io-stats.so...done.<br>
Loaded symbols for /opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/debug/io-stats.so<br>
Reading symbols from /opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/nfs/server.so...done.<br>
Loaded symbols for /opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/nfs/server.so<br>
Reading symbols from /opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/rpc-transport/socket.so...done.<br>
Loaded symbols for /opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/rpc-transport/socket.so<br>
Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols found)...done.<br>
Loaded symbols for /lib64/libnss_files.so.2<br>
Reading symbols from /lib64/libnss_dns.so.2...(no debugging symbols found)...done.<br>
Loaded symbols for /lib64/libnss_dns.so.2<br>
Reading symbols from /lib64/libresolv.so.2...(no debugging symbols found)...done.<br>
Loaded symbols for /lib64/libresolv.so.2<br>
Reading symbols from /lib64/libgcc_s.so.1...(no debugging symbols found)...done.<br>
Loaded symbols for /lib64/libgcc_s.so.1<br>
<br>
warning: no loadable sections found in added symbol-file system-supplied DSO at 0x7fffca144000<br>
Core was generated by `/opt/glusterfs/3.2.5/sbin/glusterfs -f /etc/glusterd/nfs/nfs-server.vol -p /etc'.<br>
Program terminated with signal 11, Segmentation fault.<br>
#0 ioc_writev_cbk (frame=0x2b299b7b477c, cookie=<value optimized out>, this=0x968e0b0, op_ret=-1, op_errno=9, prebuf=0x0, postbuf=0x0) at io-cache.c:1193<br>
1193 io-cache.c: No such file or directory.<br>
in io-cache.c<br>
(gdb) bt<br>
#0 ioc_writev_cbk (frame=0x2b299b7b477c, cookie=<value optimized out>, this=0x968e0b0, op_ret=-1, op_errno=9, prebuf=0x0, postbuf=0x0) at io-cache.c:1193<br>
#1 0x00002aaaab12a0b9 in ra_writev_cbk (frame=0x2b299b7b4820, cookie=<value optimized out>, this=<value optimized out>, op_ret=-1, op_errno=9, prebuf=0x0,<br>
postbuf=0x0) at read-ahead.c:685<br>
#2 0x00002aaaaaf19096 in wb_writev_cbk (frame=0x2b299b7b48c4, cookie=<value optimized out>, this=<value optimized out>, op_ret=-1, op_errno=116,<br>
prebuf=0x0, postbuf=0x0) at write-behind.c:2055<br>
#3 0x00002aaaaacf744f in dht_writev_cbk (frame=0x2b299b7b4968, cookie=<value optimized out>, this=<value optimized out>, op_ret=-1, op_errno=116,<br>
prebuf=0x0, postbuf=0x0) at dht-common.c:2685<br>
#4 0x00002aaaaaac815d in client3_1_writev (frame=0x2b299b7b4a0c, this=0x968a1b0, data=<value optimized out>) at client3_1-fops.c:3623<br>
#5 0x00002aaaaaab964b in client_writev (frame=0x2b299b7b4a0c, this=0x968a1b0, fd=0x2aaaad166a64, vector=0x2aaab431a970, count=1, off=472158208,<br>
iobref=0x2aaab4316420) at client.c:817<br>
#6 0x00002aaaaacfc690 in dht_writev (frame=0x2b299b7b4968, this=<value optimized out>, fd=0x2aaaad166a64, vector=0x2aaab431a970, count=1, off=472158208,<br>
iobref=0x2aaab4316420) at dht-common.c:2721<br>
#7 0x00002aaaaaf1f24a in wb_writev (frame=0x2b299b7b48c4, this=0x968be90, fd=0x2aaaad166a64, vector=0x2aaab431a970, count=1, offset=472158208,<br>
iobref=0x2aaab4316420) at write-behind.c:2134<br>
#8 0x00002aaaab129e57 in ra_writev (frame=0x2b299b7b4820, this=0x968cf60, fd=0x2aaaad166a64, vector=0x2aaab431a970, count=1, offset=472158208,<br>
iobref=0x2aaab4316420) at read-ahead.c:718<br>
#9 0x00002aaaab3364f0 in ioc_writev (frame=0x2b299b7b477c, this=0x968e0b0, fd=0x2aaaad166a64, vector=0x2aaab431a970, count=1, offset=472158208,<br>
iobref=0x2aaab4316420) at io-cache.c:1237<br>
#10 0x00002aaaab54eb53 in qr_writev (frame=0x2b299b7b46d8, this=0x968f100, fd=0x2aaaad166a64, vector=0x2aaab431a970, count=1, off=472158208,<br>
iobref=0x2aaab4316420) at quick-read.c:1504<br>
#11 0x00002aaaab75ed0a in io_stats_writev (frame=0x2b299b7b4634, this=0x96901d0, fd=0x2aaaad166a64, vector=0x2aaab431a970, count=1, offset=472158208,<br>
iobref=0x2aaab4316420) at io-stats.c:1982<br>
#12 0x00002aaaab981dcc in nfs_fop_write (nfsx=<value optimized out>, xl=0x96901d0, nfu=<value optimized out>, fd=0x2aaaad166a64,<br>
srciob=<value optimized out>, vector=0x2aaab431a970, count=1, offset=472158208, cbk=0x2aaaab9956a0 <nfs3svc_write_cbk>, local=0x2aaab431a5a0)<br>
at nfs-fops.c:1260<br>
#13 0x00002aaaab9900c9 in __nfs3_write_resume (cs=0x2aaab431a5a0) at nfs3.c:2025<br>
#14 0x00002aaaab993d95 in nfs3_write_resume (carg=0x2b299b7b477c) at nfs3.c:2045<br>
#15 0x00002aaaab9a4004 in nfs3_file_open_and_resume (cs=0x2aaab431a5a0, resume=<value optimized out>) at nfs3-helpers.c:2136<br>
#16 0x00002aaaab993ccc in nfs3_write_open_resume (carg=0x2b299b7b477c) at nfs3.c:2072<br>
#17 0x00002aaaab99e794 in nfs3_fh_resolve_inode_done (cs=0x2aaab431a5a0, inode=<value optimized out>) at nfs3-helpers.c:2421<br>
#18 0x00002aaaab99e813 in nfs3_fh_resolve_inode (cs=0x2aaab431a5a0) at nfs3-helpers.c:3018<br>
#19 0x00002aaaab99e8e5 in nfs3_fh_resolve_resume (cs=0x2b299b7b477c) at nfs3-helpers.c:3050<br>
#20 0x00002aaaab99e9a8 in nfs3_fh_resolve_root (cs=0x2aaab431a5a0) at nfs3-helpers.c:3104<br>
#21 0x00002aaaab99b506 in nfs3_write (req=0x2aaab41dbb34, fh=0x4042cfc0, offset=472158208, count=65536, stable=UNSTABLE, payload=..., iob=0x2aaab42b7f50)<br>
at nfs3.c:2119<br>
#22 0x00002aaaab99b780 in nfs3svc_write_vec (req=0x2aaab41dbb34, iob=0x2aaab42b7f50) at nfs3.c:2207<br>
#23 0x00002aaaab9a9e5a in nfs_rpcsvc_record_vectored_call_actor (conn=0x9692cb8) at ../../../../xlators/nfs/lib/src/rpcsvc.c:2326<br>
#24 0x00002aaaab9aa72e in nfs_rpcsvc_update_vectored_state (conn=0x9692cb8) at ../../../../xlators/nfs/lib/src/rpcsvc.c:2385<br>
#25 0x00002aaaab9aab18 in nfs_rpcsvc_record_update_state (conn=0x9692cb8, dataread=6352) at ../../../../xlators/nfs/lib/src/rpcsvc.c:2481<br>
#26 0x00002aaaab9ab4ff in nfs_rpcsvc_conn_data_handler (fd=<value optimized out>, idx=157868208, data=0x9692cb8, poll_in=1, poll_out=9, poll_err=0)<br>
at ../../../../xlators/nfs/lib/src/rpcsvc.c:2701<br>
#27 0x00002b299a8659a1 in event_dispatch_epoll_handler (event_pool=0x96ac980) at event.c:794<br>
#28 event_dispatch_epoll (event_pool=0x96ac980) at event.c:856<br>
#29 0x00002aaaab9a90c2 in nfs_rpcsvc_stage_proc (arg=<value optimized out>) at ../../../../xlators/nfs/lib/src/rpcsvc.c:64<br>
#30 0x00000035e760673d in start_thread () from /lib64/libpthread.so.0<br>
#31 0x00000035e72d40cd in clone () from /lib64/libc.so.6<br>
(gdb)<br>
<br>
<br>_______________________________________________<br>
Gluster-devel mailing list<br>
<a href="mailto:Gluster-devel@nongnu.org">Gluster-devel@nongnu.org</a><br>
<a href="https://lists.nongnu.org/mailman/listinfo/gluster-devel" target="_blank">https://lists.nongnu.org/mailman/listinfo/gluster-devel</a><br>
<br></blockquote></div><br>