While this fixes the double free, The actual fix has to copy the buffer into an ioq_entry, instead of just storing the buffer pointer. If not, there can be cases wherein by the time the ioq_entry is written to socket, the buffer might've already been freed.<br>
<br><div class="gmail_quote">On Thu, Jun 18, 2009 at 2:36 PM, Shehjar Tikoo <span dir="ltr"><<a href="mailto:shehjart@gluster.com">shehjart@gluster.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
I think I understand why you see the crash.<br>
Could you please apply the following patch and tell<br>
us if the crash is observed still?<br>
<br>
Thanks<br><font color="#888888">
Shehjar</font><div><div></div><div class="h5"><br>
<br>
<br>
<br>
Mihai wrote:<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
Hello,<br>
I'm using a server side replication on 6 servers. Glusterfsd crashes on a few hour basis:<br>
gdb -se /usr/sbin/glusterfsd -c /core.26947 GNU gdb Fedora (6.8-27.el5) Copyright (C) 2008 Free Software Foundation, Inc.<br>
License GPLv3+: GNU GPL version 3 or later <<a href="http://gnu.org/licenses/gpl.html" target="_blank">http://gnu.org/licenses/gpl.html</a>><br>
This is free software: you are free to change and redistribute it.<br>
There is NO WARRANTY, to the extent permitted by law. Type "show copying"<br>
and "show warranty" for details.<br>
This GDB was configured as "x86_64-redhat-linux-gnu"...<br>
(no debugging symbols found)<br>
<br>
warning: .dynamic section for "/usr/lib64/libglusterfs.so.0" is not at the expected address<br>
<br>
warning: difference appears to be caused by prelink, adjusting expectations Reading symbols from /usr/lib64/libglusterfs.so.0...done.<br>
Loaded symbols for /usr/lib64/libglusterfs.so.0 Reading symbols from /lib64/libdl.so.2...done.<br>
Loaded symbols for /lib64/libdl.so.2<br>
Reading symbols from /lib64/libpthread.so.0...done.<br>
Loaded symbols for /lib64/libpthread.so.0 Reading symbols from /lib64/libc.so.6...done.<br>
Loaded symbols for /lib64/libc.so.6<br>
Reading symbols from /lib64/ld-linux-x86-64.so.2...done.<br>
Loaded symbols for /lib64/ld-linux-x86-64.so.2 Reading symbols from /usr/lib64/glusterfs/2.0.2/xlator/storage/posix.so...done.<br>
Loaded symbols for /usr/lib64/glusterfs/2.0.2/xlator/storage/posix.so<br>
Reading symbols from /usr/lib64/glusterfs/2.0.2/xlator/features/locks.so...done.<br>
Loaded symbols for /usr/lib64/glusterfs/2.0.2/xlator/features/locks.so<br>
Reading symbols from /usr/lib64/glusterfs/2.0.2/xlator/performance/io-threads.so...done.<br>
Loaded symbols for /usr/lib64/glusterfs/2.0.2/xlator/performance/io-threads.so<br>
Reading symbols from /usr/lib64/glusterfs/2.0.2/xlator/protocol/client.so...done.<br>
Loaded symbols for /usr/lib64/glusterfs/2.0.2/xlator/protocol/client.so<br>
Reading symbols from /usr/lib64/glusterfs/2.0.2/xlator/cluster/replicate.so...done.<br>
Loaded symbols for /usr/lib64/glusterfs/2.0.2/xlator/cluster/replicate.so<br>
Reading symbols from /usr/lib64/glusterfs/2.0.2/xlator/protocol/server.so...done.<br>
Loaded symbols for /usr/lib64/glusterfs/2.0.2/xlator/protocol/server.so<br>
Reading symbols from /usr/lib64/glusterfs/2.0.2/transport/socket.so...done.<br>
Loaded symbols for /usr/lib64/glusterfs/2.0.2/transport/socket.so<br>
Reading symbols from /usr/lib64/glusterfs/2.0.2/auth/addr.so...done.<br>
Loaded symbols for /usr/lib64/glusterfs/2.0.2/auth/addr.so<br>
Reading symbols from /lib64/libnss_files.so.2...done.<br>
Loaded symbols for /lib64/libnss_files.so.2 Reading symbols from /lib64/libgcc_s.so.1...done.<br>
Loaded symbols for /lib64/libgcc_s.so.1<br>
Core was generated by `/usr/sbin/glusterfsd -f /etc/glusterfs/glusterfsd.vol'.<br>
Program terminated with signal 6, Aborted.<br>
[New process 26947]<br>
[New process 26956]<br>
[New process 26955]<br>
[New process 26954]<br>
[New process 26953]<br>
[New process 26952]<br>
[New process 26951]<br>
[New process 26950]<br>
[New process 26949]<br>
[New process 26948]<br>
#0 0x0000003040030215 in raise () from /lib64/libc.so.6<br>
(gdb) bt<br>
#0 0x0000003040030215 in raise () from /lib64/libc.so.6<br>
#1 0x0000003040031cc0 in abort () from /lib64/libc.so.6<br>
#2 0x000000304006a7fb in __libc_message () from /lib64/libc.so.6<br>
#3 0x0000003040071ce2 in _int_free () from /lib64/libc.so.6<br>
#4 0x000000304007590c in free () from /lib64/libc.so.6<br>
#5 0x00002aaaaaaadcc9 in __socket_ioq_entry_free (entry=0x2aaab001da30) at socket.c:331<br>
#6 0x00002aaaaaaaf1c9 in __socket_ioq_churn_entry (this=<value optimized out>, entry=0x2aaab001da30) at socket.c:368<br>
#7 0x00002aaaaaaaf8ec in socket_submit (this=0xae11a70, buf=0x2aaab00378c0 "", len=340, vector=0x0, count=<value optimized out>,<br>
iobref=<value optimized out>) at socket.c:1281<br>
#8 0x00002b2e7c775bd3 in protocol_client_xfer (frame=0x2aaab0030ab0, this=0xae0ab00, trans=0xae11a70, type=1, op=40, hdr=0x2aaab00378c0, hdrlen=340,<br>
vector=0x0, count=0, iobref=0x0) at client-protocol.c:636<br>
#9 0x00002b2e7c77bc1a in client_xattrop (frame=0x2aaab0030ab0, this=0xae0ab00, loc=0x2aaab4004238, flags=GF_XATTROP_ADD_ARRAY, dict=0x2aaab4031fc0)<br>
at client-protocol.c:1922<br>
#10 0x00002b2e7c9a2cda in afr_changelog_pre_op (frame=0x2aaab401ea70, this=0xae0b280) at afr-transaction.c:782<br>
#11 0x00002b2e7c9a2f31 in afr_lock_rec (frame=0x2aaab401ea70, this=0xae0b280, child_index=1) at afr-transaction.c:979<br>
#12 0x00002b2e7c9a36a8 in afr_lock_cbk (frame=0x2aaab401ea70, cookie=<value optimized out>, this=0xae0b280, op_ret=0, op_errno=0) at afr-transaction.c:906<br>
#13 0x00002b2e7b6a75f0 in default_inodelk_cbk (frame=<value optimized out>, cookie=<value optimized out>, this=<value optimized out>, op_ret=-1,<br>
op_errno=128) at defaults.c:1199<br>
#14 0x00002b2e7c358182 in pl_inodelk (frame=0x2aaab4034c10, this=0xae08870, volume=<value optimized out>, loc=0x2aaab4032170, cmd=7, flock=0x0)<br>
at internal.c:194<br>
#15 0x00002b2e7b6a815c in default_inodelk (frame=0x2aaab4017660, this=0xae09080, volume=0xae0b260 "replicate", loc=0x2aaab4004238, cmd=7,<br>
lock=0x7fff2f422f80) at defaults.c:1209<br>
#16 0x00002b2e7c9a33ba in afr_lock_rec (frame=0x2aaab401ea70, this=0xae0b280, child_index=0) at afr-transaction.c:1006<br>
#17 0x00002b2e7c9a35c2 in afr_transaction (frame=0x2aaab401ea70, this=0xae0b280, type=AFR_DATA_TRANSACTION) at afr-transaction.c:1170<br>
#18 0x00002b2e7c9a07cd in afr_truncate (frame=0x2aaab403ac30, this=0xae0b280, loc=0x2aaab40174a0, offset=0) at afr-inode-write.c:1224<br>
#19 0x00002b2e7cbc0969 in server_truncate_resume (frame=0x2aaab403acc0, this=<value optimized out>, loc=0x2aaab40174a0, offset=0) at server-protocol.c:4243 #20 0x00002b2e7b6b06f7 in call_resume (stub=0x2aaab4017470) at call-stub.c:2384<br>
#21 0x00002b2e7cbc4125 in server_truncate (frame=0x2aaab403acc0, bound_xl=<value optimized out>, hdr=<value optimized out>, hdrlen=<value optimized out>,<br>
iobuf=<value optimized out>) at server-protocol.c:4291<br>
#22 0x00002b2e7cbbfb20 in protocol_server_pollin (this=0xae0bdf0, trans=0xae17960) at server-protocol.c:7735<br>
#23 0x00002b2e7cbbfbfb in notify (this=0xae0bdf0, event=<value optimized out>, data=0x6) at server-protocol.c:7791<br>
#24 0x00002aaaaaaafb43 in socket_event_handler (fd=<value optimized out>, idx=11, data=0xae17960, poll_in=1, poll_out=0, poll_err=0) at socket.c:813<br>
#25 0x00002b2e7b6ba2a5 in event_dispatch_epoll (event_pool=0xae02300) at event.c:804<br>
#26 0x0000000000403899 in main ()<br>
(gdb) bt full<br>
#0 0x0000003040030215 in raise () from /lib64/libc.so.6 No symbol table info available.<br>
#1 0x0000003040031cc0 in abort () from /lib64/libc.so.6 No symbol table info available.<br>
#2 0x000000304006a7fb in __libc_message () from /lib64/libc.so.6 No symbol table info available.<br>
#3 0x0000003040071ce2 in _int_free () from /lib64/libc.so.6 No symbol table info available.<br>
#4 0x000000304007590c in free () from /lib64/libc.so.6 No symbol table info available.<br>
#5 0x00002aaaaaaadcc9 in __socket_ioq_entry_free (entry=0x2aaab001da30) at socket.c:331 No locals.<br>
#6 0x00002aaaaaaaf1c9 in __socket_ioq_churn_entry (this=<value optimized out>, entry=0x2aaab001da30) at socket.c:368<br>
ret = 0<br>
__PRETTY_FUNCTION__ = "__socket_ioq_churn_entry"<br>
#7 0x00002aaaaaaaf8ec in socket_submit (this=0xae11a70, buf=0x2aaab00378c0 "", len=340, vector=0x0, count=<value optimized out>,<br>
iobref=<value optimized out>) at socket.c:1281<br>
priv = (socket_private_t *) 0xae11ec0<br>
ret = <value optimized out><br>
need_poll_out = <value optimized out><br>
entry = (struct ioq *) 0x2aaab001da30<br>
ctx = (glusterfs_ctx_t *) 0xae02010<br>
__FUNCTION__ = "socket_submit"<br>
#8 0x00002b2e7c775bd3 in protocol_client_xfer (frame=0x2aaab0030ab0, this=0xae0ab00, trans=0xae11a70, type=1, op=40, hdr=0x2aaab00378c0, hdrlen=340,<br>
vector=0x0, count=0, iobref=0x0) at client-protocol.c:636<br>
conf = (client_conf_t *) 0xae113c0<br>
conn = (client_connection_t *) 0xae11f90<br>
callid = 309893<br>
ret = <value optimized out><br>
rsphdr = {callid = 0, type = 0, op = 0, size = 0, {req = {pid = 0, uid = 0, gid = 0}, rsp = {op_ret = 0, op_errno = 0}}}<br>
forget = {hdr = 0x0, hdrlen = 0, frame = 0x0}<br>
#9 0x00002b2e7c77bc1a in client_xattrop (frame=0x2aaab0030ab0, this=0xae0ab00, loc=0x2aaab4004238, flags=GF_XATTROP_ADD_ARRAY, dict=0x2aaab4031fc0)<br>
at client-protocol.c:1922<br>
hdr = (gf_hdr_common_t *) 0x101010101010101<br>
req = <value optimized out><br>
dict_len = 242<br>
ret = <value optimized out><br>
pathlen = <value optimized out><br>
ino = 13893685<br>
__FUNCTION__ = "client_xattrop"<br>
#10 0x00002b2e7c9a2cda in afr_changelog_pre_op (frame=0x2aaab401ea70, this=0xae0b280) at afr-transaction.c:782<br>
_new = (call_frame_t *) 0x6943<br>
priv = (afr_private_t *) 0xae13740<br>
ret = <value optimized out><br>
call_count = 1<br>
xattr = (dict_t *) 0x2aaab4031fc0<br>
local = (afr_local_t *) 0x2aaab4004200<br>
__FUNCTION__ = "afr_changelog_pre_op"<br>
#11 0x00002b2e7c9a2f31 in afr_lock_rec (frame=0x2aaab401ea70, this=0xae0b280, child_index=1) at afr-transaction.c:979<br>
local = (afr_local_t *) 0x2aaab4004200<br>
priv = (afr_private_t *) 0xae13740<br>
flock = {l_type = 1, l_whence = 12098, l_start = 0, l_len = 0, l_pid = 792866320}<br>
lower = <value optimized out><br>
higher = <value optimized out><br>
lower_name = <value optimized out><br>
higher_name = <value optimized out><br>
__FUNCTION__ = "afr_lock_rec"<br>
#12 0x00002b2e7c9a36a8 in afr_lock_cbk (frame=0x2aaab401ea70, cookie=<value optimized out>, this=0xae0b280, op_ret=0, op_errno=0) at afr-transaction.c:906 ---Type <return> to continue, or q <return> to quit---<br>
local = (afr_local_t *) 0x2aaab4004200<br>
child_index = 0<br>
call_count = 0<br>
__FUNCTION__ = "afr_lock_cbk"<br>
#13 0x00002b2e7b6a75f0 in default_inodelk_cbk (frame=<value optimized out>, cookie=<value optimized out>, this=<value optimized out>, op_ret=-1,<br>
op_errno=128) at defaults.c:1199<br>
fn = (ret_fn_t) 0x101010101010101<br>
_parent = (call_frame_t *) 0x6943<br>
#14 0x00002b2e7c358182 in pl_inodelk (frame=0x2aaab4034c10, this=0xae08870, volume=<value optimized out>, loc=0x2aaab4032170, cmd=7, flock=0x0)<br>
at internal.c:194<br>
fn = (ret_fn_t) 0x101010101010101<br>
_parent = (call_frame_t *) 0x6943<br>
op_ret = -1<br>
op_errno = 128<br>
ret = 0<br>
can_block = 1<br>
transport = <value optimized out><br>
client_pid = 1<br>
pinode = (pl_inode_t *) 0x2aaab0032810<br>
reqlock = (posix_lock_t *) 0x2aaab4032170<br>
__FUNCTION__ = "pl_inodelk"<br>
#15 0x00002b2e7b6a815c in default_inodelk (frame=0x2aaab4017660, this=0xae09080, volume=0xae0b260 "replicate", loc=0x2aaab4004238, cmd=7,<br>
lock=0x7fff2f422f80) at defaults.c:1209<br>
_new = (call_frame_t *) 0x6943<br>
#16 0x00002b2e7c9a33ba in afr_lock_rec (frame=0x2aaab401ea70, this=0xae0b280, child_index=0) at afr-transaction.c:1006<br>
_new = (call_frame_t *) 0x6943<br>
local = (afr_local_t *) 0x2aaab4004200<br>
priv = (afr_private_t *) 0xae13740<br>
flock = {l_type = 1, l_whence = 1, l_start = 0, l_len = 0, l_pid = 1074216160}<br>
lower = <value optimized out><br>
higher = <value optimized out><br>
lower_name = <value optimized out><br>
higher_name = <value optimized out><br>
__FUNCTION__ = "afr_lock_rec"<br>
#17 0x00002b2e7c9a35c2 in afr_transaction (frame=0x2aaab401ea70, this=0xae0b280, type=AFR_DATA_TRANSACTION) at afr-transaction.c:1170<br>
local = (afr_local_t *) 0x2aaab4004200<br>
priv = (afr_private_t *) 0xae13740<br>
#18 0x00002b2e7c9a07cd in afr_truncate (frame=0x2aaab403ac30, this=0xae0b280, loc=0x2aaab40174a0, offset=0) at afr-inode-write.c:1224<br>
transaction_frame = (call_frame_t *) 0x2aaab401ea70<br>
op_errno = 107<br>
__FUNCTION__ = "afr_truncate"<br>
#19 0x00002b2e7cbc0969 in server_truncate_resume (frame=0x2aaab403acc0, this=<value optimized out>, loc=0x2aaab40174a0, offset=0) at server-protocol.c:4243<br>
_new = (call_frame_t *) 0x6943<br>
__FUNCTION__ = "server_truncate_resume"<br>
#20 0x00002b2e7b6b06f7 in call_resume (stub=0x2aaab4017470) at call-stub.c:2384<br>
__FUNCTION__ = "call_resume"<br>
#21 0x00002b2e7cbc4125 in server_truncate (frame=0x2aaab403acc0, bound_xl=<value optimized out>, hdr=<value optimized out>, hdrlen=<value optimized out>,<br>
iobuf=<value optimized out>) at server-protocol.c:4291<br>
truncate_stub = (call_stub_t *) 0x0<br>
state = (server_state_t *) 0x2aaab4032000<br>
#22 0x00002b2e7cbbfb20 in protocol_server_pollin (this=0xae0bdf0, trans=0xae17960) at server-protocol.c:7735<br>
hdr = 0x2aaab4017330 ""<br>
hdrlen = 98<br>
ret = 0<br>
iobuf = (struct iobuf *) 0x0<br>
#23 0x00002b2e7cbbfbfb in notify (this=0xae0bdf0, event=<value optimized out>, data=0x6) at server-protocol.c:7791<br>
ret = <value optimized out><br>
trans = (transport_t *) 0x6943<br>
---Type <return> to continue, or q <return> to quit---<br>
peerinfo = (peer_info_t *) 0xae179d0<br>
myinfo = (peer_info_t *) 0xae17ac0<br>
__FUNCTION__ = "notify"<br>
#24 0x00002aaaaaaafb43 in socket_event_handler (fd=<value optimized out>, idx=11, data=0xae17960, poll_in=1, poll_out=0, poll_err=0) at socket.c:813<br>
this = (transport_t *) 0x6943<br>
priv = (socket_private_t *) 0xae16ce0<br>
ret = 0<br>
#25 0x00002b2e7b6ba2a5 in event_dispatch_epoll (event_pool=0xae02300) at event.c:804<br>
events = (struct epoll_event *) 0xae15990<br>
i = 0<br>
ret = 1<br>
__FUNCTION__ = "event_dispatch_epoll"<br>
#26 0x0000000000403899 in main ()<br>
No symbol table info available.<br>
<br>
<br>
<br>
<br>
<br>
_______________________________________________<br>
Gluster-devel mailing list<br>
<a href="mailto:Gluster-devel@nongnu.org" target="_blank">Gluster-devel@nongnu.org</a><br>
<a href="http://lists.nongnu.org/mailman/listinfo/gluster-devel" target="_blank">http://lists.nongnu.org/mailman/listinfo/gluster-devel</a><br>
<br>
</blockquote>
<br>
</div></div><br>_______________________________________________<br>
Gluster-devel mailing list<br>
<a href="mailto:Gluster-devel@nongnu.org">Gluster-devel@nongnu.org</a><br>
<a href="http://lists.nongnu.org/mailman/listinfo/gluster-devel" target="_blank">http://lists.nongnu.org/mailman/listinfo/gluster-devel</a><br>
<br></blockquote></div><br><br clear="all"><br>-- <br>Raghavendra G<br><br>