<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoAcetate, li.MsoAcetate, div.MsoAcetate
{mso-style-priority:99;
mso-style-link:"Balloon Text Char";
margin:0in;
margin-bottom:.0001pt;
font-size:8.0pt;
font-family:"Tahoma","sans-serif";}
span.EmailStyle17
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.BalloonTextChar
{mso-style-name:"Balloon Text Char";
mso-style-priority:99;
mso-style-link:"Balloon Text";
font-family:"Tahoma","sans-serif";}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">It is indeed ext4. I will give XFS a try.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span style="font-size:9.0pt;font-family:"Arial","sans-serif";color:#FF671F">Matthew Rinella</span></b><span style="font-size:9.0pt;font-family:"Arial","sans-serif";color:#54585A"><br>
Sr. Systems Administrator<br>
<br>
<b>Apptio, Inc. | Technology Business Management</b><br>
</span><a href="http://www.apptio.com/signature"><span style="font-size:9.0pt;font-family:"Arial","sans-serif";text-decoration:none"><img border="0" width="520" height="150" id="Picture_x0020_1" src="cid:image001.jpg@01CF637D.01818140" alt="http://media.apptio.com/signature/apptio-email-signature.jpg"></span></a><span style="font-size:9.0pt;font-family:"Arial","sans-serif";color:#54585A"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> Carlos Capriotti [mailto:capriotti.carlos@gmail.com]
<br>
<b>Sent:</b> Monday, April 28, 2014 9:20 PM<br>
<b>To:</b> Matthew Rinella<br>
<b>Cc:</b> gluster-users@gluster.org<br>
<b>Subject:</b> Re: [Gluster-users] volume start causes glusterd to core dump in 3.5.0<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">Matthew, just out of curiosity, what is the underlying file system on your gluster bricks ? <o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Reason for my asking is that there is (or there was) an issue with EXT4. If you used it, you might want to reformat them using XFS -i 512.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><o:p> </o:p></p>
<div>
<p class="MsoNormal">On Tue, Apr 29, 2014 at 1:59 AM, Matthew Rinella <<a href="mailto:MRinella@apptio.com" target="_blank">MRinella@apptio.com</a>> wrote:<o:p></o:p></p>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">I just built a pair of AWS Red Hat 6.5 instances to create a gluster replicated pair file system. I can install everything, peer probe, and create the volume, but as soon as I
try to start the volume, glusterd dumps core.<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">The tail of the log after the crash:<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">+------------------------------------------------------------------------------+<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">[2014-04-28 21:49:18.102981] I [glusterd-rpc-ops.c:356:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 84c0bb8e-bf48-4386-ada1-3e4db68b980f, host: fed-gfs4, port:
0<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">[2014-04-28 21:49:18.138936] I [glusterd-handler.c:2212:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 84c0bb8e-bf48-4386-ada1-3e4db68b980f<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">[2014-04-28 21:49:18.138982] I [glusterd-handler.c:2257:__glusterd_handle_friend_update] 0-: Received uuid: c7a11029-12ab-4e3a-b898-7c62e98fa4d1, hostname:fed-gfs3<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">[2014-04-28 21:49:18.138995] I [glusterd-handler.c:2266:__glusterd_handle_friend_update] 0-: Received my uuid as Friend<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">[2014-04-28 21:49:18.179134] I [glusterd-handshake.c:563:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 2<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">[2014-04-28 21:49:18.199020] I [glusterd-handler.c:2050:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 84c0bb8e-bf48-4386-ada1-3e4db68b980f<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">[2014-04-28 21:49:18.199111] I [glusterd-handler.c:3085:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to fed-gfs4 (0), ret: 0<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">[2014-04-28 21:49:18.222248] I [glusterd-sm.c:495:glusterd_ac_send_friend_update] 0-: Added uuid: 84c0bb8e-bf48-4386-ada1-3e4db68b980f, host: fed-gfs4<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">[2014-04-28 21:49:18.262901] I [glusterd-rpc-ops.c:553:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 84c0bb8e-bf48-4386-ada1-3e4db68b980f<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">[2014-04-28 21:49:20.401429] I [glusterd-handler.c:1169:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">[2014-04-28 21:49:20.402072] I [glusterd-handler.c:1169:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">pending frames:<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">frame : type(0) op(0)<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">patchset: git://<a href="http://git.gluster.com/glusterfs.git" target="_blank">git.gluster.com/glusterfs.git</a><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">signal received: 6<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">time of crash: 2014-04-28 21:53:12configuration details:<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">argp 1<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">backtrace 1<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">dlfcn 1<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">fdatasync 1<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">libpthread 1<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">llistxattr 1<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">setfsid 1<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">spinlock 1<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">epoll.h 1<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">xattr.h 1<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">st_atim.tv_nsec 1<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">package-string: glusterfs 3.5.0<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">/lib64/libc.so.6(+0x329a0)[0x7eff4946e9a0]<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">/lib64/libc.so.6(gsignal+0x35)[0x7eff4946e925]<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">/lib64/libc.so.6(abort+0x175)[0x7eff49470105]<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">/usr/lib64/libcrypto.so.10(+0x67ebf)[0x7eff49837ebf]<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">/usr/lib64/libcrypto.so.10(MD5_Init+0x49)[0x7eff4983e619]<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">/usr/lib64/libcrypto.so.10(MD5+0x3a)[0x7eff4983e9ea]<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">/usr/lib64/libglusterfs.so.0(md5_wrapper+0x3c)[0x7eff4ae6091c]<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">/usr/lib64/glusterfs/3.5.0/xlator/mgmt/glusterd.so(glusterd_set_socket_filepath+0x72)[0x7eff45e02b72]<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">/usr/lib64/glusterfs/3.5.0/xlator/mgmt/glusterd.so(glusterd_set_brick_socket_filepath+0x158)[0x7eff45e02df8]<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">/usr/lib64/glusterfs/3.5.0/xlator/mgmt/glusterd.so(glusterd_volume_start_glusterfs+0x4c9)[0x7eff45e094a9]<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">/usr/lib64/glusterfs/3.5.0/xlator/mgmt/glusterd.so(glusterd_brick_start+0x119)[0x7eff45e0af29]<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">/usr/lib64/glusterfs/3.5.0/xlator/mgmt/glusterd.so(glusterd_op_start_volume+0xfd)[0x7eff45e45a8d]<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">/usr/lib64/glusterfs/3.5.0/xlator/mgmt/glusterd.so(glusterd_op_commit_perform+0x53b)[0x7eff45df471b]<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">/usr/lib64/glusterfs/3.5.0/xlator/mgmt/glusterd.so(gd_commit_op_phase+0xbe)[0x7eff45e5193e]<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">/usr/lib64/glusterfs/3.5.0/xlator/mgmt/glusterd.so(gd_sync_task_begin+0x2c2)[0x7eff45e53632]<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">/usr/lib64/glusterfs/3.5.0/xlator/mgmt/glusterd.so(glusterd_op_begin_synctask+0x3b)[0x7eff45e5376b]<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">/usr/lib64/glusterfs/3.5.0/xlator/mgmt/glusterd.so(__glusterd_handle_cli_start_volume+0x1b6)[0x7eff45e46cc6]<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">/usr/lib64/glusterfs/3.5.0/xlator/mgmt/glusterd.so(glusterd_big_locked_handler+0x3f)[0x7eff45ddaf7f]<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">/usr/lib64/libglusterfs.so.0(synctask_wrap+0x12)[0x7eff4ae818e2]<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">/lib64/libc.so.6(+0x43bf0)[0x7eff4947fbf0]<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Interestingly enough I downloaded and installed 3.5.0 because this same thing happened with 3.4.2 on Red Hat 6.5 instances in AWS as well. I tore down and rebuilt them with all
the same results. Is there a library its conflicting with? I am still new to gluster, so I don’t know too many of the details of what to look for when it crashes.
<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Thanks for any help.<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><br>
_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
<a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://supercolony.gluster.org/mailman/listinfo/gluster-users</a><o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</body>
</html>