<div dir="ltr">So, I guess I figured it out. I had been looking for a volume problem based on the log messages but it turns out it was a peer definition problem. One of the files in /var/lib/glusterd/peers was empty. I was able to determine where to look based on the output of running /usr/sbin/glusterd --debug --pid-file=/var/run/glusterd.pid and then I was able to copy the missing file from one of the other peers since each peer has a file for each of the other 2 peers.<br>
<div class="gmail_extra"><br><br><div class="gmail_quote">On Mon, Oct 7, 2013 at 12:11 PM, Mark Morlino <span dir="ltr"><<a href="mailto:mark@gina.alaska.edu" target="_blank">mark@gina.alaska.edu</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr">I'm hoping that someone here can point me the right direction to help me solve a problem I am having.<div>
<br></div><div>I've got 3 gluster peers and for some reason glusterd sill not start on one of them. All are running glusterfs version 3.4.0-8.el6 on Centos 6.4 (2.6.32-358.el6.x86_64).<div>
<br></div><div>In /var/log/glusterfs/etc-glusterfs-glusterd.vol.log I see this error repeated 36 times (alternating between brick-0 and brick-1):</div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<b><i>E [glusterd-store.c:1845:glusterd_store_retrieve_volume] 0-: Unknown key: brick-0</i></b> </blockquote></blockquote><div><br></div><div>This makes some sense to me since I have 18 replica 2 volumes resulting in a total of 36 bricks.</div>
<div><br></div><div>Then there are a few more "I" messages and this is the rest of the file:</div><div><br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<b><i>E [glusterd-store.c:2472:glusterd_resolve_all_bricks] 0-glusterd: resolve brick failed in restore<br></i></b><b><i>E [xlator.c:390:xlator_init] 0-management: Initialization of volume 'management' failed, review your volfile again<br>
</i></b><b><i>E [graph.c:292:glusterfs_graph_init] 0-management: initializing translator failed<br></i></b><b><i>E [graph.c:479:glusterfs_graph_activate] 0-graph: init failed<br></i></b><b><i>W [glusterfsd.c:1002:cleanup_and_exit] (-->/usr/sbin/glusterd(main+0x5d2) [0x406802] (-->/usr/sbin/glusterd(glusterfs_volumes_init+0xb7) [0x4051b7] (-->/usr/sbin/glusterd(glusterfs_process_volfp+0x103) [0x4050c3]))) 0-: received signum (0), shutting down</i></b></blockquote>
<div><br></div><div> Here are the contents of /etc/glusterfs/glusterd.vol:</div></div><div><br></div><div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<b><i>volume management<br></i></b><b><i> type mgmt/glusterd<br></i></b><b><i> option working-directory /var/lib/glusterd<br></i></b><b><i> option transport-type socket,rdma<br></i></b><b><i> option transport.socket.keepalive-time 10<br>
</i></b><b><i> option transport.socket.keepalive-interval 2<br></i></b><b><i> option transport.socket.read-fail-log off<br></i></b><b><i>end-volume</i></b></blockquote></div><div><br></div><div>glusterd.vol is the same on all of the peers and the other ones work.</div>
<div><br></div><div>Any help on where to look next would be greatly appreciated.</div><div><br></div><div>Thanks,</div><div>Mark</div></div></div>
</blockquote></div><br></div></div>