I don't know if it will help, but I see the following in cli.log when I run replace-brick status/start:<div><br></div><div><div><font class="Apple-style-span" face="'courier new', monospace">[2011-09-16 20:54:42.535212] W [rpc-transport.c:605:rpc_transport_load] 0-rpc-transport: missing 'option transport-type'. defaulting to "socket"</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace">[2011-09-16 20:54:43.880179] I [cli-rpc-ops.c:1188:gf_cli3_1_replace_brick_cbk] 0-cli: Received resp to replace brick</font></div><div><font class="Apple-style-span" face="'courier new', monospace">[2011-09-16 20:54:43.880290] I [input.c:46:cli_batch] 0-: Exiting with: 1</font></div>
<br><div class="gmail_quote">On Fri, Sep 16, 2011 at 3:06 PM, Bryan Murphy <span dir="ltr"><<a href="mailto:bmurphy1976@gmail.com">bmurphy1976@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
I have a simple setup:<br><br><font face="'courier new', monospace">gluster> volume info<br><br>Volume Name: myvolume<br>Type: Distributed-Replicate<br>Status: Started<br>Number of Bricks: 3 x 2 = 6<br>
Transport-type: tcp<br>Bricks:<br>Brick1: 10.2.218.188:/srv<br>Brick2: 10.116.245.136:/srv<br>Brick3: 10.206.38.103:/srv<br>Brick4: 10.114.41.53:/srv<br>Brick5: 10.68.73.41:/srv<br>Brick6: 10.204.129.91:/srv</font><br><br>
I *killed* Brick #4 (kill -9 and then shut down instance).<br><br>My intention is to simulate a catastrophic failure of Brick4 and replace it with a new server.<br><br>I probed the new server, then ran the following command:<br>
<br><div><font face="'courier new', monospace">gluster> peer probe 10.76.242.97</font></div><div><font face="'courier new', monospace">Probe successful</font></div>
<div><br></div><font face="'courier new', monospace">gluster> volume replace-brick myvolume 10.114.41.53:/srv 10.76.242.97:/srv start<br>replace-brick started successfully<br></font><br>
I waited a little while, saw no traffic on the new server and then ran this:<br><br><font face="'courier new', monospace">gluster> volume replace-brick myvolume 10.114.41.53:/srv 10.76.242.97:/srv status<br>
</font><br>It never returned. Now my cluster is in some weird state. It's still serving files, I still have a job copying files to it, but I am unable to replace the bad peer with a new one.<br><br><font face="'courier new', monospace">root@ip-10-2-218-188:~# gluster volume replace-brick myvolume 10.114.41.53:/srv 10.76.242.97:/srv status<br>
replace-brick status unknown<br><br>root@ip-10-2-218-188:~# gluster volume replace-brick myvolume 10.114.41.53:/srv 10.76.242.97:/srv abort<br>replace-brick abort failed<br><br>root@ip-10-2-218-188:~# gluster volume replace-brick myvolume 10.114.41.53:/srv 10.76.242.97:/srv start<br>
replace-brick failed to start</font><br><br>How can I get my cluster back into a clean working state?<div><br></div><div>Thanks,</div><div>Bryan<br><br></div>
</blockquote></div><br></div>