I have a simple setup:<br><br><font class="Apple-style-span" face="'courier new', monospace">gluster> volume info<br><br>Volume Name: myvolume<br>Type: Distributed-Replicate<br>Status: Started<br>Number of Bricks: 3 x 2 = 6<br>
Transport-type: tcp<br>Bricks:<br>Brick1: 10.2.218.188:/srv<br>Brick2: 10.116.245.136:/srv<br>Brick3: 10.206.38.103:/srv<br>Brick4: 10.114.41.53:/srv<br>Brick5: 10.68.73.41:/srv<br>Brick6: 10.204.129.91:/srv</font><br><br>
I *killed* Brick #4 (kill -9 and then shut down instance).<br><br>My intention is to simulate a catastrophic failure of Brick4 and replace it with a new server.<br><br>I probed the new server, then ran the following command:<br>
<br><div><font class="Apple-style-span" face="'courier new', monospace">gluster> peer probe 10.76.242.97</font></div><div><font class="Apple-style-span" face="'courier new', monospace">Probe successful</font></div>
<div><br></div><font class="Apple-style-span" face="'courier new', monospace">gluster> volume replace-brick myvolume 10.114.41.53:/srv 10.76.242.97:/srv start<br>replace-brick started successfully<br></font><br>
I waited a little while, saw no traffic on the new server and then ran this:<br><br><font class="Apple-style-span" face="'courier new', monospace">gluster> volume replace-brick myvolume 10.114.41.53:/srv 10.76.242.97:/srv status<br>
</font><br>It never returned. Now my cluster is in some weird state. It's still serving files, I still have a job copying files to it, but I am unable to replace the bad peer with a new one.<br><br><font class="Apple-style-span" face="'courier new', monospace">root@ip-10-2-218-188:~# gluster volume replace-brick myvolume 10.114.41.53:/srv 10.76.242.97:/srv status<br>
replace-brick status unknown<br><br>root@ip-10-2-218-188:~# gluster volume replace-brick myvolume 10.114.41.53:/srv 10.76.242.97:/srv abort<br>replace-brick abort failed<br><br>root@ip-10-2-218-188:~# gluster volume replace-brick myvolume 10.114.41.53:/srv 10.76.242.97:/srv start<br>
replace-brick failed to start</font><br><br>How can I get my cluster back into a clean working state?<div><br></div><div>Thanks,</div><div>Bryan<br><br></div>