<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#FFFFFF">
A few things are not clear to me. Comments inline below.<br>
<br>
On 5/15/2014 5:19 AM, Lyle Plaatjes wrote:<br>
<blockquote
cite="mid:CAHoXNQkBwnP6e_GaxU=TbT4R2CfyacQiOnVBGx0gdpwAmZpyGw@mail.gmail.com"
type="cite">
<div dir="ltr">
<div>
<div>
<div>
<div>
<div>I've just started at a new company and I came
across this problem. They have webservers peering
using gluster 3.2.7.<br>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
<p>I take it that the gluster storage is on the same machines as the
web-server software?<br>
</p>
<p>Was this a replica-4 setup, where there is a replicated brick on
each of 4 nodes?<br>
</p>
<blockquote
cite="mid:CAHoXNQkBwnP6e_GaxU=TbT4R2CfyacQiOnVBGx0gdpwAmZpyGw@mail.gmail.com"
type="cite">
<div dir="ltr">
<div>
<div>
<div>
<div>The problem comes in where they upgraded the VM's to
newer versions of Ubuntu. They didn't replace the bricks
before decommissioning the other nodes. Only one node is
actually still running so that is 1 brick that actually
exists and is being replicated to the new hosts.<br>
</div>
</div>
</div>
</div>
</div>
</blockquote>
<p>So there really is no true replication going on, just all files
being served from the one gluster brick that still works? (And if
that brick dies, the whole site disappears until restored from
backup {if there is a backup})<br>
</p>
<blockquote
cite="mid:CAHoXNQkBwnP6e_GaxU=TbT4R2CfyacQiOnVBGx0gdpwAmZpyGw@mail.gmail.com"
type="cite">
<div dir="ltr">
<div>
<div>
<div>Now when one of the hosts is rebooted then gluster
doesn't mount the volume because it's looking at the 3
dead peers and one that is still fine.<br>
</div>
</div>
</div>
</div>
</blockquote>
<p>Are your new nodes (without bricks) peers in the gluster cluster?<br>
Are your mounts of the form localhost:<volume-name> ?<br>
</p>
<blockquote
cite="mid:CAHoXNQkBwnP6e_GaxU=TbT4R2CfyacQiOnVBGx0gdpwAmZpyGw@mail.gmail.com"
type="cite">
<div dir="ltr">
<div>
<div>What I need to do is replace the old dead peers with the
new ones so that the gluster volume will actually mount if a
host is rebooted.<br>
</div>
</div>
</div>
</blockquote>
Based on my guesses as to what your setup is, I think this is what I
would do.<br>
<ul>
<li>Get all web servers operating as peers in the trusted pool</li>
<ul>
<li>It is not clear whether the new web servers even have
gluster installed</li>
</ul>
<li>change /etc/fstab so that mounts are of the form
localhost:<volume-name></li>
<ul>
<li>so that it doesn't matter what other node is up or down, as
long as the volume is active</li>
<li>I don't' know what exact commands Ubuntu is using, but in
Centos 6 I use the "nofail" option in the fourth column of
/etc/fstab (where 'default' is a common entry).</li>
<ul>
<li>This allows the bootup to proceed, even though the volume
may not be mountable yet.</li>
<ul>
<li>During (Centos) bootup, mount gets a second (or third)
chance to mount things<br>
</li>
</ul>
</ul>
<li>make sure that the last two columns in /etc/fstab are "0 0"</li>
<ul>
<li>so that it doesn't try to do a filesystem check during
bootup<br>
</li>
</ul>
</ul>
<li>set up bricks on new servers</li>
<ul>
<li>if the machine names are the same as the old machines, use a
different path from the old brick path</li>
<ul>
<li>to see old brick path, run "gluster volume info
<volume-name>"</li>
</ul>
<li>put brick in /etc/fstab, so it gets mounted<br>
</li>
</ul>
<li>run "gluster volume add-brick <volume-name> replica
<current replica number + 1>
<hostname>:/<brick-path>" on each node</li>
<ul>
<li>this adds the new bricks to the volume</li>
<li>you may need to wait until one brick has healed
(synchronized) before adding the next brick</li>
<ul>
<li>even synching one brick can saturate a network link, and
bring things to their knees<br>
</li>
</ul>
<li>repeat until you have 4 bricks active</li>
</ul>
<li>run "gluster volume remove-brick <volume-name> replica
<old replica number -1>
<hostname>:/<brick-path>" to remove "historic"
bricks</li>
<ul>
<li>don't do the remove bricks before adding any bricks</li>
<ul>
<li>taking a replicated volume down to one brick and then
trying to bring it back to two bricks can be problematic on
some (maybe all) versions of gluster<br>
</li>
</ul>
</ul>
</ul>
<p>Remember, I am assuming:</p>
<ul>
<li>you have 4 web servers that should also all be gluster brick
nodes</li>
<li>you were running a replica 4 configuration</li>
</ul>
<p>Ted Miller<br>
Elkhart, IN, USA<br>
</p>
<p>I am not an official part of gluster, just another user who has
added and removed bricks a few times.<br>
<br>
</p>
</body>
</html>