OK, figured this one out (my word this took some time....). Found an attribute set on the fourth node after running through them again.<div><br></div><div>Back to work... after patching all nodes and removing/recreating the volume, this now works for me. Definitely not simple and please, if we can clean up volumes after a delete (remove .glusterfs dir and extra attributes across all nodes) that would be tremendously useful.</div>
<div><br></div><div>I'm going to throw a bunch of VMware VMs on my new volume and try it out.</div><div><br></div><div>Recommendation for you, Fernando. I had to roll my own RPMs to get close to consistency across all nodes. Basically:</div>
<div><br></div><div>- Download glusterfs-3.3.0.tar.gz, untar into its directory</div><div>- Replace the files from Anand's patch (<a href="http://review.gluster.com/#change,3617">http://review.gluster.com/#change,3617</a>)</div>
<div>- Re-tar up the file and run 'rpmbuild -ta glusterfs-3.3.0.tar.gz'</div><div>- You'll end up with RPMs in the rpmbuild/RPMS directory from which you can scp to other nodes and bring them all up on the new version</div>
<div><br></div><div>You may need to do the same with the Fuse client and work through some dependencies but it should work.</div><div><br></div><div>Many Thanks!</div><div><br></div><div>Simon<br><br><div class="gmail_quote">
On Tue, Jun 26, 2012 at 3:37 PM, Anand Avati <span dir="ltr"><<a href="mailto:anand.avati@gmail.com" target="_blank">anand.avati@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<p>Can you get the output of getfattr -d -e hex -m . /gfs</p>
<p>Avati</p>
<div class="gmail_quote">On Jun 26, 2012 5:08 PM, "Simon Blackstein" <<a href="mailto:simon@blackstein.com" target="_blank">simon@blackstein.com</a>> wrote:<br type="attribution"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div><span style="color:rgb(34,34,34);font-size:13px;font-family:arial,sans-serif">Thanks Brian.</span><br style="color:rgb(34,34,34);font-size:13px;font-family:arial,sans-serif">
<br style="color:rgb(34,34,34);font-size:13px;font-family:arial,sans-serif"><span style="color:rgb(34,34,34);font-size:13px;font-family:arial,sans-serif">Yes, got rid of the .glusterfs and .vSphereHA directory that VMware</span><br style="color:rgb(34,34,34);font-size:13px;font-family:arial,sans-serif">
<span style="color:rgb(34,34,34);font-size:13px;font-family:arial,sans-serif">makes. Rebooted, so yes it was remounted and used a different mount</span><br style="color:rgb(34,34,34);font-size:13px;font-family:arial,sans-serif">
<span style="color:rgb(34,34,34);font-size:13px;font-family:arial,sans-serif">point name. Also got rid of attribute I found set on the root:</span><br style="color:rgb(34,34,34);font-size:13px;font-family:arial,sans-serif">
<br style="color:rgb(34,34,34);font-size:13px;font-family:arial,sans-serif"><span style="color:rgb(34,34,34);font-size:13px;font-family:arial,sans-serif">setfattr -x trusted.gfid / && setfattr -x trusted.glusterfs.dht /</span><br style="color:rgb(34,34,34);font-size:13px;font-family:arial,sans-serif">
<br style="color:rgb(34,34,34);font-size:13px;font-family:arial,sans-serif"><span style="color:rgb(34,34,34);font-size:13px;font-family:arial,sans-serif">Any other tips? :)</span><br style="color:rgb(34,34,34);font-size:13px;font-family:arial,sans-serif">
<span style="color:rgb(34,34,34);font-size:13px;font-family:arial,sans-serif"><br></span></div><div><span style="color:rgb(34,34,34);font-size:13px;font-family:arial,sans-serif">Many Rgds,</span><br style="color:rgb(34,34,34);font-size:13px;font-family:arial,sans-serif">
<span style="color:rgb(34,34,34);font-size:13px;font-family:arial,sans-serif"><br></span></div><div><span style="color:rgb(34,34,34);font-size:13px;font-family:arial,sans-serif">Simon</span></div>
<br><div class="gmail_quote">On Tue, Jun 26, 2012 at 12:19 PM, Brian Candler <span dir="ltr"><<a href="mailto:B.Candler@pobox.com" target="_blank">B.Candler@pobox.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
On Tue, Jun 26, 2012 at 11:46:52AM -0700, Simon Blackstein wrote:<br>
> Basically did all of that as previously noted:<br>
<br>
And rm -rf .glusterfs ?<br>
<br>
If /gfs is the mountpoint, you could also try<br>
unmount /gfs<br>
rmdir /gfs<br>
mkdir /gfs<br>
and remount.<br>
</blockquote></div><br>
</blockquote></div>
</blockquote></div><br></div>