<div dir="ltr">
<p class="">ESXI cannot access stripped gluster volume </p>
<p class="">Hello all.</p>
<p class="">Unfortunately this is going to be a long post, so I will not spend too many words with compliments; Gluster is a great solution and I should be writing odes about it, so, bravo all of you.</p>
<p class="">A bit about me: I've been working with FreeBSD and Linux for over a decade now. I use CentOS nowadays because of some convenient features, regarding my applications.</p>
<p class="">Now a bit more about my problem: After adding my stripped gluster volume to my ESXI via NFS, I try to browse it with vSphere's browser, and it... </p>
<p class="">a) cannot see any of the folders already there. The operation never times out, and I have a line of dots telling me the system (esxi) is trying to do something. Other systems CAN access and see content on the same volume.</p>
<p class="">b) gluster or ESXI return/log NO ERROR when I (try to) create folders there, but does not show them either. Folder IS created.</p>
<p class="">c) When trying to create a virtual machine it DOES return an error. Folder and first file related to the VM ARE created but I get an error about the an "Invalid virtual machine configuration". I am under the impression esxi returns this when it tries to create the file for the virtual disk.</p>
<p class="">d) When trying to remove the volume, it DOES return an error, stating the resource is busy. I am forced to reboot the ESXI host in order to successfully delete the nfs volume.</p>
<p class="">Now, a bit about my environment:</p>
<p class="">I am one of those cursed with an Isilon, but with NO service contract and NO license. So, basically I have a big, fast and resilient NAS. Cool stuff, but with great inconveniences as well. Goes without saying that me, as a free software guy, would love to build something that can retire the isilon, or at least, move it to a secondary role.</p>
<p class="">Anyway, trying to add an alternative to it, I searched for days and decided Gluster was the way to go. And I am not going back. I will make it work.</p>
<p class="">All of my VM servers, (about 15) are spread across 3 metal boxes, and - please, don't blame me, I inherited this situation - there is no backup solution whatsoever. Gluster will, in its final configuration, run on 4 boxes, providing HA and backup.</p>
<p class="">So, on esxi, Isilon's NFS volume/share works like a charm; pure NFS sharing on CentOS works like a charm; gluster stripe - using two of my four servers - does not like me.</p>
<p class="">The very same gluster NSF volume is mounted and works happily on a ContOS client. Actually, more than one.</p>
<p class="">I have been reading literally dozens of docs, guides, manuals, VMware, Gluster and Red Hat for more than a week, and while that, I've even created ESXI VIRTUAL SERVERS *INSIDE* my EXSI physical servers, because I can no longer afford rebooting a production server whenever I need to test yet another change on gluster.</p>
<p class=""><br></p>
<p class="">My software versions:</p>
<p class="">CentOS 6.5<br>Gluster 3.4.2<br>ESXI 5.1 all patches applied<br>ESXI 5.5</p><p class="">My hardware for the nodes: 2 x Dell PE2950 with RAID5, bricks are single volume of about 1.5 TB on each node. </p>
<p class="">One stand-alone PE2900 with a single volume, on RAID5, and about 2.4 TB, which will be added to the stripe, eventually. One PE2950, brick on RAID5 800GB, which will also be added eventually.</p>
<p class="">All of them with one NIC for regular networking and a bonded nic made out of 2 physical NICs, for gluster/nfs.</p>
<p class="">ESXIs are running on R710s, lots of ram, and, at least one NIC dedicated to NFS. I have one test server running with all four NICs on the NFS netowrk.</p>
<p class="">NFS network is 9000 MTU, tuned for iSCSI (in the future).</p>
<p class="">Now, trying to make it all work, these are the steps I took:</p>
<p class="">Regarding ESXI tweaks, I've changed GLUSTER'S ping limit, lowering it to 20, to stop the volume from being intermittently inaccessible. On ESXI itself I've set the NFS max queue length to 64. </p>
<p class="">I've chmoded gluster's share with 777, and you can find my gluster tweaks for the volume below.</p>
<p class="">Gluster's NFS and regular NFS both are forcing uid and gid to nfsnobody:nfsnobody.</p>
<p class="">iptables has been disabled, along with SElinux.</p>
<p class="">Of course regular NFS is disabled. </p>
<p class="">My gluster settings:</p>
<p class=""><br></p>
<p class="">Volume Name: glvol0<br>Type: Stripe<br>Volume ID: f76af2ac-6a42-42ea-9887-941bf1600ced<br>Status: Started<br>Number of Bricks: 1 x 2 = 2<br>Transport-type: tcp<br>Bricks:<br>Brick1: 10.0.1.21:/export/glusterroot<br>
Brick2: 10.0.1.22:/export/glusterroot<br>Options Reconfigured:<br>nfs.ports-insecure: on<br>nfs.addr-namelookup: off<br>auth.reject: NONE<br>nfs.volume-access: read-write<br>nfs.nlm: off<br>network.ping-timeout: 20<br>server.root-squash: on<br>
performance.nfs.write-behind: on<br>performance.nfs.read-ahead: on<br>performance.nfs.io-cache: on<br>performance.nfs.quick-read: on<br>performance.nfs.stat-prefetch: on<br>performance.nfs.io-threads: on<br>storage.owner-uid: 65534<br>
storage.owner-gid: 65534<br>nfs.disable: off<br><br><br>My regular NFS settings, that work just the way I need:</p>
<p class="">/export/share<span class=""> </span><span class=""> </span>*(rw,all_squash,anonuid=65534,anongid=65534,no_subtree_check)</p>
<p class="">Once I get this all to work, I intend to create a nice page with instructions/info on gluster for ESXi. This field needs some more work out there.</p>
<p class="">Now the question: is there anything I forgot, overlooked and don't know ?</p>
<p class="">Could you help me ? Except for a comment from someone, saying "stripe does not work with esxi", nothing REALLY rings a bell. I've used all the pertinent info I had, and ran out of moves.</p><p class="">
My only option now would be testing a volume that is not a stripe. I'll do that for now, but I don't think it will work.</p><p class="">BTW, can I be added to the list ? </p>
<p class="">Cheers,</p>
<p class="">Carlos.</p></div>