<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
<div class="">Folks,</div>
<div class=""><br class="">
</div>
This feature:
<div class=""><br class="">
</div>
<div class=""><a href="http://review.gluster.org/#/c/4430/" class="">http://review.gluster.org/#/c/4430/</a></div>
<div class=""><br class="">
</div>
<div class="">Create bottleneck when several clients are accessing a nfs volume.</div>
<div class=""><br class="">
</div>
<div class="">On our setup:</div>
<div class=""><br class="">
</div>
<div class="">Gluster 3.5.2 under centos7.</div>
<div class=""><br class="">
</div>
<div class="">Hardware:</div>
<div class=""><br class="">
</div>
<div class=""><span class="Apple-tab-span" style="white-space:pre"></span>dual Xeon® CPU E5-2640</div>
<div class=""><span class="Apple-tab-span" style="white-space:pre"></span>64GB RAM</div>
<div class=""><span class="Apple-tab-span" style="white-space:pre"></span>SSD for rootfs</div>
<div class=""><span class="Apple-tab-span" style="white-space:pre"></span>10Gb NIC</div>
<div class=""><br class="">
</div>
<div class="">Context:</div>
<div class=""><br class="">
</div>
<div class=""><span class="Apple-tab-span" style="white-space:pre"></span>Around 700 nfs clients for small file or vm images.</div>
<div class=""><br class="">
</div>
<div class="">The issue:</div>
<div class=""><br class="">
</div>
<div class="">NFS client got intermittent hang (every minutes / for 10s each time). Even an “rpcinfo -t server nfs 3" will hang.</div>
<div class=""><br class="">
</div>
<div class="">Gluster nfs process literally eat the CPU of the server</div>
<div class=""><br class="">
</div>
<div class="">The cause:</div>
<div class=""><br class="">
</div>
<div class="">the rmtab file located in /var/lib/glusterd/nfs/ is flushed from memory to /var/lib/glusterd/nfs/rmtab.tmp. During this time, NFS server hang literraly.</div>
<div class=""><br class="">
</div>
<div class=""><img apple-inline="yes" id="8DF7DA1B-89F4-48DA-9525-6314F03A7A2F" height="498" width="902" apple-width="yes" apple-height="yes" src="cid:2BF1DF8D-4144-48FF-A63E-F5D2F61285CA@eng.timetra.com" class=""></div>
<div class=""><br class="">
</div>
<div class=""><img apple-inline="yes" id="D520A126-3048-4A74-9534-F0A65A06D16F" height="497" width="902" apple-width="yes" apple-height="yes" src="cid:D11F71BF-A4FD-4B96-B7A1-17AB2F7114A3@eng.timetra.com" class=""></div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class="">Workaroud:</div>
<div class=""><br class="">
</div>
<div class="">Move the file to memory for faster I/O using this option:</div>
<div class=""><br class="">
</div>
<div class="">set nfs.mount-rmtab: /dev/shm/glusterfs.rmtab</div>
<div class=""><br class="">
</div>
<div class="">Result:</div>
<div class=""><br class="">
</div>
<div class="">We still have some hang but for ~300ms now, the load average of the server is WAY better.</div>
<div class=""><br class="">
</div>
<div class="">Personal thought:</div>
<div class=""><br class="">
</div>
<div class="">This feature is not usable and should be disabled by default.</div>
<div class=""><br class="">
</div>
<div class="">I raised <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1166862" class="">Bug 1166862</a>.<br class="">
<div apple-content-edited="true" class="">
<div style="color: rgb(0, 0, 0); letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
-- <br class="">
Cyril Peponnet</div>
</div>
<br class="">
</div>
</body>
</html>