<html>
<head>
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<br>
<div class="moz-cite-prefix">On 10/08/2014 11:54 AM, Geoffrey
Letessier wrote:<br>
</div>
<blockquote cite="mid:47547A8B-407A-47A9-BCD6-0E808CBF87DB@cnrs.fr"
type="cite">
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252">
Hello,
<div><br>
</div>
<div>First of all, thanks to spend some times to help me to fix my
issues.</div>
<div><br>
</div>
<div>Previously, some weeks ago, i’ve already installed a
GlusterFS patched version (<a moz-do-not-send="true"
href="http://download.gluster.org/pub/gluster/glusterfs/nightly/glusterfs-3.5/epel-6-x86_64/glusterfs-3.5.20140922.da1657d-1.autobuild/">http://download.gluster.org/pub/gluster/glusterfs/nightly/glusterfs-3.5/epel-6-x86_64/glusterfs-3.5.20140922.da1657d-1.autobuild/</a> )
to fix quota daemon (and libc) issues thanks to Krutika
Dhananjay. </div>
</blockquote>
<br>
<blockquote cite="mid:47547A8B-407A-47A9-BCD6-0E808CBF87DB@cnrs.fr"
type="cite">
<div>Do this patched version you offer me is compatible with the
previously said one? If yes, is it possible to get RPM packages?</div>
<div><br>
</div>
</blockquote>
<br>
I hope it will be compatible with the quota patch. But we need to
figure out the availability of rpm packages for 3.5, hopefully i
can let you know the availability of RPM packages within 1 day.<br>
<br>
<blockquote cite="mid:47547A8B-407A-47A9-BCD6-0E808CBF87DB@cnrs.fr"
type="cite">
<div>Currently we have around 42TB used disk space (per replica);
so is there any risk to apply this patched version on a
production environment?</div>
</blockquote>
<br>
Currently rdma is a tech preview , which is aimed to release in
next versions. Once you applied the patch, all the communication
will go through rdma instead of tcp. I hope there will not be any
major risks. <br>
<br>
<blockquote cite="mid:47547A8B-407A-47A9-BCD6-0E808CBF87DB@cnrs.fr"
type="cite">
<div><br>
</div>
<div>In addition, concerning my global IO performances, do you
feel my settings as optimal?</div>
</blockquote>
<br>
I'm not sure about IO performance and settings of your volume,
perhaps some one from this group can help you about that. <br>
<br>
Rafi KC<br>
<blockquote cite="mid:47547A8B-407A-47A9-BCD6-0E808CBF87DB@cnrs.fr"
type="cite">
<div><br>
</div>
<div>Thanks in advance,</div>
<div>Geoffrey</div>
<div><br>
<div>
<div>
<div>Le 8 oct. 2014 à 06:58, Mohammed Rafi K C <<a
moz-do-not-send="true" href="mailto:rkavunga@redhat.com">rkavunga@redhat.com</a>>
a écrit :</div>
<br class="Apple-interchange-newline">
<blockquote type="cite">
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
<div bgcolor="#FFFFFF" text="#000000"> <br>
<div class="moz-cite-prefix">On 10/08/2014 04:15 AM,
Geoffrey Letessier wrote:<br>
</div>
<blockquote
cite="mid:79347B68-EC53-4B62-80F0-298130A41AF6@cnrs.fr"
type="cite">
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252">
Dears,
<div><br>
</div>
<div>I have a HPC cluster composed by 4 storage nodes
(8x 24TB RAID6 bricks, 2 per nodes) and 62 compute
nodes, interconnected via Infiniband QDR technology.</div>
<div><br>
</div>
<div>NB: each brick provide around 1.2-1.5TBs write
performances.</div>
<div> </div>
<div>My main volume is defined as below</div>
<div>
<div style="margin: 0px; font-family: Menlo; color:
rgb(255, 255, 255); background-color: rgb(0, 0,
0);"><span style="font-size: 9px;">Volume Name:
vol_home</span></div>
<div style="margin: 0px; font-family: Menlo; color:
rgb(255, 255, 255); background-color: rgb(0, 0,
0);"><span style="font-size: 9px;">Type:
Distributed-Replicate</span></div>
<div style="margin: 0px; font-family: Menlo; color:
rgb(255, 255, 255); background-color: rgb(0, 0,
0);"><span style="font-size: 9px;">Volume ID:
f6ebcfc1-b735-4a0e-b1d7-47ed2d2e7af6</span></div>
<div style="margin: 0px; font-family: Menlo; color:
rgb(255, 255, 255); background-color: rgb(0, 0,
0);"><span style="font-size: 9px;">Status: Started</span></div>
<div style="margin: 0px; font-family: Menlo; color:
rgb(255, 255, 255); background-color: rgb(0, 0,
0);"><span style="font-size: 9px;">Number of
Bricks: 4 x 2 = 8</span></div>
<div style="margin: 0px; font-family: Menlo; color:
rgb(255, 255, 255); background-color: rgb(0, 0,
0);"><span style="font-size: 9px;">Transport-type:
tcp,rdma</span></div>
<div style="margin: 0px; font-family: Menlo; color:
rgb(255, 255, 255); background-color: rgb(0, 0,
0);"><span style="font-size: 9px;">Bricks:</span></div>
<div style="margin: 0px; font-family: Menlo; color:
rgb(255, 255, 255); background-color: rgb(0, 0,
0);"><span style="font-size: 9px;">Brick1:
ib-storage1:/export/brick_home/brick1</span></div>
<div style="margin: 0px; font-family: Menlo; color:
rgb(255, 255, 255); background-color: rgb(0, 0,
0);"><span style="font-size: 9px;">Brick2:
ib-storage2:/export/brick_home/brick1</span></div>
<div style="margin: 0px; font-family: Menlo; color:
rgb(255, 255, 255); background-color: rgb(0, 0,
0);"><span style="font-size: 9px;">Brick3:
ib-storage3:/export/brick_home/brick1</span></div>
<div style="margin: 0px; font-family: Menlo; color:
rgb(255, 255, 255); background-color: rgb(0, 0,
0);"><span style="font-size: 9px;">Brick4:
ib-storage4:/export/brick_home/brick1</span></div>
<div style="margin: 0px; font-family: Menlo; color:
rgb(255, 255, 255); background-color: rgb(0, 0,
0);"><span style="font-size: 9px;">Brick5:
ib-storage1:/export/brick_home/brick2</span></div>
<div style="margin: 0px; font-family: Menlo; color:
rgb(255, 255, 255); background-color: rgb(0, 0,
0);"><span style="font-size: 9px;">Brick6:
ib-storage2:/export/brick_home/brick2</span></div>
<div style="margin: 0px; font-family: Menlo; color:
rgb(255, 255, 255); background-color: rgb(0, 0,
0);"><span style="font-size: 9px;">Brick7:
ib-storage3:/export/brick_home/brick2</span></div>
<div style="margin: 0px; font-family: Menlo; color:
rgb(255, 255, 255); background-color: rgb(0, 0,
0);"><span style="font-size: 9px;">Brick8:
ib-storage4:/export/brick_home/brick2</span></div>
<div style="margin: 0px; font-family: Menlo; color:
rgb(255, 255, 255); background-color: rgb(0, 0,
0);"><span style="font-size: 9px;">Options
Reconfigured:</span></div>
<div style="margin: 0px; font-family: Menlo; color:
rgb(255, 255, 255); background-color: rgb(0, 0,
0);"><span style="font-size: 9px;">features.quota:
on</span></div>
<div style="margin: 0px; font-family: Menlo; color:
rgb(255, 255, 255); background-color: rgb(0, 0,
0);"><span style="font-size: 9px;">diagnostics.brick-log-level:
CRITICAL</span></div>
<div style="margin: 0px; font-family: Menlo; color:
rgb(255, 255, 255); background-color: rgb(0, 0,
0);"><span style="font-size: 9px;">auth.allow:
localhost,127.0.0.1,10.*</span></div>
<div style="margin: 0px; font-family: Menlo; color:
rgb(255, 255, 255); background-color: rgb(0, 0,
0);"><span style="font-size: 9px;">nfs.disable: on</span></div>
<div style="margin: 0px; font-family: Menlo; color:
rgb(255, 255, 255); background-color: rgb(0, 0,
0);"><span style="font-size: 9px;">performance.cache-size:
64MB</span></div>
<div style="margin: 0px; font-family: Menlo; color:
rgb(255, 255, 255); background-color: rgb(0, 0,
0);"><span style="font-size: 9px;">performance.write-behind-window-size:
1MB</span></div>
<div style="margin: 0px; font-family: Menlo; color:
rgb(255, 255, 255); background-color: rgb(0, 0,
0);"><span style="font-size: 9px;">performance.quick-read:
on</span></div>
<div style="margin: 0px; font-family: Menlo; color:
rgb(255, 255, 255); background-color: rgb(0, 0,
0);"><span style="font-size: 9px;">performance.io-cache:
on</span></div>
<div style="margin: 0px; font-family: Menlo; color:
rgb(255, 255, 255); background-color: rgb(0, 0,
0);"><span style="font-size: 9px;">performance.io-thread-count:
64</span></div>
<div style="margin: 0px; font-family: Menlo; color:
rgb(255, 255, 255); background-color: rgb(0, 0,
0);"><span style="font-size: 9px;">features.default-soft-limit:
90%</span></div>
<div><br class="webkit-block-placeholder">
</div>
<div>But, in the cluster, when I try to mount my
volume specifying RDMA transport type, i notice
all my communication go through TCP stack (all
network packet are visible on ib0 network
interface with ifstat shell command), not through
RDMA</div>
<div>
<div style="margin: 0px; font-family: Menlo;
color: rgb(255, 255, 255); background-color:
rgb(0, 0, 0);"><span style="font-size: 9px;">[root@lucifer
~]# mount -t glusterfs -o
transport=rdma,direct-io-mode=disable
localhost:vol_home /home</span></div>
</div>
<div style="margin: 0px; font-family: Menlo; color:
rgb(255, 255, 255); background-color: rgb(0, 0,
0); position: static; z-index: auto;">
<div style="margin: 0px;"><span style="font-size:
9px;">[root@lucifer ~]# mount|grep
vol_home.rdma</span></div>
<div style="margin: 0px;"><span style="font-size:
9px;">localhost:vol_home.rdma on /home type
fuse.glusterfs
(rw,default_permissions,allow_other,max_read=131072)</span></div>
<div style="margin: 0px;">
<div style="margin: 0px;"><span
style="font-size: 9px;">[root@lucifer ~]#
ifstat -i ib0</span></div>
<div style="margin: 0px;"><span
style="font-size: 9px;"> ib0 </span></div>
<div style="margin: 0px;"><span
style="font-size: 9px;"> KB/s in KB/s out</span></div>
<div style="margin: 0px;"><span
style="font-size: 9px;">25313.60 6776.44</span></div>
<div style="margin: 0px;"><span
style="font-size: 9px;">26258.96 9064.92</span></div>
<div style="margin: 0px;"><span
style="font-size: 9px;">28272.97 10034.15</span></div>
<div style="margin: 0px;"><span
style="font-size: 9px;">23495.09 8504.84</span></div>
<div style="margin: 0px;"><span
style="font-size: 9px;">21842.41 7161.69</span></div>
<div style="margin: 0px;"><span
style="font-size: 9px;">^C</span></div>
</div>
</div>
<div><br>
</div>
<div>So, my best noticed throughput is around
400MBs, but basically around 200-250MBs, although
I can read on the net i can expect to achieve
around 800-900MBs -sometimes more- with RDMA
transport type.</div>
<div><br>
</div>
<div>Can anyone help me to make it work?</div>
</div>
</blockquote>
<br>
There is known issue in rdma that volume with
transport type as tcp,rdma will mount as tcp. the fix
for the same is under review. You can pull the patch
from git fetch <a moz-do-not-send="true"
class="moz-txt-link-freetext"
href="https://review.gluster.org/glusterfs">https://review.gluster.org/glusterfs</a>
refs/changes/98/8498/7 && git format-patch -1
FETCH_HEAD. Applying the patch will help to mount the
tcp,rdma volume as rdma. If you are mounting a tcp,rdma
volume as RDMA fuse mount you can also append <small><b><big>.rdma
</big></b><big>with volnam</big><big>e</big><b><big>
</big></b><big>instead of -o option.<br>
<br>
<br>
Let me know your result, If possible I would like
to know the version of gluster you are currently
using.</big><big><b> </b><br>
</big></small> Rafi KC.<br>
<br>
<blockquote
cite="mid:79347B68-EC53-4B62-80F0-298130A41AF6@cnrs.fr"
type="cite">
<div>
<div><br>
</div>
<div>In addition, are my volume settings look like
optimal?</div>
<div><br>
</div>
<div>Thanks in advance,</div>
<div>Geoffrey</div>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
Gluster-users mailing list
<a moz-do-not-send="true" class="moz-txt-link-abbreviated" href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a>
<a moz-do-not-send="true" class="moz-txt-link-freetext" href="http://supercolony.gluster.org/mailman/listinfo/gluster-users">http://supercolony.gluster.org/mailman/listinfo/gluster-users</a></pre>
</blockquote>
<br>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
<br>
</body>
</html>