<html>
<head>
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<br>
<div class="moz-cite-prefix">On 10/08/2014 04:15 AM, Geoffrey
Letessier wrote:<br>
</div>
<blockquote cite="mid:79347B68-EC53-4B62-80F0-298130A41AF6@cnrs.fr"
type="cite">
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252">
Dears,
<div><br>
</div>
<div>I have a HPC cluster composed by 4 storage nodes (8x 24TB
RAID6 bricks, 2 per nodes) and 62 compute nodes, interconnected
via Infiniband QDR technology.</div>
<div><br>
</div>
<div>NB: each brick provide around 1.2-1.5TBs write performances.</div>
<div> </div>
<div>My main volume is defined as below</div>
<div>
<div style="margin: 0px; font-family: Menlo; color: rgb(255,
255, 255); background-color: rgb(0, 0, 0);"><span
style="font-size: 9px;">Volume Name: vol_home</span></div>
<div style="margin: 0px; font-family: Menlo; color: rgb(255,
255, 255); background-color: rgb(0, 0, 0);"><span
style="font-size: 9px;">Type: Distributed-Replicate</span></div>
<div style="margin: 0px; font-family: Menlo; color: rgb(255,
255, 255); background-color: rgb(0, 0, 0);"><span
style="font-size: 9px;">Volume ID:
f6ebcfc1-b735-4a0e-b1d7-47ed2d2e7af6</span></div>
<div style="margin: 0px; font-family: Menlo; color: rgb(255,
255, 255); background-color: rgb(0, 0, 0);"><span
style="font-size: 9px;">Status: Started</span></div>
<div style="margin: 0px; font-family: Menlo; color: rgb(255,
255, 255); background-color: rgb(0, 0, 0);"><span
style="font-size: 9px;">Number of Bricks: 4 x 2 = 8</span></div>
<div style="margin: 0px; font-family: Menlo; color: rgb(255,
255, 255); background-color: rgb(0, 0, 0);"><span
style="font-size: 9px;">Transport-type: tcp,rdma</span></div>
<div style="margin: 0px; font-family: Menlo; color: rgb(255,
255, 255); background-color: rgb(0, 0, 0);"><span
style="font-size: 9px;">Bricks:</span></div>
<div style="margin: 0px; font-family: Menlo; color: rgb(255,
255, 255); background-color: rgb(0, 0, 0);"><span
style="font-size: 9px;">Brick1:
ib-storage1:/export/brick_home/brick1</span></div>
<div style="margin: 0px; font-family: Menlo; color: rgb(255,
255, 255); background-color: rgb(0, 0, 0);"><span
style="font-size: 9px;">Brick2:
ib-storage2:/export/brick_home/brick1</span></div>
<div style="margin: 0px; font-family: Menlo; color: rgb(255,
255, 255); background-color: rgb(0, 0, 0);"><span
style="font-size: 9px;">Brick3:
ib-storage3:/export/brick_home/brick1</span></div>
<div style="margin: 0px; font-family: Menlo; color: rgb(255,
255, 255); background-color: rgb(0, 0, 0);"><span
style="font-size: 9px;">Brick4:
ib-storage4:/export/brick_home/brick1</span></div>
<div style="margin: 0px; font-family: Menlo; color: rgb(255,
255, 255); background-color: rgb(0, 0, 0);"><span
style="font-size: 9px;">Brick5:
ib-storage1:/export/brick_home/brick2</span></div>
<div style="margin: 0px; font-family: Menlo; color: rgb(255,
255, 255); background-color: rgb(0, 0, 0);"><span
style="font-size: 9px;">Brick6:
ib-storage2:/export/brick_home/brick2</span></div>
<div style="margin: 0px; font-family: Menlo; color: rgb(255,
255, 255); background-color: rgb(0, 0, 0);"><span
style="font-size: 9px;">Brick7:
ib-storage3:/export/brick_home/brick2</span></div>
<div style="margin: 0px; font-family: Menlo; color: rgb(255,
255, 255); background-color: rgb(0, 0, 0);"><span
style="font-size: 9px;">Brick8:
ib-storage4:/export/brick_home/brick2</span></div>
<div style="margin: 0px; font-family: Menlo; color: rgb(255,
255, 255); background-color: rgb(0, 0, 0);"><span
style="font-size: 9px;">Options Reconfigured:</span></div>
<div style="margin: 0px; font-family: Menlo; color: rgb(255,
255, 255); background-color: rgb(0, 0, 0);"><span
style="font-size: 9px;">features.quota: on</span></div>
<div style="margin: 0px; font-family: Menlo; color: rgb(255,
255, 255); background-color: rgb(0, 0, 0);"><span
style="font-size: 9px;">diagnostics.brick-log-level:
CRITICAL</span></div>
<div style="margin: 0px; font-family: Menlo; color: rgb(255,
255, 255); background-color: rgb(0, 0, 0);"><span
style="font-size: 9px;">auth.allow: localhost,127.0.0.1,10.*</span></div>
<div style="margin: 0px; font-family: Menlo; color: rgb(255,
255, 255); background-color: rgb(0, 0, 0);"><span
style="font-size: 9px;">nfs.disable: on</span></div>
<div style="margin: 0px; font-family: Menlo; color: rgb(255,
255, 255); background-color: rgb(0, 0, 0);"><span
style="font-size: 9px;">performance.cache-size: 64MB</span></div>
<div style="margin: 0px; font-family: Menlo; color: rgb(255,
255, 255); background-color: rgb(0, 0, 0);"><span
style="font-size: 9px;">performance.write-behind-window-size:
1MB</span></div>
<div style="margin: 0px; font-family: Menlo; color: rgb(255,
255, 255); background-color: rgb(0, 0, 0);"><span
style="font-size: 9px;">performance.quick-read: on</span></div>
<div style="margin: 0px; font-family: Menlo; color: rgb(255,
255, 255); background-color: rgb(0, 0, 0);"><span
style="font-size: 9px;">performance.io-cache: on</span></div>
<div style="margin: 0px; font-family: Menlo; color: rgb(255,
255, 255); background-color: rgb(0, 0, 0);"><span
style="font-size: 9px;">performance.io-thread-count: 64</span></div>
<div style="margin: 0px; font-family: Menlo; color: rgb(255,
255, 255); background-color: rgb(0, 0, 0);"><span
style="font-size: 9px;">features.default-soft-limit: 90%</span></div>
<div><br class="webkit-block-placeholder">
</div>
<div>But, in the cluster, when I try to mount my volume
specifying RDMA transport type, i notice all my communication
go through TCP stack (all network packet are visible on ib0
network interface with ifstat shell command), not through RDMA</div>
<div>
<div style="margin: 0px; font-family: Menlo; color: rgb(255,
255, 255); background-color: rgb(0, 0, 0);"><span
style="font-size: 9px;">[root@lucifer ~]# mount -t
glusterfs -o transport=rdma,direct-io-mode=disable
localhost:vol_home /home</span></div>
</div>
<div style="margin: 0px; font-family: Menlo; color: rgb(255,
255, 255); background-color: rgb(0, 0, 0); position: static;
z-index: auto;">
<div style="margin: 0px;"><span style="font-size: 9px;">[root@lucifer
~]# mount|grep vol_home.rdma</span></div>
<div style="margin: 0px;"><span style="font-size: 9px;">localhost:vol_home.rdma
on /home type fuse.glusterfs
(rw,default_permissions,allow_other,max_read=131072)</span></div>
<div style="margin: 0px;">
<div style="margin: 0px;"><span style="font-size: 9px;">[root@lucifer
~]# ifstat -i ib0</span></div>
<div style="margin: 0px;"><span style="font-size: 9px;">
ib0 </span></div>
<div style="margin: 0px;"><span style="font-size: 9px;"> KB/s
in KB/s out</span></div>
<div style="margin: 0px;"><span style="font-size: 9px;">25313.60
6776.44</span></div>
<div style="margin: 0px;"><span style="font-size: 9px;">26258.96
9064.92</span></div>
<div style="margin: 0px;"><span style="font-size: 9px;">28272.97
10034.15</span></div>
<div style="margin: 0px;"><span style="font-size: 9px;">23495.09
8504.84</span></div>
<div style="margin: 0px;"><span style="font-size: 9px;">21842.41
7161.69</span></div>
<div style="margin: 0px;"><span style="font-size: 9px;">^C</span></div>
</div>
</div>
<div><br>
</div>
<div>So, my best noticed throughput is around 400MBs, but
basically around 200-250MBs, although I can read on the net i
can expect to achieve around 800-900MBs -sometimes more- with
RDMA transport type.</div>
<div><br>
</div>
<div>Can anyone help me to make it work?</div>
</div>
</blockquote>
<br>
There is known issue in rdma that volume with transport type as
tcp,rdma will mount as tcp. the fix for the same is under review.
You can pull the patch from git fetch
<a class="moz-txt-link-freetext" href="https://review.gluster.org/glusterfs">https://review.gluster.org/glusterfs</a> refs/changes/98/8498/7
&& git format-patch -1 FETCH_HEAD. Applying the patch will
help to mount the tcp,rdma volume as rdma. If you are mounting a
tcp,rdma volume as RDMA fuse mount you can also append <small><b><big>.rdma
</big></b><big>with volnam</big><big>e</big><b><big> </big></b><big>instead
of -o option.<br>
<br>
<br>
Let me know your result, If possible I would like to know the
version of gluster you are currently using.</big><big><b> </b><br>
</big></small> Rafi KC.<br>
<br>
<blockquote cite="mid:79347B68-EC53-4B62-80F0-298130A41AF6@cnrs.fr"
type="cite">
<div>
<div><br>
</div>
<div>In addition, are my volume settings look like optimal?</div>
<div><br>
</div>
<div>Thanks in advance,</div>
<div>Geoffrey</div>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
Gluster-users mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a>
<a class="moz-txt-link-freetext" href="http://supercolony.gluster.org/mailman/listinfo/gluster-users">http://supercolony.gluster.org/mailman/listinfo/gluster-users</a></pre>
</blockquote>
<br>
</body>
</html>