<html><head><meta http-equiv="Content-Type" content="text/html charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">Dears,<div><br></div><div>I have a HPC cluster composed by 4 storage nodes (8x 24TB RAID6 bricks, 2 per nodes) and 62 compute nodes, interconnected via Infiniband QDR technology.</div><div><br></div><div>NB: each brick provide around 1.2-1.5TBs write performances.</div><div> </div><div>My main volume is defined as below</div><div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">Volume Name: vol_home</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">Type: Distributed-Replicate</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">Volume ID: f6ebcfc1-b735-4a0e-b1d7-47ed2d2e7af6</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">Status: Started</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">Number of Bricks: 4 x 2 = 8</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">Transport-type: tcp,rdma</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">Bricks:</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">Brick1: ib-storage1:/export/brick_home/brick1</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">Brick2: ib-storage2:/export/brick_home/brick1</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">Brick3: ib-storage3:/export/brick_home/brick1</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">Brick4: ib-storage4:/export/brick_home/brick1</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">Brick5: ib-storage1:/export/brick_home/brick2</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">Brick6: ib-storage2:/export/brick_home/brick2</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">Brick7: ib-storage3:/export/brick_home/brick2</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">Brick8: ib-storage4:/export/brick_home/brick2</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">Options Reconfigured:</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">features.quota: on</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">diagnostics.brick-log-level: CRITICAL</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">auth.allow: localhost,127.0.0.1,10.*</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">nfs.disable: on</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">performance.cache-size: 64MB</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">performance.write-behind-window-size: 1MB</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">performance.quick-read: on</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">performance.io-cache: on</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">performance.io-thread-count: 64</span></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">features.default-soft-limit: 90%</span></div><div><br class="webkit-block-placeholder"></div><div>But, in the cluster, when I try to mount my volume specifying RDMA transport type, i notice all my communication go through TCP stack (all network packet are visible on ib0 network interface with ifstat shell command), not through RDMA</div><div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);"><span style="font-size: 9px;">[root@lucifer ~]# mount -t glusterfs -o transport=rdma,direct-io-mode=disable localhost:vol_home /home</span></div></div><div style="margin: 0px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0); position: static; z-index: auto;"><div style="margin: 0px;"><span style="font-size: 9px;">[root@lucifer ~]# mount|grep vol_home.rdma</span></div><div style="margin: 0px;"><span style="font-size: 9px;">localhost:vol_home.rdma on /home type fuse.glusterfs (rw,default_permissions,allow_other,max_read=131072)</span></div><div style="margin: 0px;"><div style="margin: 0px;"><span style="font-size: 9px;">[root@lucifer ~]# ifstat -i ib0</span></div><div style="margin: 0px;"><span style="font-size: 9px;"> ib0 </span></div><div style="margin: 0px;"><span style="font-size: 9px;"> KB/s in KB/s out</span></div><div style="margin: 0px;"><span style="font-size: 9px;">25313.60 6776.44</span></div><div style="margin: 0px;"><span style="font-size: 9px;">26258.96 9064.92</span></div><div style="margin: 0px;"><span style="font-size: 9px;">28272.97 10034.15</span></div><div style="margin: 0px;"><span style="font-size: 9px;">23495.09 8504.84</span></div><div style="margin: 0px;"><span style="font-size: 9px;">21842.41 7161.69</span></div><div style="margin: 0px;"><span style="font-size: 9px;">^C</span></div></div></div><div><br></div><div>So, my best noticed throughput is around 400MBs, but basically around 200-250MBs, although I can read on the net i can expect to achieve around 800-900MBs -sometimes more- with RDMA transport type.</div><div><br></div><div>Can anyone help me to make it work?</div><div><br></div><div>In addition, are my volume settings look like optimal?</div><div><br></div><div>Thanks in advance,</div><div>Geoffrey</div></div></body></html>