<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"><html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta content="text/html; charset=UTF-8" http-equiv="Content-Type"/>
</head>
<body>
<p style="margin: 0pt;">
<span>
<span></span>
Hi all,
</span>
</p>
<p style="margin: 0pt;">
<span> </span>
</p>
<p style="margin: 0pt;">
<span>i am currently in the process of deploying gluster as a storage/scratch file system for a new HPC cluster.</span>
</p>
<p style="margin: 0pt;">
<span> </span>
</p>
<p style="margin: 0pt;">
<span>For storage I use HP storage arrays (12x2 TB disks, formatted with xfs, plain vanilla options)</span>
</p>
<p style="margin: 0pt;">
<span>Performance seems to be ok as I am getting > 800 MB/sec when using hpdarm and "dd < /dev/zero > /path/to/storage/file bs=1024k count=100"</span>
</p>
<p style="margin: 0pt;">
<span> </span>
</p>
<p style="margin: 0pt;">
<span>The Infiniband fabric consists of QLE7342 cards and run the latest QLogic OFED (based on stock 1.5.3)</span>
</p>
<p style="margin: 0pt;">
<span>Performance seems to be ok here as well as with osu_bw benchmarks I am reaching 3.2 GB/s uni-directionally.</span>
</p>
<p style="margin: 0pt;">
<span>iperf reports 15 Gbps for ipoib which I think is not too bad either(connected mode, MTU 65520).</span>
</p>
<p style="margin: 0pt;">
<span> </span>
</p>
<p style="margin: 0pt;">
<span>The servers all run RHEL 5.8 and comprise of 2 X5690 CPUs and 24 GB RAM.    </span>
</p>
<p style="margin: 0pt;">
<span> </span>
</p>
<p style="margin: 0pt;">
<span>Now, if I am creating a new volume locally (transport tcp,rdma) using one brick (about 8 TB size) on one of the storage hosts and mount it on the same hosts as a gluster mount (rdma or non-rdma does not matter), the read/write performance does not exceed 400 MB/s (doing the same simple dd test as above). Same is true if I am mounting it on another node. That means I am somehow missing about a factor of 2 in performance.  </span>
</p>
<p style="margin: 0pt;">
<span> </span>
</p>
<p style="margin: 0pt;">
<span>I have been reading through the mailing list and the documentation as well and tried various options (tuning the storage, using various options with the gluster volume, etc...) but   </span>
</p>
<p style="margin: 0pt;">
<span> </span>
</p>
<p style="margin: 0pt;">
<span>What could be the problem here ? Any pointers would be appreciated.</span>
</p>
<p style="margin: 0pt;">
<span> </span>
</p>
<p style="margin: 0pt;">
<span>Many thanks,</span>
</p>
<p style="margin: 0pt;">
<span> </span>
</p>
<p style="margin: 0pt;">
<span>Michael.  </span>
</p>
<p style="margin: 0pt;">
<span> </span>
</p>
<p style="margin: 0pt;">
<span> </span>
</p>
<p style="margin: 0pt;">
<span> </span>
</p>
<p style="margin: 0px; "></p>
</body>
</html>