<div dir="ltr">Did you tweak some options belonging to the performance translator, such as io-thread-count? If not, try to increase it to 64 from 16 (default).<br><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Dec 8, 2014 at 12:10 PM, Andrew Smith <span dir="ltr"><<a href="mailto:smith.andrew.james@gmail.com" target="_blank">smith.andrew.james@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><br>
QDR Infiniband has a max theoretical input of 40Gbits, or about 4GB/s.<br>
My LSI controller RAID controllers typically deliver about 0.5-1.0 GB/s<br>
for direct disk access.<br>
<br>
I have tested it many ways. I typically start jobs on many clients and<br>
measure the total network bandwidth on the servers by monitoring the<br>
totals in /proc/net/dev or just count the bytes on the clients. I can’t<br>
get more than about 300MB/s from each server. With a single job on<br>
a single client, I can’t get more than about 100-150MB/s.<br>
<div class="HOEnZb"><div class="h5"><br>
On Dec 7, 2014, at 9:15 PM, Franco Broi <<a href="mailto:franco.broi@iongeo.com">franco.broi@iongeo.com</a>> wrote:<br>
<br>
><br>
> Our theoretical peak throughput is about 4Gbytes/sec or 4 x 10Gbits/Sec,<br>
> you can see from the graph that the maximum recorded is 3.6GB/Sec. This<br>
> was probably during periods of large sequential IO.<br>
><br>
> We have a small cluster of clients (10) with 10Gbit ethernet but the<br>
> majority of our machines (130) have gigabit. The throughput maximum for<br>
> the 10Gbit connected machines was just over 3GBytes/Sec with individual<br>
> machines recording about 800MB/Sec.<br>
><br>
> We can easily saturate our 10Gbit links on the servers as each JBOD is<br>
> capable of better than 500MB/Sec but with mixed sequential/random access<br>
> it seems like a good compromise.<br>
><br>
> We have another 2 server Gluster system with the same specs and we get<br>
> 1.8GB/Sec reads and 1.1GB/Sec writes.<br>
><br>
> What are you using to measure your throughput?<br>
><br>
> On Sun, 2014-12-07 at 20:52 -0500, Andrew Smith wrote:<br>
>> I have a similar system with 4 nodes and 2 bricks per node, where<br>
>> each brick is a single large filesystem (4TB x 24 RAID 6). The<br>
>> computers are all on QDR Infinband with Gluster using IPOIB. I<br>
>> have a cluster of Infiniband clients that access the data on the<br>
>> servers. I can only get about 1.0 to 1.2 GB/s throughput with my<br>
>> system though. Can you tell us the peak throughput that you are<br>
>> getting. I just don’t have a sense of what I should expect from<br>
>> my system. A similar Luster setup could achieve 2-3 GB/s, which<br>
>> I attributed to the fact that it didn’t use IPOIB, but instead used<br>
>> RDMA. I’d really like to know if I am wrong here and there is<br>
>> some configuration I can tweak to make things faster.<br>
>><br>
>> Andy<br>
>><br>
>> On Dec 7, 2014, at 8:43 PM, Franco Broi <<a href="mailto:franco.broi@iongeo.com">franco.broi@iongeo.com</a>> wrote:<br>
>><br>
>>> On Fri, 2014-12-05 at 14:22 +0000, Kiebzak, Jason M. wrote:<br>
>>>> May I ask why you chose to go with 4 separate bricks per server rather than one large brick per server?<br>
>>><br>
>>> Each brick is a JBOD with 16 disks running RAIDZ2. Just seemed more<br>
>>> logical to keep the bricks and ZFS filesystems confined to physical<br>
>>> hardware units, ie I could disconnect a brick and move it to another<br>
>>> server.<br>
>>><br>
>>>><br>
>>>> Thanks<br>
>>>> Jason<br>
>>>><br>
>>>> -----Original Message-----<br>
>>>> From: <a href="mailto:gluster-users-bounces@gluster.org">gluster-users-bounces@gluster.org</a> [mailto:<a href="mailto:gluster-users-bounces@gluster.org">gluster-users-bounces@gluster.org</a>] On Behalf Of Franco Broi<br>
>>>> Sent: Thursday, December 04, 2014 7:56 PM<br>
>>>> To: <a href="mailto:gluster-users@gluster.org">gluster-users@gluster.org</a><br>
>>>> Subject: [Gluster-users] A year's worth of Gluster<br>
>>>><br>
>>>><br>
>>>> 1 DHT volume comprising 16 50TB bricks spread across 4 servers. Each server has 10Gbit Ethernet.<br>
>>>><br>
>>>> Each brick is a ZOL RADIZ2 pool with a single filesystem.<br>
>>><br>
>>><br>
>>> _______________________________________________<br>
>>> Gluster-users mailing list<br>
>>> <a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
>>> <a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://supercolony.gluster.org/mailman/listinfo/gluster-users</a><br>
>><br>
><br>
><br>
<br>
_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
<a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://supercolony.gluster.org/mailman/listinfo/gluster-users</a><br>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature">Nguyen Viet Cuong<br></div>
</div></div>