<div dir="ltr">Ah ! I love the smell of disk benchmark in the morning. :) <div><br></div><div>Well, just kidding, but this is one of those questions, and I myself was also "pretty sure" I had more than enough I/O, just because the label on the disk said 15K RPM, 3 Go/s, with brand hardware and reputable controllers.</div>
<div><br></div><div>Turns out my "nice" setup performed like (well, I leave it to you to fill in the gaps).</div><div><br></div><div>Start from scratch. Test your disk setup with dd.</div><div><br></div><div>Try to match your controller's stripe size with your filesystem's block size AND dd's block size when using it.</div>
<div><br></div><div>This is a little of alchemy, and try not killing me too much for some mix-ups, but it is all worth a shot:</div><div><br></div><div>For instance, prepare you RAID volume with 512 K stripe size. In theory, if you will deal with lots of small files, make this number smaller, and vice-versa. </div>
<div><br></div><div>Next, don't forget to prepare you filesystem with XFS and -i 512. Not really performance-related, but it is in the manual. Important for Gluster.</div><div><br></div><div>Next, the DD black magic:</div>
<div><br></div><div>dd if=/dev/zero of=/your_volume_mount_point/output_file.txt bs=512k count=SOME_NUMBER</div><div><br></div><div>(double-check on the 512k, it could be 512kb or similar)</div><div><br></div><div>Now, here is how to decide what is SOME_NUMBER:</div>
<div><br></div><div>a) It will write SOME_NUMBER times 512KB of zeroes to your file. Thus, if you use 2, it will make a 1 MByte file.</div><div><br></div><div>b) you want your output file to be a little bigger than the amount of RAM on your server, to avoid caching to RAM. Normally this will also avoid caching to the controller's memory. If you have 8 GB of RAM, you will need to use something like 17000.</div>
<div><br></div><div>If you ask me why not making BS=1MB, I can only point back to the "black magic part" and explain that I am matching the 512s. Using 512 everywhere.</div><div><br></div><div>So, after a few minutes, dd should - depending on the version you have on your distro - give you a summary of the operation; I recommend also using </div>
<div><br></div><div>time dd ........</div><div><br></div><div>in order to have an idea of time taken by the operation.</div><div><br></div><div>This is a very poor way of testing performance, I know, but it is a very good start.</div>
<div><br></div><div>Also, take your time and test your server, to see if it can handle that sort of workload, again with dd:</div><div><br></div><div>dd if=/dev/zero of=/dev/null bs=512kb count=whatever_you_want_but_go_big</div>
<div><br></div><div>It will give you a rough idea of how fast your system can process information, of sorts. And see if it is able to cope with your theoretical 10 Gb network ! Mine isn't; Disk controllers will not go over 4 Gb, no matter the disk setup.</div>
<div><br></div><div>If you happen to find out that your disks are performing well, then repeat this for NFS mounts and native fuse/glusterfs mounts.</div><div><br></div><div>You might want to test your network before that, true.</div>
<div><br></div><div>Tip: make sure your network is "uncluttered" (switches and NICs) and set to jumbo frames (mtu 9000). When configuring my back end, I set the switch (dedicated to storage) as if I were using iSCSI over it. Disabled all the phone crap (oh, there it goes, I finally said it) and extra controls. </div>
<div><br></div><div>Now, sharing a bit more of "black magic", I've recently learnt that higher temperature (on network cables) can affect performance. I don't expect that to be an issue on a SPF+ style cable like you are - probably - using, but, after reading that I went to my rack only to find out that all of my cables were just behind the bloody power supplies from the servers, and they were very hot and soft.</div>
<div><br></div><div>Guess what: 8 hours downtime to redo all network (1 Gb "regular") and power cabling, which now is all nice, and even has a thermal insulation.</div><div><br></div><div>Well, if there was any performance improvement ? I cannot say, but at least this is something else to take into consideration.</div>
<div><br></div><div>Cheers.</div><div><br></div><div><br></div><div><br></div><div><br></div><div><div><br></div></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, Jun 12, 2014 at 10:40 PM, Aronesty, Erik <span dir="ltr"><<a href="mailto:earonesty@expressionanalysis.com" target="_blank">earonesty@expressionanalysis.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div lang="EN-US" link="blue" vlink="purple">
<div>
<p class="MsoNormal">I suspect I'm having performance issues because of network speeds.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><i>Supposedly</i> I have 10gbit connections on all my NAS devices, however, it seems to me that the fastest I can write is 1Gbit. When I'm copying very large files, etc, I see 'D' as the cp waits to I/O, but when I go the gluster servers,
I don't see glusterfsd waiting (D) to write to the bricks themselves. I have 4 nodes, each with 10Gbit connection, each has 2 Areca RAID controllers with 12 disk raid5, and the 2 controllers stripped into 1 large volume. Pretty sure there's plenty of i/o
left on the bricks themselves.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Is it possible that "one big file" isn't the right test… should I try 20 big files, and see how saturated my network can get?<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="font-size:8.0pt;font-family:"Arial","sans-serif"">Erik Aronesty
<br>
Senior Bioinformatics Architect</span><span style="font-size:5.0pt;font-family:"Arial","sans-serif""><u></u><u></u></span></p>
<p class="MsoNormal" style="margin-bottom:12.0pt"><b><span style="font-size:10.0pt;font-family:"Georgia","serif";color:#5f6062">EA | Quintiles<br>
</span></b><b><i><span style="font-size:8.0pt;font-family:"Arial","sans-serif";color:#d31145">Genomic Services</span></i></b><b><i><span style="font-size:5.0pt;font-family:"Arial","sans-serif";color:#ec6e30"><u></u><u></u></span></i></b></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Arial","sans-serif"">4820 Emperor Boulevard<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Arial","sans-serif"">Durham, NC 27703 USA<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="font-size:8.0pt;font-family:"Arial","sans-serif";color:#1b4579"><br>
</span><span style="font-size:8.0pt;font-family:"Arial","sans-serif"">Office: + <a href="tel:919.287.4011" value="+19192874011" target="_blank">919.287.4011</a><br>
</span><a href="mailto:kmichailo@expressionanalysis.com" target="_blank"><span style="font-size:8.0pt">erik.aronesty@quintiles.com</span></a><span style="font-size:8.0pt;color:#1b4579"><br>
</span><span style="font-size:5.0pt;font-family:"Arial","sans-serif";color:#1b4579"><br>
</span><a href="http://www.quintiles.com/" target="_blank"><span style="font-size:8.0pt">www.quintiles.com</span></a><span style="font-size:8.0pt;font-family:"Arial","sans-serif";color:#1b4579"> </span><span style="font-size:5.0pt;font-family:"Arial","sans-serif";color:#1b4579"><br>
</span><a href="http://www.expressionanalysis.com/" target="_blank"><span style="font-size:8.0pt">www.expressionanalysis.com</span></a><span style="font-size:8.0pt;color:#1b4579"><br>
</span><a href="https://www.twitter.com/simulx" target="_blank"><span style="color:windowtext;text-decoration:none"><img border="0" width="16" height="16" src="cid:image001.jpg@01CF865D.0BE43760" alt="cid:image001.jpg@01CDEF4B.84C3E9F0"></span></a><span style="color:#1c467a"> </span><a href="http://www.facebook.com/aronesty" target="_blank"><span style="color:windowtext;text-decoration:none"><img border="0" width="16" height="16" src="cid:image002.jpg@01CF865D.0BE43760" alt="cid:image002.jpg@01CDEF4B.84C3E9F0"></span></a><span style="color:#1c467a"> </span><a href="http://www.linkedin.com/in/earonesty" target="_blank"><span style="color:windowtext;text-decoration:none"><img border="0" width="16" height="16" src="cid:image003.jpg@01CF865D.0BE43760" alt="cid:image003.jpg@01CDEF4B.84C3E9F0"></span></a><span style="font-size:8.0pt;color:#1b4579"><u></u><u></u></span></p>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div>
<br>_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
<a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://supercolony.gluster.org/mailman/listinfo/gluster-users</a><br></blockquote></div><br></div>