<html><head><meta http-equiv="Content-Type" content="text/html charset=windows-1252"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;"><div apple-content-edited="true"><div style="margin: 0cm 0cm 0.0001pt;"><div style="margin: 0cm 0cm 0.0001pt;"><span><div>Carlos:</div><div><br></div><div>The RAM disks are 1GB each, which is way more than needed in my case. There are the really small writes/reads seems o be really costly on the performance with gluster. The storage bricks have 64GB of RAM each. Unfortunately, I haven’t figured out how to export the RAM disks and still be able to export the gluster volumes via NFS. So the RAM disks are currently residing on another server. Using a SSD for scratching wouldn’t make me sleep well at night.. :P</div><div><br></div><div>Yes, rhs stands for Red Hat Storage. Quoted from the Administration Guide regarding rhs-high-throughput:</div></span></div><div style="margin: 0cm 0cm 0.0001pt;"><br></div><div style="margin: 0cm 0cm 0.0001pt;">The profile performs the following:</div><div style="margin: 0cm 0cm 0.0001pt;"><span class="Apple-tab-span" style="white-space:pre"> </span>* Increases read ahead to 64MB</div><div style="margin: 0cm 0cm 0.0001pt;"><span class="Apple-tab-span" style="white-space:pre"> </span>* Changes I/O scheduler to deadline</div><div style="margin: 0cm 0cm 0.0001pt;"><span class="Apple-tab-span" style="white-space:pre"> </span>* Disables power-saving mode</div><div style="margin: 0cm 0cm 0.0001pt;"><span><br class="Apple-interchange-newline" style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;">
</span></div><div style="margin: 0cm 0cm 0.0001pt;">Thanks for the tip regarding the buffers. Tried it and ran some timings with real world applications (tools for transient analysis). But didn’t recognise any noticeable difference in my case. Actually, I don’t believe the storage solution is the bottleneck in this setup atm. Which is great news, especially since I was thinking about throwing it out a window a few weeks back. :D </div><div style="margin: 0cm 0cm 0.0001pt;"><br></div><div style="margin: 0cm 0cm 0.0001pt;">Regards,</div><div style="margin: 0cm 0cm 0.0001pt;">Robin</div>
</div></div><br><div><div>On 11 Mar 2014, at 11:17 am, Carlos Capriotti <<a href="mailto:capriotti.carlos@gmail.com">capriotti.carlos@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1"><div dir="ltr">Robin, would you mind elaborating a bit more on a few details ? This sounds VERY promising.<div><br></div><div>I know I could spend hours on the internet, googlling my way, but that way I'd be loosing about 50% of the objective info.</div>
<div><br></div><div>About the RAM disks, how big are those, and more to the point, how much memory do you have, so we can work out a ratio here. I feel that this RAM disk is a VERY impacting possibility here (positive), and since I am gathering all hints and hacks I can, for a VMware-based environment, THAT one would most likely be a winner ! i would like to test it next week. Maybe using SSD instead ? (not that I have one to spare, of course).</div>
<div><br></div><div>Now, regarding </div><div><span style="font-family:arial,sans-serif;font-size:13px"><br></span></div><div><span style="font-family:arial,sans-serif;font-size:13px"><b>'tuned-adm profile; tuned-adm profile rhs-high-throughput’ on all storage bricks</b></span><br>
</div><div><span style="font-family:arial,sans-serif;font-size:13px"><br></span></div><div> I cannot remember seeing those option anywhere. Where did you find them ? Does "rhs" stand for Red Hat Storage ?</div><div>
<br></div><div><br></div><div>One last comment that might help: when mounting NFS, tweak the buffers, and async operations with this:</div><div><br></div><div>-o rw,async,vers=3,rsize=65536,wsize=65536<br></div><div><br></div>
<div>KR,</div><div><br></div><div>Carlos</div><div><br></div><div><br></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Tue, Mar 11, 2014 at 10:37 AM, Robin Jonsson <span dir="ltr"><<a href="mailto:Robin.Jonsson@enfo.se" target="_blank">Robin.Jonsson@enfo.se</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word"><div>Alexander:<div><br></div><div>I have also experienced the stalls you are explaining. This was in a 2 brick setup running replicated volumes used by a 20 node HPC. </div>
<div><br></div><div>In my case this was solved by: </div><div><br></div><div>* Replace FUSE with NFS</div><div><span style="white-space:pre-wrap"> </span>* This is by far the biggest booster</div><div>* RAM disks for the scratch directories (not connected to gluster at all)</div>
<div><span style="white-space:pre-wrap"> </span>* If you’re not sure where these directories are, run ‘gluster volume top <volume> write list-cnt 10’</div><div>* 'tuned-adm profile; tuned-adm profile rhs-high-throughput’ on all storage bricks</div>
<div>* The following volume options</div><div><span style="white-space:pre-wrap"> </span>* cluster.nufa: enable</div><div><span style="white-space:pre-wrap"> </span>* performance.quick-read: on</div><div><span style="white-space:pre-wrap"> </span>* performance.open-behind: on</div>
<div>* Mount option on clients</div><div><span style="white-space:pre-wrap"> </span>* noatime</div><div><span style="white-space:pre-wrap"> </span>* Use only where access time isn’t needed.</div><div><span style="white-space:pre-wrap"> </span>* Major booster for small file writes in my case. Even with the FUSE client.</div>
<div><br></div><div>Hope this helps, </div><div><br></div><div>Regards,</div><div>Robin</div></div><div><div class="h5"><div><div style="margin:0cm 0cm 0.0001pt"><div style="margin:0cm 0cm 0.0001pt"><span><br style="text-indent:0px;letter-spacing:normal;font-variant:normal;text-align:start;font-style:normal;font-weight:normal;line-height:normal;text-transform:none;font-size:12px;white-space:normal;font-family:Helvetica;word-spacing:0px">
</span></div>
</div></div><br><div><div>On 10 Mar 2014, at 19:06 pm, Alexander Valys <<a href="mailto:avalys@avalys.net" target="_blank">avalys@avalys.net</a>> wrote:</div><br><blockquote type="cite">A quick performance question.<br>
<br>I have a small cluster of 4 machines, 64 cores in total. I am running a scientific simulation on them, which writes at between 0.1 and 10 MB/s (total) to roughly 64 HDF5 files. Each HDF5 file is written by only one process. The writes are not continuous, but consist of writing roughly 1 MB of data to each file every few seconds. <br>
<br>Writing to HDF5 involves a lot of reading the file metadata and random seeking within the file, since we are actually writing to about 30 datasets inside each file. I am hosting the output on a distributed gluster volume (one brick local to each machine) to provide a unified namespace for the (very rare) case when each process needs to read the other's files. <br>
<br>I am seeing somewhat lower performance than I expected, i.e. a factor of approximately 4 less throughput than each node writing locally to the bare drives. I expected the write-behind cache to buffer each write, but it seems that the writes are being quickly flushed across the network regardless of what write-behind cache size I use (32 MB currently), and the simulation stalls while waiting for the I/O operation to finish. Anyone have any suggestions as to what to look at? I am using gluster 3.4.2 on ubuntu 12.04. I have flush-behind turned on, and have mounted the volume with direct-io-mode=disable, and have the cache size set to 256M. <br>
<br>The nodes are connected via a dedicated gigabit ethernet network, carrying only gluster traffic (no simulation traffic).<br><br>(sorry if this message comes through twice, I sent it yesterday but was not subscribed)<br>
_______________________________________________<br>Gluster-users mailing list<br><a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br><a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://supercolony.gluster.org/mailman/listinfo/gluster-users</a><br>
</blockquote></div><br></div></div></div><br>_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
<a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://supercolony.gluster.org/mailman/listinfo/gluster-users</a><br></blockquote></div><br></div>
</blockquote></div><br></body></html>