<html><head><meta http-equiv="Content-Type" content="text/html charset=windows-1252"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;"><div>Alexander:<div><br></div><div>I have also experienced the stalls you are explaining. This was in a 2 brick setup running replicated volumes used by a 20 node HPC. </div><div><br></div><div>In my case this was solved by: </div><div><br></div><div>* Replace FUSE with NFS</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>* This is by far the biggest booster</div><div>* RAM disks for the scratch directories (not connected to gluster at all)</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>* If you’re not sure where these directories are, run ‘gluster volume top <volume> write list-cnt 10’</div><div>* 'tuned-adm profile; tuned-adm profile rhs-high-throughput’ on all storage bricks</div><div>* The following volume options</div><div><span class="Apple-tab-span" style="white-space: pre;"> </span>* cluster.nufa: enable</div><div><span class="Apple-tab-span" style="white-space: pre;"> </span>* performance.quick-read: on</div><div><span class="Apple-tab-span" style="white-space: pre;"> </span>* performance.open-behind: on</div><div>* Mount option on clients</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>* noatime</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>* Use only where access time isn’t needed.</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>* Major booster for small file writes in my case. Even with the FUSE client.</div><div><br></div><div>Hope this helps, </div><div><br></div><div>Regards,</div><div>Robin</div></div><div apple-content-edited="true"><div style="margin: 0cm 0cm 0.0001pt;"><div style="margin: 0cm 0cm 0.0001pt;"><span><br class="Apple-interchange-newline" style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;">
</span></div>
</div></div><br><div><div>On 10 Mar 2014, at 19:06 pm, Alexander Valys <<a href="mailto:avalys@avalys.net">avalys@avalys.net</a>> wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite">A quick performance question.<br><br>I have a small cluster of 4 machines, 64 cores in total. I am running a scientific simulation on them, which writes at between 0.1 and 10 MB/s (total) to roughly 64 HDF5 files. Each HDF5 file is written by only one process. The writes are not continuous, but consist of writing roughly 1 MB of data to each file every few seconds. <br><br>Writing to HDF5 involves a lot of reading the file metadata and random seeking within the file, since we are actually writing to about 30 datasets inside each file. I am hosting the output on a distributed gluster volume (one brick local to each machine) to provide a unified namespace for the (very rare) case when each process needs to read the other's files. <br><br>I am seeing somewhat lower performance than I expected, i.e. a factor of approximately 4 less throughput than each node writing locally to the bare drives. I expected the write-behind cache to buffer each write, but it seems that the writes are being quickly flushed across the network regardless of what write-behind cache size I use (32 MB currently), and the simulation stalls while waiting for the I/O operation to finish. Anyone have any suggestions as to what to look at? I am using gluster 3.4.2 on ubuntu 12.04. I have flush-behind turned on, and have mounted the volume with direct-io-mode=disable, and have the cache size set to 256M. <br><br>The nodes are connected via a dedicated gigabit ethernet network, carrying only gluster traffic (no simulation traffic).<br><br>(sorry if this message comes through twice, I sent it yesterday but was not subscribed)<br>_______________________________________________<br>Gluster-users mailing list<br><a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>http://supercolony.gluster.org/mailman/listinfo/gluster-users<br></blockquote></div><br></body></html>