HI, <br> Is there a way to get good performance when an application does small writes?<br>Most of the apllications using NetCDF write big files (upto 100GB) but using small block-sized writes(Block size less than 1KB)<br>
<br>--------------------------------------------------------------------------------------------------<br>[root@cola5 scripts]# dd if=/dev/zero of=/h1/junk bs=512 count=1024000<br>1024000+0 records in<br>1024000+0 records out<br>
524288000 bytes (524 MB) copied, 70.7706 seconds, 7.4 MB/s<br>[root@cola5 scripts]# dd if=/dev/zero of=/h1/junk bs=1k count=1024000<br>1024000+0 records in<br>1024000+0 records out<br>1048576000 bytes (1.0 GB) copied, 59.6961 seconds, 17.6 MB/s<br>
[root@cola5 scripts]# dd if=/dev/zero of=/h1/junk bs=16k count=64000<br>64000+0 records in<br>64000+0 records out<br>1048576000 bytes (1.0 GB) copied, 4.42826 seconds, 237 MB/s<br>-----------------------------------------------------------------------------------------------<br>
<br>For very small block-sized writes write-behind does not seem to help? How to improve small write<br>caching?<br>Al<br><br><br><br><br><br><br><br><div class="gmail_quote">On Mon, Jun 4, 2012 at 11:13 AM, Raghavendra G <span dir="ltr"><<a href="mailto:raghavendra@gluster.com" target="_blank">raghavendra@gluster.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi,<br><br>The purpose of performance translators is to decrease system call latency of applications and increase responsiveness of glusterfs. <br>
<br>The standard approach used within glusterfs to decrease system call latency is making sure we avoid network roundtrip time as part of the fop processing. And based on what fop we are dealing with, we have different translators like read-ahead, io-cache, write-behind, quick-read, md-cache.<br>
<ul><li>Though read-ahead and io-cache both serve read calls, the difference lies in that read-ahead can even serve first read on an offset (since it would have read-ahead on a read with lesser offset) and io-cache can serve only requests after first read on an offset from its cache. read-ahead can have negative performance impact in the form of cache maintanence on random reads.In case of read-ahead, cache is maintained per-fd basis and io-cache maintains per-inode cache. Ceiling for cache limits can be configured.<br>
</li><li>write-behind takes the responsibility of storing writes in its cache and syncing it with disk in background. Because of this fact, we may not able to find out the fate of a write from an application in return value of that write. However write-behind communicates errors to application either in return value of current or future writes or close call. Paranoid applications which need to know errors during any writes previously done, should do an fsync. There is another option flush-behind which when turned on, makes flush calls sent as part of close, background. The consequence of doing flush in background is that posix locks on that fd might not be cleared as soon as close returns.</li>
<li>quick-read optimizes reads by storing small files in its cache. It gets the contents of entire file as part of lookup call done during path to inode conversion. It assumes that all opens are done with an intention of doing reads and hence doesn't really send open to translators below if the file is cached. However, it maintains the abstraction by doing open as part of other fd based fops (like fstat, fsync etc). Because of this, read-intensive applications like a web-server serving lots of small files, can save network round trip for two fops - open and read (It used to save close roundtrip call too, but with close implemented as a callback of fd-destroy, network roundtrip time is eliminated altogether).</li>
<li>md-cache is a translator that caches metadata like stats, certain extended attributes of files.<br></li></ul>One of the strategies to increase responsiveness is to introduce asynchronous nature - one doesn't block on a single operation to complete before taking another - during fop processing. Again asynchronous nature can be achieved using single or multiple threads. The first approach is effective only when there are blocking components in the system, like I/O with network or disk. Performance translators does not do anything helpful in this aspect (STACK_WIND and STACK_UNWIND macros, non-blocking sockets etc help here). It is in introducing parallel processing as call proceeds through gluster translator graph where io-threads (a performance translator) comes into picture. Apart from introducing parallelism, io-threads implements priority based processing of fops, which helps to increase responsiveness. There are other threads within a glusterfs process which are not maintained by io-threads like fuse-reader, posix janitor, a thread which polls on network sockets, threads processing send/receive completion queues in infiniband, threads introduced by syncops, thread processing timer events etc.<br>
<br>regards,<span class="HOEnZb"><font color="#888888"><br><div class="gmail_quote">-- <br>Raghavendra G<br><br>
<br><br>
</div>
</font></span><br>_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
<a href="http://gluster.org/cgi-bin/mailman/listinfo/gluster-users" target="_blank">http://gluster.org/cgi-bin/mailman/listinfo/gluster-users</a><br>
<br></blockquote></div><br>