Hi Daniel,<br><br>We ran some performance tests in our lab, with reasonable cache-size for io-cache (so that it won't consume all the memory present in the system). <br><br>setup: A simple client/server configuration with/without io-cache on client side.<br>
====<br><br>Here are the results.<br><br>without io-cache:<br>=============<br>[root@client05 test]# time bonnie++ -u anush<br>Using uid:1013, gid:1013.<br>Version 1.03c ------Sequential Output------ --Sequential Input- --Random-<br>
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--<br>Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP<br>client05 4G 24972 36 28187 1 13149 1 26898 40 30860 1 234.5 0<br>
------Sequential Create------ --------Random Create--------<br> -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--<br> files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP<br>
16 1226 2 4956 2 1806 2 1201 3 5522 3 1874 3<br><br>with io-cache:<br>===========<br>Version 1.03c ------Sequential Output------ --Sequential Input- --Random-<br> -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--<br>
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP<br>client05 4G 24920 37 28529 1 4576 0 36690 52 61428 3 186.2 0<br> ------Sequential Create------ --------Random Create--------<br>
-Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--<br> files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP<br> 16 1230 2 4945 3 1816 3 1134 2 5576 5 1906 3<br>
<br>As you can see the test which is most affected by the presence of io-cache is Rewrite. During rewrite, bonnie++ reads an 8kb block, dirties it and writes back to same location in file. This test consists of 3 system calls - read, lseek and write. Since cache is flushed at each write, effectively data is always read from backend filesystem. However the difference is that, since io-cache stores data in pages of 128KB each, it also does reads in terms of 128KB block sizes. This it does in the hope that the overhead involved in reading 128KB blocks (instead of in block-sizes what fuse has asked) is acceptable, since successive reads can be fetched from cache. Hence every read of 8KB block size from bonnie++ is transformed into a read of 128KB block-size is io-cache is present. This is the cause of lower performance when io-cache is present during rewrites.<br>
<br>To confirm what we had suspected, we changed the page-size in io-cache to 8KB and following are the results.<br>without io-cache:<br>=============<br>[root@client05 test]# bonnie++ -u anush<br>Version 1.03c ------Sequential Output------ --Sequential Input- --Random-<br>
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--<br>Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP<br>client05 4G 24917 35 28243 1 13130 1 27295 39 30836 1 239.8 0<br>
------Sequential Create------ --------Random Create--------<br> -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--<br> files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP<br>
16 1224 2 4949 2 1798 2 1186 2 5509 3 1876 2<br><br>with io-cache:<br>==========<br>[root@client05 test]# bonnie++ -u anush <br>
Version 1.03c ------Sequential Output------ --Sequential Input- --Random-<br>
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--<br>Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP<br>client05 4G 24925 36 28019 2 12786 1 31224 44 28096 1 220.8 0<br>
------Sequential Create------ --------Random Create--------<br> -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--<br> files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP<br>
16 1192 3 4897 2 1784 3 1215 3 5477 4 1842 3<br><br>As we can see, there is not much difference in Rewrite performance with/without io-cache. Hence if there are writes to the same offsets from which reads have been done, data will always be read from back-end and overhead of io-cache can be significant. In these scenarios io-cache is not recommended. However for other cases, such as mostly read-only operations, performance boost from io-cache can be significant.<br>
<br>regards,<br><div class="gmail_quote">On Tue, Mar 2, 2010 at 1:30 PM, Daniel Maher <span dir="ltr"><<a href="mailto:dma%2Bgluster@witbe.net" target="_blank">dma+gluster@witbe.net</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><div>Daniel Maher wrote:<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
Raghavendra G wrote:<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<br>
However, at two points during the multi-day test run, something<br>
strange happened. The time to completion dropped _dramatically_,<br>
and stayed there for numerous iterations, before jumping back up again :<br>
<br>
<br>
Mostly reads are being served from io-cache?<br>
</blockquote>
<br>
Perhaps ; it is worth noting that even though the operations are consistent, the data are being generated randomly. I concede that, statistically speaking, some of those 0's and 1's would be cached effectively, but this shouldn't account for a sudden ~ 50% increase in efficiency that, just as suddenly as it appears, disappears again.<br>
<br>
While it is irresponsible to extrapolate based on three points, my newest test run with io-cache disabled has yielded 10m30s, 10m36s, and 10m34s so far...<br>
</blockquote>
<br></div>
After hundreds of iterations the average « real » time per run was 10m25.522s . This was with io-cache totally disabled.<br>
<br>
Thus, it has been shown that given a series of systematic read and write operations on progressively larger files filled with random data, the usage of io-cache is not appropriate (and will cause severe performance problems).<br>
<br>
Of course, one could have postulated this intuitively - but there's nothing like some hard data to back up a hypothesis. :)<br>
<br>
The real mystery is why the test with a small io-cache yielded two groups of highly varient TTCs...<div><div></div><div><br>
<br>
<br>
-- <br>
Daniel Maher <dma+gluster AT witbe DOT net><br>
<br>
<br>
_______________________________________________<br>
Gluster-devel mailing list<br>
<a href="mailto:Gluster-devel@nongnu.org" target="_blank">Gluster-devel@nongnu.org</a><br>
<a href="http://lists.nongnu.org/mailman/listinfo/gluster-devel" target="_blank">http://lists.nongnu.org/mailman/listinfo/gluster-devel</a><br>
</div></div></blockquote></div><br><br clear="all"><br>-- <br>Raghavendra G<br><br>