I will do this today. I noticed that I already have vm.drop_caches set to 3 via sysctl.conf, based on a suggestion from you from long ago. Should I delete this under normal usage? Is it possible that this setting, enabled by default, is causing my problems?<br clear="all">
<br>Dan<br>
<br><br><div class="gmail_quote">On Mon, Feb 23, 2009 at 3:23 PM, Amar Tumballi (bulde) <span dir="ltr"><<a href="mailto:amar@gluster.com">amar@gluster.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
Hi Gordan and Dan,<br><br>It would help me a lot if its possible for you to get the info as described below, <br><br>compile glusterfs like<br><br>bash# make clean > /dev/null<br>bash# make CFLAGS="-g -O0 -DDEBUG" > /dev/null<br>
bash# make install<br><br>run the process which consumes memory (mostly client process) like below:<br><br>bash# glusterfs <any argument you give generally> -N<br><this process will run in foreground now><br>
<br>
Open another terminal<br><br>bash# ps aux | grep glusterfs<br>bash# kill -s SIGUSR1 <pid of glusterfs -N process><br><Check in other terminal for memory usage stats><br><br>bash# <run your application over glusterfs as you do till you get high memory usage of glusterfs.. ><br>
bash# kill -s SIGUSR1 <pid of glusterfs -N process><br>
<Check the stat in another terminal><br><br>bash# echo 3 > /proc/sys/vm/drop_caches<br>bash# kill -s SIGUSR1 <pid of glusterfs -N process><br>
<Check the stat in another terminal><br>
<br>Even after dropping caches, if 'in use bytes =' in malloc stats shows a high value, then it is a leak. If its showing less, but just 'system bytes = ' is a high value, this means glusterfs is not really consuming high memory, but the problem is really in the memory allocation segments.<br>
<br>Regards,<br>Amar<br><br>NOTE: 'malloc_stats' will be printed to 'stdout' if we enable -DDEBUG while compiling glusterfs, as it hits performance badly otherwise.<br><br><br><div class="gmail_quote">2009/2/23 Gordan Bobic <span dir="ltr"><<a href="mailto:gordan@bobich.net" target="_blank">gordan@bobich.net</a>></span><div>
<div></div><div class="Wj3C7c"><br>
<blockquote class="gmail_quote" style="border-left:1px solid rgb(204, 204, 204);margin:0pt 0pt 0pt 0.8ex;padding-left:1ex"><div>Dan Parsons wrote:<br>
<blockquote class="gmail_quote" style="border-left:1px solid rgb(204, 204, 204);margin:0pt 0pt 0pt 0.8ex;padding-left:1ex">
I'm having an issue with glusterfs exceeding its cache-size value. Right now I have it set to 4000MB and I've seen it climb as high as 4800MB. If I set it to 5000, I've seen it go as high as 6000MB. This is a problem because it causes me to set the value very low so that my apps don't get pushed into swap. Is there any way to fix this? To get it to stick to the limit I set and not exceed?<br>
</blockquote>
<br></div>
It's possible you are running into the same memory leak that I'm seeing, and I'm not using io-cache or any other performance translators at all. With rootfs on Gluster, doing a kernel compile (kernel source tree being on NFS, so this won't be contributing to the bloat, hopefully) makes the glusterfsd bloat by about 80MB per pass, and never frees it.<br>
<br>
Gordan<br>
<br>
<br>
_______________________________________________<br>
Gluster-devel mailing list<br>
<a href="mailto:Gluster-devel@nongnu.org" target="_blank">Gluster-devel@nongnu.org</a><br>
<a href="http://lists.nongnu.org/mailman/listinfo/gluster-devel" target="_blank">http://lists.nongnu.org/mailman/listinfo/gluster-devel</a><br>
</blockquote></div></div></div><font color="#888888"><br><br clear="all"><br>-- <br>Amar Tumballi<br>Gluster/GlusterFS Hacker<br>[bulde on #gluster/<a href="http://irc.gnu.org" target="_blank">irc.gnu.org</a>]<br><a href="http://www.zresearch.com" target="_blank">http://www.zresearch.com</a> - Commoditizing Super Storage!<br>
</font><br>_______________________________________________<br>
Gluster-devel mailing list<br>
<a href="mailto:Gluster-devel@nongnu.org">Gluster-devel@nongnu.org</a><br>
<a href="http://lists.nongnu.org/mailman/listinfo/gluster-devel" target="_blank">http://lists.nongnu.org/mailman/listinfo/gluster-devel</a><br>
<br></blockquote></div><br>