<span class="Apple-style-span" style="border-collapse: collapse; "><div>Good afternoon gluster-users.</div><div><br></div><div>The glusterd memory footprint on one of the two (always the same one, the first in the cluster - server01) servers keeps growing. The performance of the cluster gets slower as the memory usage gets larger. The load average on the server also increases and by the time the server process is using 100meg of resident memory (showed by top) its almost using 100% of the cpu. Both server processes start out using about 9megs of memory.</div>
<div><br></div><div>Both backend server are identical with identical configuration files (with descriptor changes). Both are an identical build of Centos 5.3 with a kernel version of 2.6.18-128.1.6.el5. We have 3 XFS partitions of 30TB in size on each server (90TB total on each server). Simple raid1 setup between the two servers. Same hardware, same software, everything is identical.</div>
<div><br></div><div>Restarting the glusterd process every 12 hours seems silly, anybody have a better solution?</div><div><br></div><div>thanks,</div><div>liam</div><div><br></div><div>server1 glusterd.vol (this is the server with the memory leak):</div>
<div><br></div><div><div>volume intstore01a</div><div> type storage/posix</div><div> option directory /intstore/intstore01a/gdata</div><div>end-volume</div><div><br></div><div>volume intstore01b</div><div> type storage/posix</div>
<div> option directory /intstore/intstore01b/gdata</div><div>end-volume</div><div><br></div><div>volume intstore01c</div><div> type storage/posix</div><div> option directory /intstore/intstore01c/gdata</div><div>end-volume</div>
<div><br></div><div>volume locksa</div><div> type features/posix-locks</div><div> option mandatory-locks on</div><div> subvolumes intstore01a</div><div>end-volume</div><div><br></div><div>volume locksb</div><div> type features/posix-locks</div>
<div> option mandatory-locks on</div><div> subvolumes intstore01b</div><div>end-volume</div><div><br></div><div>volume locksc</div><div> type features/posix-locks</div><div> option mandatory-locks on</div><div> subvolumes intstore01c</div>
<div>end-volume</div><div><br></div><div>volume brick1a</div><div> type performance/io-threads</div><div> option thread-count 8</div><div> subvolumes locksa</div><div>end-volume</div><div><br></div><div>volume brick1b</div>
<div> type performance/io-threads</div><div> option thread-count 8</div><div> subvolumes locksb</div><div>end-volume</div><div><br></div><div>volume brick1c</div><div> type performance/io-threads</div><div> option thread-count 8</div>
<div> subvolumes locksc</div><div>end-volume</div><div><br></div><div>volume server</div><div> type protocol/server</div><div> option transport-type tcp</div><div> option auth.addr.brick1a.allow x.x.x.*</div><div> option auth.addr.brick1b.allow x.x.x.*</div>
<div> option auth.addr.brick1c.allow x.x.x.*</div><div> subvolumes brick1a brick1b brick1c</div><div>end-volume</div><div><br></div><div>server2 glusterd.vol (no memory leak):</div><div><br></div><div><div>volume intstore02a</div>
<div> type storage/posix</div><div> option directory /intstore/intstore02a/gdata</div><div>end-volume</div><div><br></div><div>volume intstore02b</div><div> type storage/posix</div><div> option directory /intstore/intstore02b/gdata</div>
<div>end-volume</div><div><br></div><div>volume intstore02c</div><div> type storage/posix</div><div> option directory /intstore/intstore02c/gdata</div><div>end-volume</div><div><br></div><div>volume locksa</div><div> type features/posix-locks</div>
<div> option mandatory-locks on</div><div> subvolumes intstore02a</div><div>end-volume</div><div><br></div><div>volume locksb</div><div> type features/posix-locks</div><div> option mandatory-locks on</div><div> subvolumes intstore02b</div>
<div>end-volume</div><div><br></div><div>volume locksc</div><div> type features/posix-locks</div><div> option mandatory-locks on</div><div> subvolumes intstore02c</div><div>end-volume</div><div><br></div><div>volume brick2a</div>
<div> type performance/io-threads</div><div> option thread-count 8</div><div> subvolumes locksa</div><div>end-volume</div><div><br></div><div>volume brick2b</div><div> type performance/io-threads</div><div> option thread-count 8</div>
<div> subvolumes locksb</div><div>end-volume</div><div><br></div><div>volume brick2c</div><div> type performance/io-threads</div><div> option thread-count 8</div><div> subvolumes locksc</div><div>end-volume</div><div>
<br></div><div>volume server</div><div> type protocol/server</div><div> option transport-type tcp</div><div> option auth.addr.brick2a.allow x.x.x.*</div><div> option auth.addr.brick2b.allow x.x.x.*</div><div> option auth.addr.brick2c.allow x.x.x.*</div>
<div> subvolumes brick2a brick2b brick2c</div><div>end-volume</div><div><br></div><div>We have only two clients, both are Centos 5.3 clients and both have the exact gluster.vol configuration file.</div><div><br></div><div>
client volume file:</div><div><br></div><div><div>volume brick1a</div><div> type protocol/client</div><div> option transport-type tcp</div><div> option remote-host server01</div><div> option remote-subvolume brick1a</div>
<div>end-volume</div><div><br></div><div>volume brick1b</div><div> type protocol/client</div><div> option transport-type tcp</div><div> option remote-host server01</div><div> option remote-subvolume brick1b</div><div>
end-volume</div><div><br></div><div>volume brick1c</div><div> type protocol/client</div><div> option transport-type tcp</div><div> option remote-host server01</div><div> option remote-subvolume brick1c</div><div>end-volume</div>
<div><br></div><div>volume brick2a</div><div> type protocol/client</div><div> option transport-type tcp</div><div> option remote-host server02</div><div> option remote-subvolume brick2a</div><div>end-volume</div><div>
<br></div><div>volume brick2b</div><div> type protocol/client</div><div> option transport-type tcp</div><div> option remote-host server02</div><div> option remote-subvolume brick2b</div><div>end-volume</div><div><br></div>
<div>volume brick2c</div><div> type protocol/client</div><div> option transport-type tcp</div><div> option remote-host server02</div><div> option remote-subvolume brick2c</div><div>end-volume</div><div><br></div><div>
volume bricks1</div><div> type cluster/replicate</div><div> subvolumes brick1a brick2a</div><div>end-volume</div><div><br></div><div>volume bricks2</div><div> type cluster/replicate</div><div> subvolumes brick1b brick2b</div>
<div>end-volume</div><div><br></div><div>volume bricks3</div><div> type cluster/replicate</div><div> subvolumes brick1c brick2c</div><div>end-volume</div><div><br></div><div>volume distribute</div><div> type cluster/distribute</div>
<div> subvolumes bricks1 bricks2 bricks3</div><div>end-volume</div><div><br></div><div>volume writebehind</div><div> type performance/write-behind</div><div> option block-size 1MB</div><div> option cache-size 64MB</div>
<div> option flush-behind on</div><div> subvolumes distribute</div><div>end-volume</div><div><br></div><div>volume cache</div><div> type performance/io-cache</div><div> option cache-size 2048MB</div><div> subvolumes writebehind</div>
<div>end-volume</div></div></div></div></span>