Thomas,<br> thanks for reporting the observation with fcntl locks. There was a memory leak in posix-locks which was fixed in 1.4 tree, but had somehow missed the backport to 1.3 tree. Please find the memory leak fix in the latest revision of glusterfs--mainline--2.5.<br>
<br>thanks,<br>avati<br><br><div class="gmail_quote">2008/11/4 Thomas Conway-Poulsen <span dir="ltr"><<a href="mailto:tecp@conwayit.dk">tecp@conwayit.dk</a>></span><br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
Hi Krishna<br>
<br>
Running with fuse-2.7.3glfs10.<br>
<br>
The client setup are as follows:<br>
<br>
------------------------------------------------------------------------------------<br>
mounts to /mnt/gluster/index:<br>
------------------------------------------------------------------------------------<br>
volume gluster01<br>
type protocol/client<br>
option transport-type tcp/client<br>
option remote-host <a href="http://10.11.253.1" target="_blank">10.11.253.1</a><br>
option remote-subvolume gluster-storage-index<br>
end-volume<br>
<br>
volume gluster02<br>
type protocol/client<br>
option transport-type tcp/client<br>
option remote-host <a href="http://10.11.253.2" target="_blank">10.11.253.2</a><br>
option remote-subvolume gluster-storage-index<br>
end-volume<br>
<br>
volume gluster03<br>
type protocol/client<br>
option transport-type tcp/client<br>
option remote-host <a href="http://10.11.253.3" target="_blank">10.11.253.3</a><br>
option remote-subvolume gluster-storage-index<br>
end-volume<br>
<br>
volume gluster04<br>
type protocol/client<br>
option transport-type tcp/client<br>
option remote-host <a href="http://10.11.253.4" target="_blank">10.11.253.4</a><br>
option remote-subvolume gluster-storage-index<br>
end-volume<br>
<br>
volume gluster05<br>
type protocol/client<br>
option transport-type tcp/client<br>
option remote-host <a href="http://10.11.253.5" target="_blank">10.11.253.5</a><br>
option remote-subvolume gluster-storage-index<br>
end-volume<br>
<br>
volume gluster06<br>
type protocol/client<br>
option transport-type tcp/client<br>
option remote-host <a href="http://10.11.253.6" target="_blank">10.11.253.6</a><br>
option remote-subvolume gluster-storage-index<div class="Ih2E3d"><br>
end-volume<br>
<br>
volume gluster-storage-index-namespace<br></div>
type protocol/client<br>
option transport-type tcp/client<br>
option remote-host <a href="http://10.11.253.1" target="_blank">10.11.253.1</a><br>
option remote-subvolume gluster-storage-index-namespace<div class="Ih2E3d"><br>
end-volume<br>
<br>
volume gluster-storage-index<br></div>
type cluster/unify<br>
option scheduler rr<br>
option namespace gluster-storage-index-namespace<br>
subvolumes gluster01 gluster02 gluster03 gluster04 gluster05 gluster06<div class="Ih2E3d"><br>
end-volume<br>
<br>
volume gluster-storage-index-iothreads<br>
type performance/io-threads<br></div>
subvolumes gluster-storage-index<br>
option cache-size 64MB<br>
option thread-count 8<br>
end-volume<br>
<br>
volume gluster-gluster-index-readahead<br>
type performance/read-ahead<br>
subvolumes gluster-storage-index-iothreads<br>
end-volume<br><div class="Ih2E3d">
<br>
volume gluster-storage-index-writebehind<br>
type performance/write-behind<br>
subvolumes gluster-storage-index-readahead<br>
end-volume<br>
<br></div>
volume gluster-storage-index-iocache<br>
type performance/io-cache<br>
option cache-size 256MB<div class="Ih2E3d"><br>
subvolumes gluster-storage-index-writebehind<br>
end-volume<br>
<br></div>
------------------------------------------------------------------------------------<br>
mounts to /mnt/gluster/data:<br>
------------------------------------------------------------------------------------<br>
volume cindex01<br>
type protocol/client<br>
option transport-type tcp/client<br>
option remote-host <a href="http://10.11.253.1" target="_blank">10.11.253.1</a><br>
option remote-subvolume gluster-storage-data<br>
end-volume<br>
<br>
volume cindex02<br>
type protocol/client<br>
option transport-type tcp/client<br>
option remote-host <a href="http://10.11.253.2" target="_blank">10.11.253.2</a><br>
option remote-subvolume gluster-storage-data<br>
end-volume<br>
<br>
volume cindex03<br>
type protocol/client<br>
option transport-type tcp/client<br>
option remote-host <a href="http://10.11.253.3" target="_blank">10.11.253.3</a><br>
option remote-subvolume gluster-storage-data<br>
end-volume<br>
<br>
volume cindex04<br>
type protocol/client<br>
option transport-type tcp/client<br>
option remote-host <a href="http://10.11.253.4" target="_blank">10.11.253.4</a><br>
option remote-subvolume gluster-storage-data<br>
end-volume<br>
<br>
volume cindex05<br>
type protocol/client<br>
option transport-type tcp/client<br>
option remote-host <a href="http://10.11.253.5" target="_blank">10.11.253.5</a><br>
option remote-subvolume gluster-storage-data<br>
end-volume<br>
<br>
volume cindex06<br>
type protocol/client<br>
option transport-type tcp/client<br>
option remote-host <a href="http://10.11.253.6" target="_blank">10.11.253.6</a><br>
option remote-subvolume gluster-storage-data<div class="Ih2E3d"><br>
end-volume<br>
<br>
volume gluster-storage-data-namespace<br></div>
type protocol/client<br>
option transport-type tcp/client<br>
option remote-host <a href="http://10.11.253.1" target="_blank">10.11.253.1</a><br>
option remote-subvolume gluster-storage-data-namespace<div class="Ih2E3d"><br>
end-volume<br>
<br>
volume gluster-storage-data<br></div>
type cluster/unify<br>
option scheduler rr<br>
option namespace gluster-storage-data-namespace<br>
subvolumes cindex01 cindex02 cindex03 cindex04 cindex05 cindex06<div class="Ih2E3d"><br>
end-volume<br>
<br>
volume gluster-storage-data-iothreads<br>
type performance/io-threads<br></div>
subvolumes gluster-storage-data<br>
option cache-size 64MB<br>
option thread-count 8<div class="Ih2E3d"><br>
end-volume<br>
<br>
volume gluster-storage-data-writebehind<br>
type performance/write-behind<br></div><div class="Ih2E3d">
subvolumes gluster-storage-data-iothreads<br>
end-volume<br>
<br></div>
volume gluster-storage-data-iocache<div class="Ih2E3d"><br>
type performance/io-cache<br>
option cache-size 128MB<br>
subvolumes gluster-storage-data-writebehind<br>
end-volume<br>
<br></div>
We can only see the memleak, when running multiple clients under heavy file-load over many files.<br>
<br>
The memory seems stable when we dont use fcnt locks-<br>
<br>
bool lock(int file_fd) {<br>
struct flock lock_info;<br>
lock_info.l_type = F_RDLCK;<br>
lock_info.l_whence = SEEK_SET;<br>
lock_info.l_start = 0;<br>
lock_info.l_len = 0;<br>
<br>
if(fcntl(file_fd, F_SETLK, &lock_info) == 0) return true;<br>
else return false;<br>
}<br>
<br>
bool unlock(int file_fd) {<br>
struct flock lock_info;<br>
lock_info.l_type = F_UNLCK;<br>
lock_info.l_whence = SEEK_SET;<br>
lock_info.l_start = 0;<br>
lock_info.l_len = 0;<br>
<br>
if(fcntl(file_fd, F_SETLKW, &lock_info) == 0) return true;<br>
else return false;<br>
}<br>
<br>
bool lock(int file_fd, int start, int length) {<br>
struct flock lock_info;<br>
lock_info.l_type = F_RDLCK;<br>
lock_info.l_whence = SEEK_SET;<br>
lock_info.l_start = start;<br>
lock_info.l_len = length;<br>
<br>
if(fcntl(file_fd, F_SETLK, &lock_info) == 0) return true;<br>
else return false;<br>
}<br>
<br>
bool unlock(int file_fd, int start, int length) {<br>
struct flock lock_info;<br>
lock_info.l_type = F_UNLCK;<br>
lock_info.l_whence = SEEK_SET;<br>
lock_info.l_start = start;<br>
lock_info.l_len = length;<br>
<br>
if(fcntl(file_fd, F_SETLKW, &lock_info) == 0) return true;<br>
else return false;<br>
}<br>
<br>
<br>
Hope this helps,<br><font color="#888888">
<br>
Thomas Conway.</font><div><div></div><div class="Wj3C7c"><br>
<br>
On Nov 4, 2008, at 8:07 AM, Krishna Srinivas wrote:<br>
<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
Thomas,<br>
We want to reproduce the leak in our setup to fix it. What is your<br>
setup on the client side? How many servers do you have? What are the<br>
applications you run on the mount point? Do you observe leak only when<br>
"certain" operations are done? (I am just looking for more clues)<br>
<br>
Thanks<br>
Krishna<br>
<br>
On Sun, Nov 2, 2008 at 5:08 PM, Thomas Conway-Poulsen <<a href="mailto:tecp@conwayit.dk" target="_blank">tecp@conwayit.dk</a>> wrote:<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
Hi devel,<br>
<br>
The Glusterfsd process is using much more memory than we expected it to use,<br>
maybe there is some memory leak?<br>
<br>
It keeps eating until the process dies.<br>
<br>
Is there any way to set the maximum memory usage ?<br>
<br>
root 15436 1.5 85.5 2180580 1764980 ? Ssl Oct28 105:20 [glusterfs]<br>
<br>
Our server configuration:<br>
------------------------------------------------<br>
volume gluster-storage-data-export<br>
type storage/posix<br>
option directory /mnt/gluster-storage-server/data/export<br>
end-volume<br>
<br>
volume gluster-storage-data-namespace<br>
type storage/posix<br>
option directory /mnt/gluster-storage-server/data/namespace<br>
end-volume<br>
<br>
volume gluster-storage-data-iothreads<br>
type performance/io-threads<br>
option thread-count 2<br>
option cache-size 32MB<br>
subvolumes gluster-storage-data-export<br>
end-volume<br>
<br>
volume gluster-storage-data-locks<br>
type features/posix-locks<br>
subvolumes gluster-storage-data-iothreads<br>
end-volume<br>
<br>
volume gluster-storage-data-readahead<br>
type performance/read-ahead<br>
subvolumes gluster-storage-data-locks<br>
end-volume<br>
<br>
volume gluster-storage-data-writebehind<br>
type performance/write-behind<br>
subvolumes gluster-storage-data-readahead<br>
end-volume<br>
<br>
volume gluster-storage-data<br>
type performance/io-cache<br>
option cache-size 128MB<br>
subvolumes gluster-storage-data-writebehind<br>
end-volume<br>
<br>
volume gluster-storage-index-export<br>
type storage/posix<br>
option directory /mnt/gluster-storage-server/index/export<br>
end-volume<br>
<br>
volume gluster-storage-index-namespace<br>
type storage/posix<br>
option directory /mnt/gluster-storage-server/index/namespace<br>
end-volume<br>
<br>
volume gluster-storage-index-iothreads<br>
type performance/io-threads<br>
option thread-count 2<br>
option cache-size 32MB<br>
subvolumes gluster-storage-index-export<br>
end-volume<br>
<br>
volume gluster-storage-index-locks<br>
type features/posix-locks<br>
subvolumes gluster-storage-index-export<br>
end-volume<br>
<br>
volume gluster-storage-index-readahead<br>
type performance/read-ahead<br>
subvolumes gluster-storage-index-locks<br>
end-volume<br>
<br>
volume gluster-storage-index-writebehind<br>
type performance/write-behind<br>
subvolumes gluster-storage-index-readahead<br>
end-volume<br>
<br>
volume gluster-storage-index<br>
type performance/io-cache<br>
option cache-size 128MB<br>
subvolumes gluster-storage-index-writebehind<br>
end-volume<br>
<br>
volume gluster-server<br>
type protocol/server<br>
subvolumes gluster-storage-index gluster-storage-index-namespace<br>
gluster-storage-data gluster-storage-data-namespace<br>
option transport-type tcp/server<br>
option auth.ip.gluster-storage-index.allow *<br>
option auth.ip.gluster-storage-index-namespace.allow *<br>
option auth.ip.gluster-storage-data.allow *<br>
option auth.ip.gluster-storage-data-namespace.allow *<br>
end-volume<br>
<br>
<br>
Here is the pmap dump:<br>
-----------------------------------------------------<br>
15436: [glusterfs]<br>
Address Kbytes Mode Offset Device Mapping<br>
0000000000400000 16 r-x-- 0000000000000000 008:00001 glusterfs<br>
0000000000504000 4 rw--- 0000000000004000 008:00001 glusterfs<br>
0000000000505000 497276 rw--- 0000000000505000 000:00000 [ anon ]<br>
0000000040000000 4 ----- 0000000040000000 000:00000 [ anon ]<br>
0000000040001000 8192 rw--- 0000000040001000 000:00000 [ anon ]<br>
0000000040801000 4 ----- 0000000040801000 000:00000 [ anon ]<br>
0000000040802000 8192 rw--- 0000000040802000 000:00000 [ anon ]<br>
0000000041002000 4 ----- 0000000041002000 000:00000 [ anon ]<br>
0000000041003000 8192 rw--- 0000000041003000 000:00000 [ anon ]<br>
</blockquote>
<br>
</blockquote>
<br>
</div></div><br>_______________________________________________<br>
Gluster-devel mailing list<br>
<a href="mailto:Gluster-devel@nongnu.org">Gluster-devel@nongnu.org</a><br>
<a href="http://lists.nongnu.org/mailman/listinfo/gluster-devel" target="_blank">http://lists.nongnu.org/mailman/listinfo/gluster-devel</a><br>
<br></blockquote></div><br><br clear="all"><br>-- <br>If I traveled to the end of the rainbow<br>As Dame Fortune did intend,<br>Murphy would be there to tell me<br>The pot's at the other end.<br>