<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"MS Mincho";
panose-1:2 2 6 9 4 2 5 8 3 4;}
@font-face
{font-family:"MS Mincho";
panose-1:2 2 6 9 4 2 5 8 3 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:"\@MS Mincho";
panose-1:2 2 6 9 4 2 5 8 3 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoAcetate, li.MsoAcetate, div.MsoAcetate
{mso-style-priority:99;
mso-style-link:"Balloon Text Char";
margin:0cm;
margin-bottom:.0001pt;
font-size:8.0pt;
font-family:"Tahoma","sans-serif";}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri","sans-serif";
color:windowtext;}
span.BalloonTextChar
{mso-style-name:"Balloon Text Char";
mso-style-priority:99;
mso-style-link:"Balloon Text";
font-family:"Tahoma","sans-serif";}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal">Hello All,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I’m having some trouble with the memory allocation of my glusterfsd processes. The server keeps running out of memory, no matter how much I add (it’s a vm) and I just don’t understand how gluster allocates memory to its processes.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">The situation is as follows: rhel 6.4, 16 GB ram, gluster 3.4.2-1<o:p></o:p></p>
<p class="MsoNormal">I have one big lun of 20 TB that contains all my bricks. I started with 4 GB memory, but saw it was getting full, so I stepped it up to 8GB and later 16. The weird thing is VMware claims only 50% of the memory is actively used so I guess
this is all due to caching?<o:p></o:p></p>
<p class="MsoNormal">I tried the flushcaches thing, but I don’t see to free up much.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">If I check my memory consumption I see this:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND<o:p></o:p></p>
<p class="MsoNormal">26800 root 20 0 2593m 1.9g 1088 S 17.3 12.5 5261:43 glusterfsd<o:p></o:p></p>
<p class="MsoNormal"><span lang="NL-BE">26914 root 20 0 2541m 1.9g 1072 S 15.9 12.2 1081:04 glusterfsd<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="NL-BE">33299 root 20 0 2938m 1.9g 1056 S 15.9 12.1 1787:40 glusterfsd<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="NL-BE">26872 root 20 0 2093m 1.3g 1080 S 15.6 8.6 3391:02 glusterfsd<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="NL-BE">26995 root 20 0 1828m 1.3g 1072 S 8.0 8.1 782:29.85 glusterfsd<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="NL-BE">26934 root 20 0 1891m 1.0g 376 S 0.0 6.6 795:44.19 glusterfsd<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="NL-BE">37651 root 20 0 1309m 1.0g 332 S 0.0 6.6 0:03.52 glusterfs<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="NL-BE">27015 root 20 0 1374m 780m 388 S 0.0 4.9 580:29.19 glusterfsd<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="NL-BE">29712 root 20 0 989m 504m 1076 S 20.2 3.2 227:47.87 glusterfsd<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="NL-BE">8635 root 20 0 2415m 186m 392 S 0.0 1.2 594:02.47 glusterfsd<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="NL-BE">31859 root 20 0 643m 174m 392 S 0.0 1.1 99:06.60 glusterfsd<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="NL-BE">27308 root 20 0 440m 169m 1296 S 10.0 1.1 646:30.02 glusterfs<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="NL-BE">27458 root 20 0 415m 149m 1280 S 12.3 0.9 781:36.04 glusterfs<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="NL-BE">27358 root 20 0 420m 143m 1272 S 10.3 0.9 553:04.97 glusterfs<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="NL-BE">7453 root 20 0 609m 133m 872 S 0.0 0.8 27:49.83 glusterfsd<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="NL-BE">27508 root 20 0 410m 117m 1272 S 9.6 0.7 560:42.51 glusterfs<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="NL-BE">8608 root 20 0 2418m 111m 372 S 0.0 0.7 785:53.31 glusterfsd<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="NL-BE">27408 root 20 0 373m 96m 1272 S 9.6 0.6 431:32.28 glusterfs<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="NL-BE">27558 root 20 0 368m 87m 344 S 0.0 0.6 318:37.92 glusterfs<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="NL-BE">40715 root 20 0 203m 86m 1036 S 0.3 0.5 3:41.78 puppetd<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="NL-BE">30940 root 20 0 674m 78m 960 S 0.0 0.5 40:18.17 glusterfsd<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="NL-BE">19363 root 20 0 1437m 74m 728 S 0.0 0.5 60:12.02 glusterfsd<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="NL-BE">8773 root 20 0 685m 71m 956 S 0.0 0.4 19:40.05 glusterfsd<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="NL-BE"> 789 root 20 0 625m 67m 900 S 0.0 0.4 22:45.64 glusterfsd<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="NL-BE">1599 root 20 0 1291m 41m 956 S 0.0 0.3 13:03.23 glusterfsd<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="NL-BE">27808 root 20 0 482m 36m 344 S 0.0 0.2 42:40.67 glusterfs<o:p></o:p></span></p>
<p class="MsoNormal">47625 root 20 0 677m 31m 900 S 0.0 0.2 4:09.29 glusterfsd<o:p></o:p></p>
<p class="MsoNormal">30539 root 20 0 1381m 27m 852 S 0.0 0.2 20:31.90 glusterfsd<o:p></o:p></p>
<p class="MsoNormal">Etc…<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Right now I’m at:<o:p></o:p></p>
<p class="MsoNormal"> total used free shared buffers cached<o:p></o:p></p>
<p class="MsoNormal">Mem: 16326684 16176692 149992 0 14284 59520<o:p></o:p></p>
<p class="MsoNormal">-/+ buffers/cache: 16102888 223796<o:p></o:p></p>
<p class="MsoNormal">Swap: 4194296 4194252 44<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">… the lack of free swap worries me a lot<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Is it normal for some volumes to consume that much memory? How does gluster decide how much memory it should allocate? It there any way to influence this?<o:p></o:p></p>
<p class="MsoNormal">From time to time bricks switch themselves offline. Is this due to the lack of free memory?<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Any help to shed some light on this situation would be most welcome as I have no idea where to start…..<o:p></o:p></p>
</div>
</body>
</html>