<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=ISO-8859-1">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<blockquote type="cite"><b>Tom Hall</b> <a
href="mailto:gluster-users%40gluster.org?Subject=Re:%20Re%3A%20%5BGluster-users%5D%20Gluster%20machines%20slowing%20down%20over%20time&In-Reply-To=%3CCAMvQ07jForynyik2NYmYnKq-kAa5wHoZLwOU8HDtYXKChM2ZXQ%40mail.gmail.com%3E"
title="[Gluster-users] Gluster machines slowing down over time">thattommyhall
at gmail.com </a><br>
<i>Wed Dec 12 12:47:09 UTC 2012</i>
<ul>
<li>Previous message: <a
href="http://gluster.org/pipermail/gluster-users/2012-December/034974.html">[Gluster-users]
Gluster machines slowing down over time
</a></li>
<li>Next message: <a
href="http://gluster.org/pipermail/gluster-users/2012-December/034973.html">[Gluster-users]
glusterfs-3.4.0qa5 released
</a></li>
<li> <b>Messages sorted by:</b> <a
href="http://gluster.org/pipermail/gluster-users/2012-December/date.html#34976">[
date ]</a> <a
href="http://gluster.org/pipermail/gluster-users/2012-December/thread.html#34976">[
thread ]</a> <a
href="http://gluster.org/pipermail/gluster-users/2012-December/subject.html#34976">[
subject ]</a> <a
href="http://gluster.org/pipermail/gluster-users/2012-December/author.html#34976">[
author ]</a> </li>
</ul>
<hr>
<pre>On Wed, Dec 12, 2012 at 10:41 AM, Brian Candler <<a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users">B.Candler at pobox.com</a>> wrote:
><i> On Tue, Dec 11, 2012 at 04:13:15PM +0000, Tom Hall wrote:
</i>><i> > I have 2 gluster servers in replicated mode on EC2 with ~4G RAM
</i>><i> >
</i>><i> > CPU and RAM look fine but over time the system becomes sluggish,
</i>><i> > particularly networking.
</i>><i> >
</i>><i> > I notice when sshing into the machine takes ages and running remote
</i>><i> > commands with capistrano takes longer and longer.
</i>><i>
</i>><i> Do you have swap configured? Is the system going into swap?
</i>
No swap (default on that size EC2 instance)
total used free shared buffers cached
Mem: 3750 3731 19 0 274 2929
-/+ buffers/cache: 526 3224
Swap: 0 0 0
Caching heavily but not using so much RAM
><i>
</i>
><i>
</i>Have a look at output of "free", "vmstat 5" (for a minute or two),
><i> "iostat 5 -x" (ditto)
</i>><i>
</i>><i>
</i>I'll get it next time we experience the issue.
Seems to slow down over a week or so so it might be a while...
Thanks guys!
Tom</pre>
</blockquote>
Dear All-<br>
I too have been experiencing a gradual slowing of GlusterFS over
time, accompanied by increases in CPU load and memory usage.
However, the storage server CPU load does not seem to be related to
the I/O load imposed by users. The following three Ganglia plots
for the past month illustrate the point.<br>
<br>
<a class="moz-txt-link-freetext" href="http://www.nerc-essc.ac.uk/~dab/GlusterFS_Ganglia_CPU_Mem_MonthTo8thMarch2013.jpeg">http://www.nerc-essc.ac.uk/~dab/GlusterFS_Ganglia_CPU_Mem_MonthTo8thMarch2013.jpeg</a><br>
<a class="moz-txt-link-freetext" href="http://www.nerc-essc.ac.uk/~dab/Rocks_Ganglia_CPU_Mem_MonthTo8thMarch2013.jpeg">http://www.nerc-essc.ac.uk/~dab/Rocks_Ganglia_CPU_Mem_MonthTo8thMarch2013.jpeg</a><br>
<a class="moz-txt-link-freetext" href="http://www.nerc-essc.ac.uk/~dab/GlusterFS-remus_Ganglia_CPU_Mem_MonthTo8thMarch2013.jpeg">http://www.nerc-essc.ac.uk/~dab/GlusterFS-remus_Ganglia_CPU_Mem_MonthTo8thMarch2013.jpeg</a><br>
<br>
The first plot shows the average storage server CPU load and memory
usage rising during the course of the past month, which became
almost unmanageable and prompted me to restart all the servers a few
days ago.<br>
<br>
The second plot shows the CPU load and memory usage on my small
Rocks compute cluster, which is responsible for most of the I/O load
on the GlusterFS cluster.<br>
<br>
The third chart shows CPU and memory usage on a typical storage
server, which has been restarted a couple of times in the past
month. The first restart was at the beginning of week 7 on the
plot, and the second was a couple days ago at the same time as the
other servers.<br>
<br>
There is no evidence that the I/O load on the storage cluster has
been increasing over the past month, so why does the CPU load on the
storage servers gradually increase (along with memory usage) until
they have to be restarted? Restarting the all the servers can be
very disruptive because it seems to take a long time for them all to
start talking to each other properly again. Once everything has
settled down everyone breathes a sigh of relief and enjoys
relatively fast and responsive I/O for a few days, before things
start to slow down again. It looks as if I'm going to have to start
doing this routinely every weekend, but this isn't how I was
planning to spend my weekends for the foreseeable future. Are there
any patches in the pipeline that are likely to address this issue?<br>
<br>
-Dan<br>
<pre class="moz-signature" cols="72">--
Dan Bretherton
ESSC Computer System Manager
Department of Meteorology
Harry Pitt Building, 3 Earley Gate
University of Reading
Reading, RG6 7BE (or RG6 6AL for postal service deliveries)
UK
Tel. +44 118 378 5205, Fax: +44 118 378 6413
--
## Please sponsor me to run in VSO's 30km Race to the Eye ##
## <a class="moz-txt-link-freetext" href="http://www.justgiving.com/DanBretherton">http://www.justgiving.com/DanBretherton</a> ##
</pre>
</body>
</html>