<html>
<head>
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<br>
<div class="moz-cite-prefix">On 07/07/2014 07:22 PM, Gurdeep Singh
(Guru) wrote:<br>
</div>
<blockquote
cite="mid:32AD0B84-7C38-4F5F-BEFE-76F72F7F2A43@bazaari.com.au"
type="cite">
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252">
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252">
<div style="margin: 0px; font-size: 10px; font-family: Monaco;
color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">[guru@srv2
~]$ ps -v 1055</div>
<div style="margin: 0px; font-size: 10px; font-family: Monaco;
color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">Warning:
bad syntax, perhaps a bogus '-'? See
/usr/share/doc/procps-3.2.8/FAQ</div>
<div style="margin: 0px; font-size: 10px; font-family: Monaco;
color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">
PID TTY STAT TIME MAJFL TRS DRS RSS %MEM COMMAND</div>
<div style="margin: 0px; font-size: 10px; font-family: Monaco;
color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);"> 1055
? Ssl 86:01 31 0 319148 33092 3.2
/usr/sbin/glusterfs --volfile-server=srv2 --volfile-id=/gv0
/var/www/html/image/</div>
<div style="margin: 0px; font-size: 10px; font-family: Monaco;
color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">[guru@srv2
~]$ </div>
<div><br>
</div>
</blockquote>
Gurdeep,<br>
Don't see anything odd here :-(. Mount is looking up files
and brick is serving it. Why don't you keep a watch on the process
'18629' and similar processes in the cluster. Do a ps aux | grep
glustershd to get the pids. There will be one such process per
machine in the cluster. Check how much it consumes. That is the only
process which does operations without any operations on mounts.<br>
<br>
Pranith<br>
<br>
<blockquote
cite="mid:32AD0B84-7C38-4F5F-BEFE-76F72F7F2A43@bazaari.com.au"
type="cite">
<div><br>
</div>
<div><br>
</div>
<div>
<div>On 7 Jul 2014, at 11:49 pm, Pranith Kumar Karampuri <<a
moz-do-not-send="true" href="mailto:pkarampu@redhat.com">pkarampu@redhat.com</a>>
wrote:</div>
<br class="Apple-interchange-newline">
<blockquote type="cite">
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
<div text="#000000" bgcolor="#FFFFFF"> <br>
<div class="moz-cite-prefix">On 07/07/2014 07:03 PM, Gurdeep
Singh (Guru) wrote:<br>
</div>
<blockquote
cite="mid:19E47DE9-3A91-47EC-8083-3E8E2574A045@bazaari.com.au"
type="cite">
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252">
Hello Niels,
<div><br>
</div>
<div>I did a net hogs on the interface to see what process
might be using the bandwidth,</div>
<div><br>
</div>
<div>
<div style="margin: 0px; font-size: 10px; font-family:
Monaco; color: rgb(245, 245, 245); background-color:
rgb(0, 0, 0);">NetHogs version 0.8.0</div>
<div style="margin: 0px; font-size: 10px; font-family:
Monaco; color: rgb(245, 245, 245); background-color:
rgb(0, 0, 0); min-height: 14px;"><br>
</div>
<div style="margin: 0px; font-size: 10px; font-family:
Monaco; background-color: rgb(245, 245, 245);"> PID
USER PROGRAM
DEV SENT RECEIVED </div>
<div style="margin: 0px; font-size: 10px; font-family:
Monaco; color: rgb(245, 245, 245); background-color:
rgb(0, 0, 0);">18611 root /usr/sbin/glusterfsd
tun0<span
class="Apple-tab-span" style="white-space:pre"> </span>
16.307 17.547 KB/sec</div>
<div style="margin: 0px; font-size: 10px; font-family:
Monaco; color: rgb(245, 245, 245); background-color:
rgb(0, 0, 0);">1055 root /usr/sbin/glusterfs
tun0<span
class="Apple-tab-span" style="white-space:pre"> </span>
17.249 16.259 KB/sec</div>
<div style="margin: 0px; font-size: 10px; font-family:
Monaco; color: rgb(245, 245, 245); background-color:
rgb(0, 0, 0);">13439 guru sshd: guru@pts/0
tun0<span
class="Apple-tab-span" style="white-space:pre"> </span>
0.966 0.051 KB/sec</div>
<div style="margin: 0px; font-size: 10px; font-family:
Monaco; color: rgb(245, 245, 245); background-color:
rgb(0, 0, 0);">18625 root /usr/sbin/glusterfs
tun0<span
class="Apple-tab-span" style="white-space:pre"> </span>
0.000 0.000 KB/sec</div>
<div style="margin: 0px; font-size: 10px; font-family:
Monaco; color: rgb(245, 245, 245); background-color:
rgb(0, 0, 0);">18629 root /usr/sbin/glusterfs
tun0<span
class="Apple-tab-span" style="white-space:pre"> </span>
0.000 0.000 KB/sec</div>
<div style="margin: 0px; font-size: 10px; font-family:
Monaco; color: rgb(245, 245, 245); background-color:
rgb(0, 0, 0);">9636 root /usr/sbin/glusterd
tun0<span
class="Apple-tab-span" style="white-space:pre"> </span>
0.000 0.000 KB/sec</div>
<div style="margin: 0px; font-size: 10px; font-family:
Monaco; color: rgb(245, 245, 245); background-color:
rgb(0, 0, 0);">? root unknown TCP
0.000
0.000 KB/sec</div>
<div style="margin: 0px; font-size: 10px; font-family:
Monaco; color: rgb(245, 245, 245); background-color:
rgb(0, 0, 0); min-height: 14px;"><br>
</div>
<div style="margin: 0px; font-size: 10px; font-family:
Monaco; background-color: rgb(245, 245, 245);"> TOTAL
34.523 33.856 KB/sec </div>
<div style="margin: 0px; font-size: 10px; font-family:
Monaco; color: rgb(245, 245, 245); background-color:
rgb(0, 0, 0); min-height: 14px;"><br>
</div>
<div style="margin: 0px; font-size: 10px; font-family:
Monaco; color: rgb(245, 245, 245); background-color:
rgb(0, 0, 0); min-height: 14px;"><br>
</div>
<div>
<div><br>
</div>
<div><br>
</div>
</div>
</div>
</blockquote>
Which process corresponds to '1055'?<br>
<br>
Pranith<br>
<blockquote
cite="mid:19E47DE9-3A91-47EC-8083-3E8E2574A045@bazaari.com.au"
type="cite">
<div>
<div>
<div>Its glusterfs and glusterfsd process.</div>
<div><br>
</div>
<div>I looked at the capture file and see that the
lookup is being made on random files.</div>
<div><br>
</div>
<div>For PID information, please see this:</div>
<div><br>
</div>
<div>
<div style="margin: 0px; font-size: 10px;
font-family: Monaco; color: rgb(245, 245, 245);
background-color: rgb(0, 0, 0);">[guru@srv2 ~]$
sudo netstat -tpn | grep 49152</div>
<div style="margin: 0px; font-size: 10px;
font-family: Monaco; color: rgb(245, 245, 245);
background-color: rgb(0, 0, 0);">tcp 0
0 127.0.0.1:49152 127.0.0.1:1012
ESTABLISHED 18611/glusterfsd </div>
<div style="margin: 0px; font-size: 10px;
font-family: Monaco; color: rgb(245, 245, 245);
background-color: rgb(0, 0, 0);">tcp 0
0 127.0.0.1:49152 127.0.0.1:1016
ESTABLISHED 18611/glusterfsd </div>
<div style="margin: 0px; font-size: 10px;
font-family: Monaco; color: rgb(245, 245, 245);
background-color: rgb(0, 0, 0);">tcp 0
0 127.0.0.1:1016 127.0.0.1:49152
ESTABLISHED 18625/glusterfs </div>
<div style="margin: 0px; font-size: 10px;
font-family: Monaco; color: rgb(245, 245, 245);
background-color: rgb(0, 0, 0);">tcp 0
0 10.8.0.6:1021 10.8.0.1:49152
ESTABLISHED 1055/glusterfs </div>
<div style="margin: 0px; font-size: 10px;
font-family: Monaco; color: rgb(245, 245, 245);
background-color: rgb(0, 0, 0);">tcp 0
0 10.8.0.6:49152 10.8.0.1:1017
ESTABLISHED 18611/glusterfsd </div>
<div style="margin: 0px; font-size: 10px;
font-family: Monaco; color: rgb(245, 245, 245);
background-color: rgb(0, 0, 0);">tcp 0
0 10.8.0.6:1020 10.8.0.1:49152
ESTABLISHED 18629/glusterfs </div>
<div style="margin: 0px; font-size: 10px;
font-family: Monaco; color: rgb(245, 245, 245);
background-color: rgb(0, 0, 0);">tcp 0
0 127.0.0.1:1023 127.0.0.1:49152
ESTABLISHED 18629/glusterfs </div>
<div style="margin: 0px; font-size: 10px;
font-family: Monaco; color: rgb(245, 245, 245);
background-color: rgb(0, 0, 0);">tcp 0
0 10.8.0.6:49152 10.8.0.1:1022
ESTABLISHED 18611/glusterfsd </div>
<div style="margin: 0px; font-size: 10px;
font-family: Monaco; color: rgb(245, 245, 245);
background-color: rgb(0, 0, 0);">tcp 0
0 10.8.0.6:49152 10.8.0.1:1021
ESTABLISHED 18611/glusterfsd </div>
<div style="margin: 0px; font-size: 10px;
font-family: Monaco; color: rgb(245, 245, 245);
background-color: rgb(0, 0, 0);">tcp 0
0 127.0.0.1:49152 127.0.0.1:1023
ESTABLISHED 18611/glusterfsd </div>
<div style="margin: 0px; font-size: 10px;
font-family: Monaco; color: rgb(245, 245, 245);
background-color: rgb(0, 0, 0);">tcp 0
0 127.0.0.1:1012 127.0.0.1:49152
ESTABLISHED 1055/glusterfs </div>
<div style="margin: 0px; font-size: 10px;
font-family: Monaco; color: rgb(245, 245, 245);
background-color: rgb(0, 0, 0);">tcp 0
0 10.8.0.6:1019 10.8.0.1:49152
ESTABLISHED 18625/glusterfs </div>
<div style="margin: 0px; font-size: 10px;
font-family: Monaco; color: rgb(245, 245, 245);
background-color: rgb(0, 0, 0);">[guru@srv2 ~]$ ps
-v 18611</div>
<div style="margin: 0px; font-size: 10px;
font-family: Monaco; color: rgb(245, 245, 245);
background-color: rgb(0, 0, 0);">Warning: bad
syntax, perhaps a bogus '-'? See
/usr/share/doc/procps-3.2.8/FAQ</div>
<div style="margin: 0px; font-size: 10px;
font-family: Monaco; color: rgb(245, 245, 245);
background-color: rgb(0, 0, 0);"> PID TTY
STAT TIME MAJFL TRS DRS RSS %MEM COMMAND</div>
<div style="margin: 0px; font-size: 10px;
font-family: Monaco; color: rgb(245, 245, 245);
background-color: rgb(0, 0, 0);">18611 ?
Ssl 14:12 0 0 650068 20404 2.0
/usr/sbin/glusterfsd -s srv2 --volfile-id
gv0.srv2.root-gluster-vol0 -p
/var/lib/glusterd/vols/gv0</div>
<div style="margin: 0px; font-size: 10px;
font-family: Monaco; color: rgb(245, 245, 245);
background-color: rgb(0, 0, 0);">[guru@srv2 ~]$ ps
-v 18629</div>
<div style="margin: 0px; font-size: 10px;
font-family: Monaco; color: rgb(245, 245, 245);
background-color: rgb(0, 0, 0);">Warning: bad
syntax, perhaps a bogus '-'? See
/usr/share/doc/procps-3.2.8/FAQ</div>
<div style="margin: 0px; font-size: 10px;
font-family: Monaco; color: rgb(245, 245, 245);
background-color: rgb(0, 0, 0);"> PID TTY
STAT TIME MAJFL TRS DRS RSS %MEM COMMAND</div>
<div style="margin: 0px; font-size: 10px;
font-family: Monaco; color: rgb(245, 245, 245);
background-color: rgb(0, 0, 0);">18629 ?
Ssl 0:04 0 0 333296 17380 1.7
/usr/sbin/glusterfs -s localhost --volfile-id
gluster/glustershd -p
/var/lib/glusterd/glustershd/r</div>
<div style="margin: 0px; font-size: 10px;
font-family: Monaco; color: rgb(245, 245, 245);
background-color: rgb(0, 0, 0);">[guru@srv2 ~]$ </div>
<div style="margin: 0px; font-size: 10px;
font-family: Monaco; color: rgb(245, 245, 245);
background-color: rgb(0, 0, 0);">[guru@srv2 ~]$ </div>
<div style="margin: 0px; font-size: 10px;
font-family: Monaco; color: rgb(245, 245, 245);
background-color: rgb(0, 0, 0);">[guru@srv2 ~]$ ps
-v 18629</div>
<div style="margin: 0px; font-size: 10px;
font-family: Monaco; color: rgb(245, 245, 245);
background-color: rgb(0, 0, 0);">Warning: bad
syntax, perhaps a bogus '-'? See
/usr/share/doc/procps-3.2.8/FAQ</div>
<div style="margin: 0px; font-size: 10px;
font-family: Monaco; color: rgb(245, 245, 245);
background-color: rgb(0, 0, 0);"> PID TTY
STAT TIME MAJFL TRS DRS RSS %MEM COMMAND</div>
<div style="margin: 0px; font-size: 10px;
font-family: Monaco; color: rgb(245, 245, 245);
background-color: rgb(0, 0, 0);">18629 ?
Ssl 0:04 0 0 333296 17380 1.7
/usr/sbin/glusterfs -s localhost --volfile-id
gluster/glustershd -p
/var/lib/glusterd/glustershd/run/glustershd.pid -l
/var/log/glusterfs/glustershd.log -S
/var/run/823fa3197e2d1841be888</div>
<div style="margin: 0px; font-size: 10px;
font-family: Monaco; color: rgb(245, 245, 245);
background-color: rgb(0, 0, 0);">[guru@srv2 ~]$ </div>
<div style="margin: 0px; font-size: 10px;
font-family: Monaco; color: rgb(245, 245, 245);
background-color: rgb(0, 0, 0);">[guru@srv2 ~]$ </div>
<div style="margin: 0px; font-size: 10px;
font-family: Monaco; color: rgb(245, 245, 245);
background-color: rgb(0, 0, 0);">[guru@srv2 ~]$ </div>
<div style="margin: 0px; font-size: 10px;
font-family: Monaco; color: rgb(245, 245, 245);
background-color: rgb(0, 0, 0);">[guru@srv2 ~]$ </div>
<div style="margin: 0px; font-size: 10px;
font-family: Monaco; color: rgb(245, 245, 245);
background-color: rgb(0, 0, 0);">[guru@srv2 ~]$ ps
-v 18629</div>
<div style="margin: 0px; font-size: 10px;
font-family: Monaco; color: rgb(245, 245, 245);
background-color: rgb(0, 0, 0);">Warning: bad
syntax, perhaps a bogus '-'? See
/usr/share/doc/procps-3.2.8/FAQ</div>
<div style="margin: 0px; font-size: 10px;
font-family: Monaco; color: rgb(245, 245, 245);
background-color: rgb(0, 0, 0);"> PID TTY
STAT TIME MAJFL TRS DRS RSS %MEM COMMAND</div>
<div style="margin: 0px; font-size: 10px;
font-family: Monaco; color: rgb(245, 245, 245);
background-color: rgb(0, 0, 0);">18629 ?
Ssl 0:04 0 0 333296 17380 1.7
/usr/sbin/glusterfs -s localhost --volfile-id
gluster/glustershd -p
/var/lib/glusterd/glustershd/run/glustershd.pid -l
/var/log/glusterfs/glustershd.log -S
/var/run/823fa3197e2d1841be8881500723b063.socket
--xlator-option
*replicate*.node-uuid=84af83c9-0a29-</div>
<div style="margin: 0px; font-size: 10px;
font-family: Monaco; color: rgb(245, 245, 245);
background-color: rgb(0, 0, 0);">[guru@srv2 ~]$ </div>
<div style="margin: 0px; font-size: 10px;
font-family: Monaco; color: rgb(245, 245, 245);
background-color: rgb(0, 0, 0);">[guru@srv2 ~]$ </div>
<div style="margin: 0px; font-size: 10px;
font-family: Monaco; color: rgb(245, 245, 245);
background-color: rgb(0, 0, 0);">[guru@srv2 ~]$ ps
-v 18625</div>
<div style="margin: 0px; font-size: 10px;
font-family: Monaco; color: rgb(245, 245, 245);
background-color: rgb(0, 0, 0);">Warning: bad
syntax, perhaps a bogus '-'? See
/usr/share/doc/procps-3.2.8/FAQ</div>
<div style="margin: 0px; font-size: 10px;
font-family: Monaco; color: rgb(245, 245, 245);
background-color: rgb(0, 0, 0);"> PID TTY
STAT TIME MAJFL TRS DRS RSS %MEM COMMAND</div>
<div style="margin: 0px; font-size: 10px;
font-family: Monaco; color: rgb(245, 245, 245);
background-color: rgb(0, 0, 0);">18625 ?
Ssl 0:03 0 0 239528 41040 4.0
/usr/sbin/glusterfs -s localhost --volfile-id
gluster/nfs -p /var/lib/glusterd/nfs/run/nfs.pid
-l /var/log/glusterfs/nfs.log -S
/var/run/5ad5b036fd636cc5dddffa73593e4089.socket</div>
<div style="margin: 0px; font-size: 10px;
font-family: Monaco; color: rgb(245, 245, 245);
background-color: rgb(0, 0, 0);">[guru@srv2 ~]$
sudo nethogs tun0</div>
<div style="margin: 0px; font-size: 10px;
font-family: Monaco; color: rgb(245, 245, 245);
background-color: rgb(0, 0, 0);">Waiting for first
packet to arrive (see <a moz-do-not-send="true"
href="http://sourceforge.net/">sourceforge.net</a>
bug 1019381)</div>
<div style="margin: 0px; font-size: 10px;
font-family: Monaco; color: rgb(245, 245, 245);
background-color: rgb(0, 0, 0);">[guru@srv2 ~]$
rpm -qa | grep gluster</div>
<div style="margin: 0px; font-size: 10px;
font-family: Monaco; color: rgb(245, 245, 245);
background-color: rgb(0, 0, 0);">glusterfs-3.5.1-1.el6.x86_64</div>
<div style="margin: 0px; font-size: 10px;
font-family: Monaco; color: rgb(245, 245, 245);
background-color: rgb(0, 0, 0);">glusterfs-cli-3.5.1-1.el6.x86_64</div>
<div style="margin: 0px; font-size: 10px;
font-family: Monaco; color: rgb(245, 245, 245);
background-color: rgb(0, 0, 0);">glusterfs-libs-3.5.1-1.el6.x86_64</div>
<div style="margin: 0px; font-size: 10px;
font-family: Monaco; color: rgb(245, 245, 245);
background-color: rgb(0, 0, 0);">glusterfs-fuse-3.5.1-1.el6.x86_64</div>
<div style="margin: 0px; font-size: 10px;
font-family: Monaco; color: rgb(245, 245, 245);
background-color: rgb(0, 0, 0);">glusterfs-server-3.5.1-1.el6.x86_64</div>
<div style="margin: 0px; font-size: 10px;
font-family: Monaco; color: rgb(245, 245, 245);
background-color: rgb(0, 0, 0);">glusterfs-api-3.5.1-1.el6.x86_64</div>
<div style="margin: 0px; font-size: 10px;
font-family: Monaco; color: rgb(245, 245, 245);
background-color: rgb(0, 0, 0);">[guru@srv2 ~]$ </div>
</div>
<div><br>
</div>
<div><br>
</div>
<div>I don’t see anything odd here. Please suggest.</div>
<div><br>
</div>
<div>Thanks,</div>
<div>Gurdeep.</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div>On 7 Jul 2014, at 9:06 pm, Niels de Vos <<a
moz-do-not-send="true"
href="mailto:ndevos@redhat.com">ndevos@redhat.com</a>>
wrote:</div>
<br class="Apple-interchange-newline">
<blockquote type="cite">On Sun, Jul 06, 2014 at
11:28:51PM +1000, Gurdeep Singh (Guru) wrote:<br>
<blockquote type="cite">Hello,<br>
<br>
I have setup gluster in replicate type and its
working fine.<br>
<br>
I am seeing a constant chatting between the hosts
for lookup call and <br>
lookup reply. I am trying to understand as to why
this traffic is <br>
being initiated constantly. Please look at the
attached image. This <br>
traffic is using around 200KB/s of constant
bandwidth and is <br>
exhausting our allocated monthly bandwidth on our
2 VPS.<br>
</blockquote>
<br>
You can use Wireshark to identify which process does
the LOOKUP calls. <br>
For this, do the following:<br>
<br>
1. select a LOOKUP Call<br>
2. enable the 'packet details' pane (found in the
main menu, 'view')<br>
3. expand the 'Transmission Control Protocol' tree<br>
4. check the 'Source port' of the LOOKUP Call<br>
<br>
Together with the 'Source' and the 'Source port' you
can go to the <br>
server that matches the 'Source' address. A command
like this would give <br>
you the PID of the process in the right column:<br>
<br>
# netstat -tpn | grep $SOURCE_PORT<br>
<br>
And with 'ps -v $PID' you can check which process is
responsible for the <br>
LOOKUP. This process can be a fuse-mount,
self-heal-daemon or any other <br>
glusterfs-client. Depending on the type of client,
you maybe can tune <br>
the workload or other options a little.<br>
<br>
In Wireshark you can also check what filename is
LOOKUP'd, just expand <br>
the 'GlusterFS' part in the 'packet details' and
check the 'Basename'. <br>
Maybe this filename (without directory structure)
does give you any <br>
ideas of which activity is causing the LOOKUPs.<br>
<br>
HTH,<br>
Niels<br>
<br>
<blockquote type="cite"><br>
The configuration I have for Gluster is:<br>
<br>
[guru@srv1 ~]$ sudo gluster volume info<br>
[sudo] password for guru: <br>
<br>
Volume Name: gv0<br>
Type: Replicate<br>
Volume ID: dc8dc3f2-f5bd-4047-9101-acad04695442<br>
Status: Started<br>
Number of Bricks: 1 x 2 = 2<br>
Transport-type: tcp<br>
Bricks:<br>
Brick1: srv1:/root/gluster-vol0<br>
Brick2: srv2:/root/gluster-vol0<br>
Options Reconfigured:<br>
cluster.lookup-unhashed: on<br>
performance.cache-refresh-timeout: 60<br>
performance.cache-size: 1GB<br>
storage.health-check-interval: 30<br>
<br>
<br>
<br>
Please suggest how to fine tune the RPC
calls/reply. <br>
</blockquote>
<br>
<br>
</blockquote>
</div>
<br>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
Gluster-users mailing list
<a moz-do-not-send="true" class="moz-txt-link-abbreviated" href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a>
<a moz-do-not-send="true" class="moz-txt-link-freetext" href="http://supercolony.gluster.org/mailman/listinfo/gluster-users">http://supercolony.gluster.org/mailman/listinfo/gluster-users</a></pre>
</blockquote>
<br>
</div>
</blockquote>
</div>
<br>
</blockquote>
<br>
</body>
</html>