<html><head><meta http-equiv="Content-Type" content="text/html charset=windows-1252"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">Hello Niels,<div><br></div><div>I did a net hogs on the interface to see what process might be using the bandwidth,</div><div><br></div><div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">NetHogs version 0.8.0</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0); min-height: 14px;"><br></div><div style="margin: 0px; font-size: 10px; font-family: Monaco; background-color: rgb(245, 245, 245);"> PID USER PROGRAM DEV SENT RECEIVED </div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">18611 root /usr/sbin/glusterfsd tun0<span class="Apple-tab-span" style="white-space:pre"> </span> 16.307 17.547 KB/sec</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">1055 root /usr/sbin/glusterfs tun0<span class="Apple-tab-span" style="white-space:pre"> </span> 17.249 16.259 KB/sec</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">13439 guru sshd: guru@pts/0 tun0<span class="Apple-tab-span" style="white-space:pre"> </span> 0.966 0.051 KB/sec</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">18625 root /usr/sbin/glusterfs tun0<span class="Apple-tab-span" style="white-space:pre"> </span> 0.000 0.000 KB/sec</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">18629 root /usr/sbin/glusterfs tun0<span class="Apple-tab-span" style="white-space:pre"> </span> 0.000 0.000 KB/sec</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">9636 root /usr/sbin/glusterd tun0<span class="Apple-tab-span" style="white-space:pre"> </span> 0.000 0.000 KB/sec</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">? root unknown TCP 0.000 0.000 KB/sec</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0); min-height: 14px;"><br></div><div style="margin: 0px; font-size: 10px; font-family: Monaco; background-color: rgb(245, 245, 245);"> TOTAL 34.523 33.856 KB/sec </div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0); min-height: 14px;"><br></div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0); min-height: 14px;"><br></div><div><div><br></div><div><br></div><div>Its glusterfs and glusterfsd process.</div><div><br></div><div>I looked at the capture file and see that the lookup is being made on random files.</div><div><br></div><div>For PID information, please see this:</div><div><br></div><div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">[guru@srv2 ~]$ sudo netstat -tpn | grep 49152</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">tcp 0 0 127.0.0.1:49152 127.0.0.1:1012 ESTABLISHED 18611/glusterfsd </div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">tcp 0 0 127.0.0.1:49152 127.0.0.1:1016 ESTABLISHED 18611/glusterfsd </div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">tcp 0 0 127.0.0.1:1016 127.0.0.1:49152 ESTABLISHED 18625/glusterfs </div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">tcp 0 0 10.8.0.6:1021 10.8.0.1:49152 ESTABLISHED 1055/glusterfs </div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">tcp 0 0 10.8.0.6:49152 10.8.0.1:1017 ESTABLISHED 18611/glusterfsd </div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">tcp 0 0 10.8.0.6:1020 10.8.0.1:49152 ESTABLISHED 18629/glusterfs </div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">tcp 0 0 127.0.0.1:1023 127.0.0.1:49152 ESTABLISHED 18629/glusterfs </div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">tcp 0 0 10.8.0.6:49152 10.8.0.1:1022 ESTABLISHED 18611/glusterfsd </div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">tcp 0 0 10.8.0.6:49152 10.8.0.1:1021 ESTABLISHED 18611/glusterfsd </div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">tcp 0 0 127.0.0.1:49152 127.0.0.1:1023 ESTABLISHED 18611/glusterfsd </div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">tcp 0 0 127.0.0.1:1012 127.0.0.1:49152 ESTABLISHED 1055/glusterfs </div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">tcp 0 0 10.8.0.6:1019 10.8.0.1:49152 ESTABLISHED 18625/glusterfs </div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">[guru@srv2 ~]$ ps -v 18611</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">Warning: bad syntax, perhaps a bogus '-'? See /usr/share/doc/procps-3.2.8/FAQ</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);"> PID TTY STAT TIME MAJFL TRS DRS RSS %MEM COMMAND</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">18611 ? Ssl 14:12 0 0 650068 20404 2.0 /usr/sbin/glusterfsd -s srv2 --volfile-id gv0.srv2.root-gluster-vol0 -p /var/lib/glusterd/vols/gv0</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">[guru@srv2 ~]$ ps -v 18629</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">Warning: bad syntax, perhaps a bogus '-'? See /usr/share/doc/procps-3.2.8/FAQ</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);"> PID TTY STAT TIME MAJFL TRS DRS RSS %MEM COMMAND</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">18629 ? Ssl 0:04 0 0 333296 17380 1.7 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/lib/glusterd/glustershd/r</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">[guru@srv2 ~]$ </div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">[guru@srv2 ~]$ </div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">[guru@srv2 ~]$ ps -v 18629</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">Warning: bad syntax, perhaps a bogus '-'? See /usr/share/doc/procps-3.2.8/FAQ</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);"> PID TTY STAT TIME MAJFL TRS DRS RSS %MEM COMMAND</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">18629 ? Ssl 0:04 0 0 333296 17380 1.7 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/lib/glusterd/glustershd/run/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/823fa3197e2d1841be888</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">[guru@srv2 ~]$ </div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">[guru@srv2 ~]$ </div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">[guru@srv2 ~]$ </div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">[guru@srv2 ~]$ </div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">[guru@srv2 ~]$ ps -v 18629</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">Warning: bad syntax, perhaps a bogus '-'? See /usr/share/doc/procps-3.2.8/FAQ</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);"> PID TTY STAT TIME MAJFL TRS DRS RSS %MEM COMMAND</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">18629 ? Ssl 0:04 0 0 333296 17380 1.7 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/lib/glusterd/glustershd/run/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/823fa3197e2d1841be8881500723b063.socket --xlator-option *replicate*.node-uuid=84af83c9-0a29-</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">[guru@srv2 ~]$ </div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">[guru@srv2 ~]$ </div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">[guru@srv2 ~]$ ps -v 18625</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">Warning: bad syntax, perhaps a bogus '-'? See /usr/share/doc/procps-3.2.8/FAQ</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);"> PID TTY STAT TIME MAJFL TRS DRS RSS %MEM COMMAND</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">18625 ? Ssl 0:03 0 0 239528 41040 4.0 /usr/sbin/glusterfs -s localhost --volfile-id gluster/nfs -p /var/lib/glusterd/nfs/run/nfs.pid -l /var/log/glusterfs/nfs.log -S /var/run/5ad5b036fd636cc5dddffa73593e4089.socket</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">[guru@srv2 ~]$ sudo nethogs tun0</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">Waiting for first packet to arrive (see <a href="http://sourceforge.net">sourceforge.net</a> bug 1019381)</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">[guru@srv2 ~]$ rpm -qa | grep gluster</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">glusterfs-3.5.1-1.el6.x86_64</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">glusterfs-cli-3.5.1-1.el6.x86_64</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">glusterfs-libs-3.5.1-1.el6.x86_64</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">glusterfs-fuse-3.5.1-1.el6.x86_64</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">glusterfs-server-3.5.1-1.el6.x86_64</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">glusterfs-api-3.5.1-1.el6.x86_64</div><div style="margin: 0px; font-size: 10px; font-family: Monaco; color: rgb(245, 245, 245); background-color: rgb(0, 0, 0);">[guru@srv2 ~]$ </div></div><div><br></div><div><br></div><div>I don’t see anything odd here. Please suggest.</div><div><br></div><div>Thanks,</div><div>Gurdeep.</div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div>On 7 Jul 2014, at 9:06 pm, Niels de Vos <<a href="mailto:ndevos@redhat.com">ndevos@redhat.com</a>> wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite">On Sun, Jul 06, 2014 at 11:28:51PM +1000, Gurdeep Singh (Guru) wrote:<br><blockquote type="cite">Hello,<br><br>I have setup gluster in replicate type and its working fine.<br><br>I am seeing a constant chatting between the hosts for lookup call and <br>lookup reply. I am trying to understand as to why this traffic is <br>being initiated constantly. Please look at the attached image. This <br>traffic is using around 200KB/s of constant bandwidth and is <br>exhausting our allocated monthly bandwidth on our 2 VPS.<br></blockquote><br>You can use Wireshark to identify which process does the LOOKUP calls. <br>For this, do the following:<br><br>1. select a LOOKUP Call<br>2. enable the 'packet details' pane (found in the main menu, 'view')<br>3. expand the 'Transmission Control Protocol' tree<br>4. check the 'Source port' of the LOOKUP Call<br><br>Together with the 'Source' and the 'Source port' you can go to the <br>server that matches the 'Source' address. A command like this would give <br>you the PID of the process in the right column:<br><br> # netstat -tpn | grep $SOURCE_PORT<br><br>And with 'ps -v $PID' you can check which process is responsible for the <br>LOOKUP. This process can be a fuse-mount, self-heal-daemon or any other <br>glusterfs-client. Depending on the type of client, you maybe can tune <br>the workload or other options a little.<br><br>In Wireshark you can also check what filename is LOOKUP'd, just expand <br>the 'GlusterFS' part in the 'packet details' and check the 'Basename'. <br>Maybe this filename (without directory structure) does give you any <br>ideas of which activity is causing the LOOKUPs.<br><br>HTH,<br>Niels<br><br><blockquote type="cite"><br>The configuration I have for Gluster is:<br><br>[guru@srv1 ~]$ sudo gluster volume info<br>[sudo] password for guru: <br><br>Volume Name: gv0<br>Type: Replicate<br>Volume ID: dc8dc3f2-f5bd-4047-9101-acad04695442<br>Status: Started<br>Number of Bricks: 1 x 2 = 2<br>Transport-type: tcp<br>Bricks:<br>Brick1: srv1:/root/gluster-vol0<br>Brick2: srv2:/root/gluster-vol0<br>Options Reconfigured:<br>cluster.lookup-unhashed: on<br>performance.cache-refresh-timeout: 60<br>performance.cache-size: 1GB<br>storage.health-check-interval: 30<br><br><br><br>Please suggest how to fine tune the RPC calls/reply. <br></blockquote><br><br></blockquote></div><br></div></body></html>