<div dir="ltr"><div class="gmail_extra">KP, Vipul,</div><div class="gmail_extra"><br></div><div class="gmail_extra">It will be awesome to get io-stats like instrumentation on the client side. Here are some further thoughts on how to implement that. If you have a recent git HEAD build, I would suggest that you explore the latency stats on the client side exposed through meta at $MNT/.meta/graphs/active/$xlator/profile. You can enable latency measurement with "echo 1 > $MNT/.meta/measure_latency". I would suggest extending these stats with the extra ones io-stats has, and make glusterfsiostats expose these stats. </div>
<div class="gmail_extra"><br></div><div class="gmail_extra">If you can compare libglusterfs/src/latency.c:gf_latency_begin(), gf_latency_end() and gf_latency_udpate() with the macros in io-stats.c <span style="color:rgb(0,0,0)">UPDATE_PROFILE_STATS() and </span><span style="color:rgb(0,0,0)">START_FOP_LATENCY(), you will quickly realize how a lot of logic is duplicated between io-stats and latency.c. If you can enhance latency.c and make it capture the remaining stats what io-stats is capturing, t</span>he benefits of this approach would be:<div class="gmail_extra">
<br></div><div class="gmail_extra">- stats are already getting captured at all xlator levels, and not just at the position where io-stats is inserted</div><div class="gmail_extra">- file like interface makes the stats more easily inspectable and consumable, and updated on the fly</div>
<div class="gmail_extra">- conforms with the way rest of the internals are exposed through $MNT/.meta</div><div class="gmail_extra"><br></div><div class="gmail_extra">In order to this, you might want to look into:</div><div class="gmail_extra">
<br></div><div class="gmail_extra">- <span style="color:rgb(0,0,0)">latency.c as of today captures fop count, mean latency, total time, whereas io-stats measures these along with min-time, max-time and block-size histogram.</span></div>
<div class="gmail_extra"><span style="color:rgb(0,0,0)">- extend gf_proc_dump_latency_info() to dump the new stats</span></div><div class="gmail_extra"><span style="color:rgb(0,0,0)">- either prettify that output like 'volume profile info' output, or JSONify it like xlators/meta/src/frames-file.c</span></div>
<div class="gmail_extra"><span style="color:rgb(0,0,0)">- add support for cumulative vs interval stats (store an extra copy of this->latencies[])</span></div><div class="gmail_extra"><span style="color:rgb(0,0,0)"><br>
</span></div><div class="gmail_extra"><span style="color:rgb(0,0,0)">etc..</span></div><div class="gmail_extra"><span style="color:rgb(0,0,0)"><br></span></div><div class="gmail_extra"><span style="color:rgb(0,0,0)">Thanks!</span></div>
<div class="gmail_extra"><span style="color:rgb(0,0,0)"><br></span></div><div class="gmail_extra"><br></div><div class="gmail_quote">On Fri, Apr 25, 2014 at 9:09 PM, Krishnan Parthasarathi <span dir="ltr"><<a href="mailto:kparthas@redhat.com" target="_blank">kparthas@redhat.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">[Resending due to gluster-devel mailing list issue]<br>
<br>
Apologies for the late reply.<br>
<br>
glusterd uses its socket connection with brick processes (where io-stats xlator is loaded) to<br>
gather information from io-stats via an RPC request. This facility is restricted to brick processes<br>
as it stands today.<br>
<br>
Some background ...<br>
io-stats xlator is loaded, both in GlusterFS mounts and brick processes. So, we have the capabilities<br>
to monitor I/O statistics on both sides. To collect I/O statistics at the server side, we have<br>
<br>
# gluster volume profile VOLNAME [start | info | stop]<br>
AND<br>
#gluster volume top VOLNAME info [and other options]<br>
<br>
We don't have a usable way of gathering I/O statistics (not monitoring, though the counters could be enhanced)<br>
at the client-side, ie. for a given mount point. This is the gap glusterfsiostat aims to fill. We need to remember<br>
that the machines hosting GlusterFS mounts may not have glusterd installed on them.<br>
<br>
We are considering rrdtool as a possible statistics database because it seems like a natural choice for storing time-series<br>
data. rrdtool is capable of answering high-level statistical queries on statistics that were logged in it by io-stats xlator<br>
over and above printing running counters periodically.<br>
<br>
Hope this gives some more clarity on what we are thinking.<br>
<br>
thanks,<br>
Krish<br>
----- Original Message -----<br>
<br>
> Probably me not understanding.<br>
<br>
> the comment "iostats making data available to glusterd over RPC" - is what I<br>
> latched on to. I wondered whether this meant that a socket could be opened<br>
> that way to get at the iostats data flow.<br>
<br>
> Cheers,<br>
<br>
> PC<br>
<br>
> ----- Original Message -----<br>
<br>
> > From: "Vipul Nayyar" <<a href="mailto:nayyar_vipul@yahoo.com">nayyar_vipul@yahoo.com</a>><br>
><br>
> > To: "Paul Cuzner" <<a href="mailto:pcuzner@redhat.com">pcuzner@redhat.com</a>>, "Krishnan Parthasarathi"<br>
> > <<a href="mailto:kparthas@redhat.com">kparthas@redhat.com</a>><br>
><br>
> > Cc: "Vijay Bellur" <<a href="mailto:vbellur@redhat.com">vbellur@redhat.com</a>>, "gluster-devel"<br>
> > <<a href="mailto:gluster-devel@nongnu.org">gluster-devel@nongnu.org</a>><br>
><br>
> > Sent: Thursday, 20 February, 2014 5:06:27 AM<br>
><br>
> > Subject: Re: [Gluster-devel] New project on the Forge - gstatus<br>
><br>
<br>
> > Hi Paul,<br>
><br>
<br>
> > I'm really not sure, if this can be done in python(at least comfortably).<br>
> > Maybe we can tread on the same path as Justin's glusterflow in python. But<br>
> > I<br>
> > don't think, all the io-stats counters will be available with the way how<br>
> > Justin's used Jeff Darcy's previous work to build his tool. I can be wrong.<br>
> > My knowledge is a bit incomplete and based on very less experience as a<br>
> > user<br>
> > and an amateur Gluster developer. Please do correct me, if I can be.<br>
><br>
<br>
> > Regards<br>
><br>
> > Vipul Nayyar<br>
><br>
_______________________________________________<br>
Gluster-devel mailing list<br>
<a href="mailto:Gluster-devel@gluster.org">Gluster-devel@gluster.org</a><br>
<a href="http://supercolony.gluster.org/mailman/listinfo/gluster-devel" target="_blank">http://supercolony.gluster.org/mailman/listinfo/gluster-devel</a><br>
</blockquote></div><br></div></div>