<div dir="ltr">+1 on Vijay's comment. Logging infrastructure is useless for quota monitoring. It should be handled via user interface (CLI and alerting). gluster cli can report them similar to volume top and profile commands. Even if we dont implement alert, sysadmin may simply put a cron job around gluster cli.</div>
<div class="gmail_extra"><br><br><div class="gmail_quote">On Mon, Nov 25, 2013 at 8:57 AM, Vijay Bellur <span dir="ltr"><<a href="mailto:vbellur@redhat.com" target="_blank">vbellur@redhat.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="im">On 11/19/2013 03:08 PM, Anuradha Talur wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi all,<br>
<br>
This mail is regarding quota logging for the following conditions :<br>
<br>
1. Soft-limit alerts - "Usage crossed soft-limit" and "usage above soft-limit"<br>
2. Quota exceeded warning<br>
<br>
Issue with the soft-limit alerts logging --<br>
Currently, the soft-limit alerts are logged in the brick logs when a write<br>
is made and soft-limit is exceeded. In a scenario where there are a large<br>
number of bricks, it is difficult for the system admin to poll all the<br>
bricks for a soft-limit alert.<br>
</blockquote>
<br></div>
If polling is the mechanism for a soft-limit alert, why not have it as part of a command? The quota list CLI or an alternate new command can provide the list of directories on which soft-limit has been breached. This CLI can be polled to determine which of the directories exceeds soft-limit.<div class="im">
<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
Issue with the quota exceeded logging --<br>
When the quota exceeds, op_errno is set to EDQUOT and eventually<br>
"Quota exceeded" logs are being logged in the client logs (nfs.log, fuse mnt log,..)<br>
as of now (Similar to other op_errors logging).<br>
An issue has been raised stating that the quota hard-limit and soft-limit<br>
exceeding logs should be logged in same file.<br>
</blockquote>
<br></div>
Why is logging being preferred over CLI? If logging has to be the preferred mechanism for alerting, can we not use LOG_ALERT across all log files?<br>
<br>
Logs being seen in nfs.log and fuse mount log are useful in determining what errors are being returned to applications. In addition logging in fuse and bricks can potentially be used by different consumers. The fuse log files would be useful for an administrator who manages compute and the brick log files are useful for an administrator who manages storage.<div class="im">
<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
So a need for a unified logging has been proposed. We came up with the<br>
solution that all the soft-limit alert logs and the quota exceeded logs<br>
should be logged in quotad.log file.<br>
</blockquote>
<br></div>
Why cannot these logs be in the brick log files? Since both quota and marker xlators are part of the brick stack, logging in brick log files seems to be a better choice.<div class="im"><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Resulting in the limit exceeded logs being logged in all the nodes in the<br>
cluster. The sys admin can poll any one node to get the status of soft-limit<br>
and hard-limit.<br>
</blockquote>
<br></div>
Wouldn't we be flooding all nodes with similar information? With a centralized logging infrastructure, we might end up with a lot of repeated/redundant logs.<div class="im"><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
Approach proposed to achieve this solution --<br>
During the event of soft-limit/hard-limit exceeding, glusterd is informed.<br>
glusterd of the current node should interact with the glusterd of the other<br>
nodes in the cluster indicating the issue.<br>
All the glusterd's talk to their respective quotad to log the limit exceed<br>
in their logs.<br>
<br>
The following bugs are related to the points made here --<br>
<a href="https://bugzilla.redhat.com/show_bug.cgi?id=1020816" target="_blank">https://bugzilla.redhat.com/<u></u>show_bug.cgi?id=1020816</a><br>
<a href="https://bugzilla.redhat.com/show_bug.cgi?id=1019302" target="_blank">https://bugzilla.redhat.com/<u></u>show_bug.cgi?id=1019302</a><br>
<br>
<br>
Is this approach recommended?<br>
</blockquote>
<br></div>
I don't think this is a very clean approach. Having a CLI and/or providing the right log messages in brick log files at level LOG_ALERT seems to be good enough for alerting.<span class="HOEnZb"><font color="#888888"><br>
<br>
-Vijay</font></span><div class="HOEnZb"><div class="h5"><br>
<br>
<br>
<br>
<br>
______________________________<u></u>_________________<br>
Gluster-devel mailing list<br>
<a href="mailto:Gluster-devel@nongnu.org" target="_blank">Gluster-devel@nongnu.org</a><br>
<a href="https://lists.nongnu.org/mailman/listinfo/gluster-devel" target="_blank">https://lists.nongnu.org/<u></u>mailman/listinfo/gluster-devel</a><br>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br>-ab<div><br>Imagination is more important than knowledge --Albert Einstein</div>
</div>