<html><body><div style="font-family: garamond,new york,times,serif; font-size: 12pt; color: #000000"><div><br></div><div><br></div><hr id="zwchr"><blockquote style="border-left:2px solid #1010FF;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;" data-mce-style="border-left: 2px solid #1010FF; margin-left: 5px; padding-left: 5px; color: #000; font-weight: normal; font-style: normal; text-decoration: none; font-family: Helvetica,Arial,sans-serif; font-size: 12pt;"><b>From: </b>"David Gibbons" <david.c.gibbons@gmail.com><br><b>To: </b>"Krutika Dhananjay" <kdhananj@redhat.com><br><b>Cc: </b>"gluster-users" <gluster-users@gluster.org><br><b>Sent: </b>Tuesday, December 2, 2014 6:54:59 PM<br><b>Subject: </b>Re: [Gluster-users] Upgraded from 3.4.1 to 3.5.2, quota no longer working<br><div><br></div><div dir="ltr">Thank you for the assistance.<div><br></div><div>Yesterday we started to have bricks on one server randomly crash. When the one server crashed, it would lock up the bricks on its replica as well. I ended up upgrading to 3.5.3, and noticed in the process that the libgfrpc and libgfxdr libraries were out of date on the server that was having crashed bricks. Upgrading to 3.5.3 and replacing the old versions of the libraries on the cranky server seems to have made everything happy again. </div></div></blockquote><div>Yes, one brick crash and a couple of other issues with quota that were reported in 3.5.2 were fixed in 3.5.3. Good to note that things are working fine now. :)</div><div>-Krutika</div><blockquote style="border-left:2px solid #1010FF;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;" data-mce-style="border-left: 2px solid #1010FF; margin-left: 5px; padding-left: 5px; color: #000; font-weight: normal; font-style: normal; text-decoration: none; font-family: Helvetica,Arial,sans-serif; font-size: 12pt;"><div dir="ltr"><div>Thanks again!</div><div>Dave</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Dec 2, 2014 at 2:28 AM, Krutika Dhananjay <span dir="ltr"><<a href="mailto:kdhananj@redhat.com" target="_blank" data-mce-href="mailto:kdhananj@redhat.com">kdhananj@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex" data-mce-style="margin: 0 0 0 .8ex; border-left: 1px #ccc solid; padding-left: 1ex;"><div><div style="font-family:garamond,new york,times,serif;font-size:12pt;color:#000000" data-mce-style="font-family: garamond,new york,times,serif; font-size: 12pt; color: #000000;"><div>Hi,</div><div><br></div><div>Are you sure the post-upgrade script ran to completion?</div><div>Here is one way to confirm whether that is the case: check if the quota configured directories have an xattr called "trusted.glusterfs.quota.limit-set" set on them in the respective bricks.</div><div><br></div><div>For example, here's what mine looks like:</div><div><br></div><div><p style="margin:0px" data-mce-style="margin: 0px;">[root@haddock 1]# pwd<br>/brick/1<br>[root@haddock 1]# getfattr -d -m . -e hex 1<br># file: 1<br>security.selinux=0x73797374656d5f753a6f626a6563745f723a64656661756c745f743a733000<br>trusted.gfid=0x57d0a561ca574d1cb0428f38d1c06e85<br>trusted.glusterfs.dht=0x00000001000000007fffffffffffffff<br>trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri=0x0000000000000a00<br>trusted.glusterfs.quota.dirty=0x3000<br>trusted.glusterfs.quota.limit-set=0x0000000006400000ffffffffffffffff<br>trusted.glusterfs.quota.size=0x0000000000000a00</p><p style="margin:0px" data-mce-style="margin: 0px;"><br></p><p style="margin:0px" data-mce-style="margin: 0px;">where /brick/1 is the brick directory and under it "1" is the name of one of the quota-configured directories.</p></div><div><br></div><div>I believe your quota configurations are backed up at /var/tmp/glusterfs/quota-config-backup/vol_<volname> which you can use to get the quota-configured directory names.</div><div>As for operating version, I think it is sufficient for it to be at 3 for the 3.5.x quota to work.</div><div><br></div><div>-Krutika</div><div><br></div><hr><blockquote style="border-left:2px solid #1010ff;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt" data-mce-style="border-left: 2px solid #1010ff; margin-left: 5px; padding-left: 5px; color: #000; font-weight: normal; font-style: normal; text-decoration: none; font-family: Helvetica,Arial,sans-serif; font-size: 12pt;"><b>From: </b>"David Gibbons" <<a href="mailto:david.c.gibbons@gmail.com" target="_blank" data-mce-href="mailto:david.c.gibbons@gmail.com">david.c.gibbons@gmail.com</a>><br><b>To: </b>"Krutika Dhananjay" <<a href="mailto:kdhananj@redhat.com" target="_blank" data-mce-href="mailto:kdhananj@redhat.com">kdhananj@redhat.com</a>><br><b>Cc: </b>"gluster-users" <<a href="mailto:gluster-users@gluster.org" target="_blank" data-mce-href="mailto:gluster-users@gluster.org">gluster-users@gluster.org</a>><br><b>Sent: </b>Monday, December 1, 2014 6:35:55 PM<br><b>Subject: </b>Re: [Gluster-users] Upgraded from 3.4.1 to 3.5.2, quota no longer working<div><div class="h5"><br><div><br></div><div dir="ltr">Certainly, thank you for your response:<div><br></div><div>Quotad is running on all nodes:</div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex" data-mce-style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: #cccccc; border-left-style: solid; padding-left: 1ex;"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex" data-mce-style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: #cccccc; border-left-style: solid; padding-left: 1ex;">[root@gfs-a-1 ~]# ps aux | grep quotad</blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex" data-mce-style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: #cccccc; border-left-style: solid; padding-left: 1ex;">root 3004 0.0 0.4 241368 68552 ? Ssl Nov30 0:07 /usr/local/sbin/glusterfs -s localhost --volfile-id gluster/quotad -p /var/lib/glusterd/quotad/run/quotad.pid -l /usr/local/var/log/glusterfs/quotad.log -S /var/run/9d02605105ef0e74d913a4671c1143a1.socket --xlator-option *replicate*.data-self-heal=off --xlator-option *replicate*.metadata-self-heal=off --xlator-option *replicate*.entry-self-heal=off</blockquote></blockquote><div><br></div><div>And the relevant output from gluster volume status shares per your request:</div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex" data-mce-style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: #cccccc; border-left-style: solid; padding-left: 1ex;">[root@gfs-a-1 ~]# gluster volume status shares | grep Quota</blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex" data-mce-style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: #cccccc; border-left-style: solid; padding-left: 1ex;">Quota Daemon on localhost N/A Y 3004</blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex" data-mce-style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: #cccccc; border-left-style: solid; padding-left: 1ex;">Quota Daemon on gfs-a-3 N/A Y 32307</blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex" data-mce-style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: #cccccc; border-left-style: solid; padding-left: 1ex;">Quota Daemon on gfs-a-4 N/A Y 10818</blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex" data-mce-style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: #cccccc; border-left-style: solid; padding-left: 1ex;">Quota Daemon on gfs-a-2 N/A Y 12292</blockquote><div> No log entries are created in /var/log/glusterfs/quotad.log when I run a quota list; all of the log entries are from yesterday. They do indicate a version mis-match, although I can't seem to locate where that version is specified:</div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex" data-mce-style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: #cccccc; border-left-style: solid; padding-left: 1ex;"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex" data-mce-style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: #cccccc; border-left-style: solid; padding-left: 1ex;">[2014-11-30 13:21:55.173081] I [client-handshake.c:1474:client_setvolume_cbk] 0-shares-client-14: Server and Client lk-version numbers are not same, reopening the fds</blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex" data-mce-style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: #cccccc; border-left-style: solid; padding-left: 1ex;">[2014-11-30 13:21:55.173170] I [client-handshake.c:450:client_set_lk_version_cbk] 0-shares-client-14: Server lk version = 1</blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex" data-mce-style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: #cccccc; border-left-style: solid; padding-left: 1ex;">[2014-11-30 13:21:55.178739] I [rpc-clnt.c:1729:rpc_clnt_reconfig] 0-shares-client-9: changing port to 49154 (from 0)</blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex" data-mce-style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: #cccccc; border-left-style: solid; padding-left: 1ex;">[2014-11-30 13:21:55.181170] I [client-handshake.c:1677:select_server_supported_programs] 0-shares-client-9: Using Program GlusterFS 3.3, Num (1298437), Version (330)</blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex" data-mce-style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: #cccccc; border-left-style: solid; padding-left: 1ex;">[2014-11-30 13:21:55.181386] I [client-handshake.c:1462:client_setvolume_cbk] 0-shares-client-9: Connected to <a href="http://172.16.10.13:49154" target="_blank" data-mce-href="http://172.16.10.13:49154">172.16.10.13:49154</a>, attached to remote volume '/mnt/a-3-shares-brick-3/brick'.</blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex" data-mce-style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: #cccccc; border-left-style: solid; padding-left: 1ex;">[2014-11-30 13:21:55.181401] I [client-handshake.c:1474:client_setvolume_cbk] 0-shares-client-9: Server and Client lk-version numbers are not same, reopening the fds</blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex" data-mce-style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: #cccccc; border-left-style: solid; padding-left: 1ex;">[2014-11-30 13:21:55.181535] I [client-handshake.c:450:client_set_lk_version_cbk] 0-shares-client-9: Server lk version = 1</blockquote></blockquote><div><br></div><div>I see the operational mode for the volume as "3". I saw a non-related thread that indicated this number should be more digits on a cluster running 3.5.2. The other thread also indicated that quota may not work if the volume version number was not compatible with the quota version running on the cluster. I can't seem to find the link right now.</div><div><br></div><div>It's almost as if the volume version did not get upgraded when the server version was upgraded. Is that possible?</div><div><br></div><div>Cheers,</div><div>Dave</div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Sun, Nov 30, 2014 at 11:46 PM, Krutika Dhananjay <span dir="ltr"><<a href="mailto:kdhananj@redhat.com" target="_blank" data-mce-href="mailto:kdhananj@redhat.com">kdhananj@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex" data-mce-style="margin: 0 0 0 .8ex; border-left: 1px #ccc solid; padding-left: 1ex;"><div><div style="font-family:garamond,new york,times,serif;font-size:12pt;color:#000000" data-mce-style="font-family: garamond,new york,times,serif; font-size: 12pt; color: #000000;"><div>Hi,</div><div><br></div><div>Could you confirm whether quotad (Quota Daemon) is online from the output of `gluster volume status shares`?</div><div><br></div><div>Also, could you share quota daemon's log file from the node where you executed `quota list` command, which you will find at @ /var/log/glusterfs/quotad.log?</div><div><br></div><div>-Krutika</div><div><br></div><hr><blockquote style="border-left:2px solid #1010ff;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt" data-mce-style="border-left: 2px solid #1010ff; margin-left: 5px; padding-left: 5px; color: #000; font-weight: normal; font-style: normal; text-decoration: none; font-family: Helvetica,Arial,sans-serif; font-size: 12pt;"><b>From: </b>"David Gibbons" <<a href="mailto:david.c.gibbons@gmail.com" target="_blank" data-mce-href="mailto:david.c.gibbons@gmail.com">david.c.gibbons@gmail.com</a>><br><b>To: </b>"gluster-users" <<a href="mailto:gluster-users@gluster.org" target="_blank" data-mce-href="mailto:gluster-users@gluster.org">gluster-users@gluster.org</a>><br><b>Sent: </b>Sunday, November 30, 2014 9:57:02 PM<br><b>Subject: </b>[Gluster-users] Upgraded from 3.4.1 to 3.5.2, quota no longer working<div><div><br><div><br></div><div dir="ltr">Hi All,<div><br></div><div>I performed a long-awaited upgrade from 3.4.1 to 3.5.2 today following the instructions for an offline upgrade outlined here:</div><div><a href="http://www.gluster.org/community/documentation/index.php/Upgrade_to_3.5" target="_blank" data-mce-href="http://www.gluster.org/community/documentation/index.php/Upgrade_to_3.5">http://www.gluster.org/community/documentation/index.php/Upgrade_to_3.5</a><br></div><div><br></div><div>I ran the pre- and post- upgrade scripts as instructed, intending to move the quotas over to the new version. The upgrade seemed to go well, the volume is online and it appears to be functioning properly.</div><div><br></div><div>When I attempt to check quotas, the list is empty:</div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex" data-mce-style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: #cccccc; border-left-style: solid; padding-left: 1ex;"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex" data-mce-style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: #cccccc; border-left-style: solid; padding-left: 1ex;">[root@gfs-a-1 glusterfs]# gluster volume quota shares list</blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex" data-mce-style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: #cccccc; border-left-style: solid; padding-left: 1ex;"> Path Hard-limit Soft-limit Used Available</blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex" data-mce-style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: #cccccc; border-left-style: solid; padding-left: 1ex;">--------------------------------------------------------------------------------</blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex" data-mce-style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: #cccccc; border-left-style: solid; padding-left: 1ex;">[root@gfs-a-1 glusterfs]#</blockquote><div><br></div></blockquote><div><br></div><div>And upon execution of that command, the cli.log file fills up with entries like this. I am assuming it's one cli log entry per quota entry:</div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex" data-mce-style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: #cccccc; border-left-style: solid; padding-left: 1ex;"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex" data-mce-style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: #cccccc; border-left-style: solid; padding-left: 1ex;">[2014-11-30 14:00:02.154143] W [cli-rpc-ops.c:2469:print_quota_list_from_quotad] 0-cli: path key is not present in dict</blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex" data-mce-style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: #cccccc; border-left-style: solid; padding-left: 1ex;">[2014-11-30 14:00:02.160507] W [cli-rpc-ops.c:2469:print_quota_list_from_quotad] 0-cli: path key is not present in dict</blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex" data-mce-style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: #cccccc; border-left-style: solid; padding-left: 1ex;">[2014-11-30 14:00:02.167947] W [cli-rpc-ops.c:2469:print_quota_list_from_quotad] 0-cli: path key is not present in dic</blockquote></blockquote><div><br></div><div>So, it appears that somehow the quota database has become offline or corrupt. Any thoughts on what I can do to resolve this?</div><div><br></div><div>I have checked all of the binaries on all 4 machines in the cluster, and they all appear to be running the correct version:</div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex" data-mce-style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: #cccccc; border-left-style: solid; padding-left: 1ex;"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex" data-mce-style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: #cccccc; border-left-style: solid; padding-left: 1ex;">[root@gfs-a-1 glusterfs]# glusterfsd --version</blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex" data-mce-style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: #cccccc; border-left-style: solid; padding-left: 1ex;">glusterfs 3.5.2 built on Nov 30 2014 08:16:37</blockquote></blockquote><div><br></div><div>Cheers,</div><div>Dave </div></div><br></div></div>_______________________________________________<br>Gluster-users mailing list<br><a href="mailto:Gluster-users@gluster.org" target="_blank" data-mce-href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br><a href="http://supercolony.gluster.org/mailman/listinfo/gluster-users" target="_blank" data-mce-href="http://supercolony.gluster.org/mailman/listinfo/gluster-users">http://supercolony.gluster.org/mailman/listinfo/gluster-users</a></blockquote><div><br></div></div></div></blockquote></div><br></div></div></div></blockquote><div><br></div></div></div></blockquote></div><br></div></blockquote><div><br></div></div></body></html>