<div dir="ltr">Thanks Justin, I found the problem. The VM can be deleted now.<div><br></div><div>Turns out, there was more than enough time for the rebalance to complete. But we hit a race, which caused a command to fail.</div>
<div><br></div><div>The particular test that failed is waiting for rebalance to finish. It does this by doing a 'gluster volume rebalance <> status' command and checking the result. The EXPECT_WITHIN function runs this command till we have a match, the command fails or the timeout happens.</div>
<div><br></div><div>For a rebalance status command, glusterd sends a request to the rebalance process (as a brick_op) to get the latest stats. It had done the same in this case as well. But while glusterd was waiting for the reply, the rebalance completed and the process stopped itself. This caused the rpc connection between glusterd and rebalance proc to close. This caused the all pending requests to be unwound as failures. Which in turnlead to the command failing.</div>
<div><br></div><div>I cannot think of a way to avoid this race from within glusterd. For this particular test, we could avoid using the 'rebalance status' command if we directly checked the rebalance process state using its pid etc. I don't particularly approve of this approach, as I think I used the 'rebalance status' command for a reason. But I currently cannot recall the reason, and if cannot come with it soon, I wouldn't mind changing the test to avoid rebalance status.</div>
<div><br></div><div>~kaushal</div><div><br></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, May 22, 2014 at 5:22 PM, Justin Clift <span dir="ltr"><<a href="mailto:justin@gluster.org" target="_blank">justin@gluster.org</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="">On 22/05/2014, at 12:32 PM, Kaushal M wrote:<br>
> I haven't yet. But I will.<br>
><br>
> Justin,<br>
> Can I get take a peek inside the vm?<br>
<br>
</div>Sure.<br>
<br>
IP: 23.253.57.20<br>
User: root<br>
Password: foobar123<br>
<br>
The stdout log from the regression test is in /tmp/regression.log.<br>
<br>
The GlusterFS git repo is in /root/glusterfs. Um, you should be<br>
able to find everything else pretty easily.<br>
<br>
Btw, this is just a temp VM, so feel free to do anything you want<br>
with it. When you're finished with it let me know so I can delete<br>
it. :)<br>
<span class="HOEnZb"><font color="#888888"><br>
+ Justin<br>
</font></span><div class="HOEnZb"><div class="h5"><br>
<br>
> ~kaushal<br>
><br>
><br>
> On Thu, May 22, 2014 at 4:53 PM, Pranith Kumar Karampuri <<a href="mailto:pkarampu@redhat.com">pkarampu@redhat.com</a>> wrote:<br>
> Kaushal,<br>
> Rebalance status command seems to be failing sometimes. I sent a mail about such spurious failure earlier today. Did you get a chance to look at the logs and confirm that rebalance didn't fail and it is indeed a timeout?<br>
><br>
> Pranith<br>
> ----- Original Message -----<br>
> > From: "Kaushal M" <<a href="mailto:kshlmster@gmail.com">kshlmster@gmail.com</a>><br>
> > To: "Pranith Kumar Karampuri" <<a href="mailto:pkarampu@redhat.com">pkarampu@redhat.com</a>><br>
> > Cc: "Justin Clift" <<a href="mailto:justin@gluster.org">justin@gluster.org</a>>, "Gluster Devel" <<a href="mailto:gluster-devel@gluster.org">gluster-devel@gluster.org</a>><br>
> > Sent: Thursday, May 22, 2014 4:40:25 PM<br>
> > Subject: Re: [Gluster-devel] bug-857330/normal.t failure<br>
> ><br>
> > The test is waiting for rebalance to finish. This is a rebalance with some<br>
> > actual data so it could have taken a long time to finish. I did set a<br>
> > pretty high timeout, but it seems like it's not enough for the new VMs.<br>
> ><br>
> > Possible options are,<br>
> > - Increase this timeout further<br>
> > - Reduce the amount of data. Currently this is 100 directories with 10<br>
> > files each of size between 10-500KB<br>
> ><br>
> > ~kaushal<br>
> ><br>
> ><br>
> > On Thu, May 22, 2014 at 3:59 PM, Pranith Kumar Karampuri <<br>
> > <a href="mailto:pkarampu@redhat.com">pkarampu@redhat.com</a>> wrote:<br>
> ><br>
> > > Kaushal has more context about these CCed. Keep the setup until he<br>
> > > responds so that he can take a look.<br>
> > ><br>
> > > Pranith<br>
> > > ----- Original Message -----<br>
> > > > From: "Justin Clift" <<a href="mailto:justin@gluster.org">justin@gluster.org</a>><br>
> > > > To: "Pranith Kumar Karampuri" <<a href="mailto:pkarampu@redhat.com">pkarampu@redhat.com</a>><br>
> > > > Cc: "Gluster Devel" <<a href="mailto:gluster-devel@gluster.org">gluster-devel@gluster.org</a>><br>
> > > > Sent: Thursday, May 22, 2014 3:54:46 PM<br>
> > > > Subject: bug-857330/normal.t failure<br>
> > > ><br>
> > > > Hi Pranith,<br>
> > > ><br>
> > > > Ran a few VM's with your Gerrit CR 7835 applied, and in "DEBUG"<br>
> > > > mode (I think).<br>
> > > ><br>
> > > > One of the VM's had a failure in bug-857330/normal.t:<br>
> > > ><br>
> > > > Test Summary Report<br>
> > > > -------------------<br>
> > > > ./tests/basic/rpm.t (Wstat: 0 Tests: 0<br>
> > > Failed:<br>
> > > > 0)<br>
> > > > Parse errors: Bad plan. You planned 8 tests but ran 0.<br>
> > > > ./tests/bugs/bug-857330/normal.t (Wstat: 0 Tests: 24<br>
> > > Failed:<br>
> > > > 1)<br>
> > > > Failed test: 13<br>
> > > > Files=230, Tests=4369, 5407 wallclock secs ( 2.13 usr 1.73 sys +<br>
> > > 941.82<br>
> > > > cusr 645.54 csys = 1591.22 CPU)<br>
> > > > Result: FAIL<br>
> > > ><br>
> > > > Seems to be this test:<br>
> > > ><br>
> > > > COMMAND="volume rebalance $V0 status"<br>
> > > > PATTERN="completed"<br>
> > > > EXPECT_WITHIN 300 $PATTERN get-task-status<br>
> > > ><br>
> > > > Is this one on your radar already?<br>
> > > ><br>
> > > > Btw, this VM is still online. Can give you access to retrieve logs<br>
> > > > if useful.<br>
> > > ><br>
> > > > + Justin<br>
> > > ><br>
> > > > --<br>
> > > > Open Source and Standards @ Red Hat<br>
> > > ><br>
> > > > <a href="http://twitter.com/realjustinclift" target="_blank">twitter.com/realjustinclift</a><br>
> > > ><br>
> > > ><br>
> > > _______________________________________________<br>
> > > Gluster-devel mailing list<br>
> > > <a href="mailto:Gluster-devel@gluster.org">Gluster-devel@gluster.org</a><br>
> > > <a href="http://supercolony.gluster.org/mailman/listinfo/gluster-devel" target="_blank">http://supercolony.gluster.org/mailman/listinfo/gluster-devel</a><br>
> > ><br>
> ><br>
><br>
<br>
--<br>
Open Source and Standards @ Red Hat<br>
<br>
<a href="http://twitter.com/realjustinclift" target="_blank">twitter.com/realjustinclift</a><br>
<br>
</div></div></blockquote></div><br></div>