<div dir="ltr">On Wed, Jul 31, 2013 at 5:11 AM, Jeff Darcy <span dir="ltr"><<a href="mailto:jdarcy@redhat.com" target="_blank">jdarcy@redhat.com</a>></span> wrote:<br><div class="gmail_extra"><div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="im">On 07/31/2013 07:35 AM, Amar Tumballi wrote:<br>
</div><div class="im"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
I was trying to fire some regression builds on very minor patches today, and<br>
noticed (always known, but faced pain of 'waiting' today) that we can fire<br>
regression build on only one patch (or a patchset if its submitted with<br>
dependency added while submitting). And each regression run takes approx<br>
30mins.<br>
<br>
With this model, we can at max take only ~45 patches in a day, which won't<br>
scale up if we want to grow with more people participating in code<br>
contribution. Would be great to have an option to submit regression run with<br>
multiple patch numbers, (technically they should be applicable one top of<br>
other in any order if not dependent), and it should work fine. That way, we<br>
can handle more review load in future.<br>
</blockquote>
<br></div>
Maybe my brain has been baked too much by the sun, but I thought I'd seen cases<br>
where a regression run on a patch with dependencies automatically validated<br>
everything in the stack. Not so? That still places a burden on patch<br>
submitters to make sure dependencies are specified (shouldn't be a problem<br>
since the current tendency is to *over*specify dependencies) and on the person<br>
starting the run to pick the top of the stack, but it does allow us to kill<br>
multiple birds with one stone.<br>
<br>
As for scaling, isn't the basic solution to add more worker machines? That<br>
would multiply the daily throughput by the number of workers, and decrease<br>
latency for simultaneously submitted runs proportionally.<div class="HOEnZb"><div class="h5"><br></div></div></blockquote><div><br></div><div>The flip side of having too many patches regression-tested in parallel is that, since the regression test applies the patch in question on top of the current git HEAD _at the time of test execution_, we lose out on testing the "combined effect" of those multiple patches. This can result in master branch being in broken state even though every patch is tested (in isolation). And the breakage will be visible much later - when an unrelated patch is tested after the patches get (successfully tested and) merged independently. This has happened before too, even with the current "test one patch at a time" model. E.g:</div>
<div><br></div><div>1 - Patch A is tested [success]</div><div>2 - Patch B is tested [success]</div><div>3 - Patch A is merged</div><div>4 - Patch B is merged</div><div><git HEAD is broken now></div><div>5 - Patch C is tested [failure, because combined effect of A + B is tested only now]</div>
<div><br></div><div>The serial nature of today's testing limits such delays to some extent, as tested patches keep getting merged before regression test of new patches start. Parallelizing tests too much could potentially increase this "danger window".</div>
<div><br></div><div>On the other hand, to guarantee master is never broken, test + merge must be a strictly serial operation (i.e do not even start new regression job until the previous patch is tested and merged). That is even worse, for sure.</div>
<div><br></div><div>In the end we probably need a combination of the two strategies</div><div><br></div><div>- Ability to test multiple patches at the same time (solves regression throughput to some extent and increases "integrated testing" of patches for their combined effect.</div>
<div><br></div><div>- Ability to run tests in parallel (of the patch sets) where testing patch sets can be formed such that the two groups are really independent and there is very less chance of their combined effect to result in a regression (e.g one patch set for a bunch of patches in glusterd and another patch set for a bunch of patches in data path).</div>
<div><br></div><div>Avati</div></div></div></div>