Re: Sending out a request for more buildfarm animals?

On 05/08/2014 12:21 PM, Tomas Vondra wrote:
> On 6.5.2014 23:01, Tomas Vondra wrote:
>> On 6.5.2014 22:24, Tom Lane wrote:
>>> Tomas Vondra <tv@fuzzy.cz> writes:
>>>> I recall there was a call for more animals with CLOBBER_CACHE_ALWAYS
>>>> some time ago, so I went and enabled that on all three animals. Let's
>>>> see how long that will take.
>>>> I see there are more 'clobber' options in the code: CLOBBER_FREED_MEMORY
>>>> and CLOBBER_CACHE_RECURSIVELY. Would that be a good idea to enable these
>>>> as well?
>>>> The time requirements will be much higher (especially for the
>>>> RECURSIVELY option), but running that once a week shouldn't be a big
>>>> deal - the machine is pretty much dedicated to the buildfarm.
>>> I've never had the patience to run the regression tests to completion
>>> with CLOBBER_CACHE_RECURSIVELY at all, let alone do it on a regular
>>> basis. (I wonder if there's some easy way to run it for just a few
>>> regression tests...)
>> Now, that's a challenge ;-)
>>
>>> I think testing CLOBBER_FREED_MEMORY would be sensible though.
>> OK, I've enabled this for now.
> Hmmmm, with CLOBBER_CACHE_ALWAYS + CLOBBER_FREED_MEMORY the tests take
> ~20h on a single branch/animal. With a single locale (e.g. "C") it would
> take ~4h, but we're testing a bunch of additional czech/slovak locales.
>
> The tests are running in sequence (magpie->treepie->fulmar) so with all
> 6 branches, this would take ~14 days to complete. I don't mind the
> machine is running tests 100% of the time, that's why it's in buildfarm,
> but I'd rather see the failures soon after the commit (and two weeks is
> well over the "soon" edge, IMHO).
>
> So I'm thinking about how to improve this. I'd like to keep the options
> for all the branches (e.g. not just HEAD, as a few other animals do).
> But I'm thinking about running the tests in parallel, somehow - the
> machine has 4 cores, and most of the time only one of them is used. I
> don't expect a perfect ~3x speedup, but getting ~2x would be nice.
>
> Any recommendations how to do that? I see there's 'base_port' in the
> config - is it enough to tweak this, or do I need to run separate the
> animals using e.g. lxc?


Here is what I do on my FreeBSD VM. I have 2 animals, nightjar and 
friarbird. They have the same buildroot. friarbird is set up to build 
with CLOBBER_CACHE_ALWAYS, building just HEAD and just testing C locale; 
nightjar builds all branches we are interested in and tests locale 
cs_CZ.utf8 in addition to C.

Other than those differences they are pretty similar.

Here is the crontab that drives them:
   27 5-22 * * * cd bf && ./run_branches.pl --run-all --verbose   --config=nightjarx.conf >> bf.out 2>&1   20 0 * * *
cdbf && ./run_branches.pl --run-all --verbose   --config=friarbird.conf --skip-steps=install-check >> bf.out 2>&1
 


The buildfarm code has enough locking smarts to make sure we don't get 
any build collisions doing this.

If you have an animal to do a special type of build (e.g. CLOBBER_foo) 
then it's probably a good idea to set a note for that animal - see the 
buildfarm program setnotes.pl. friarbird has the note set "Uses 
-DCLOBBER_CACHE_ALWAYS".


If you want to do this in parallel, then you will need different 
buildroots and different base ports for each animal. I would not run the 
same animal on different branches concurrently, that is quite likely to 
end up in port collisions.


HTH.

cheers

andrew