Обсуждение: Improving tracking/processing of buildfarm test failures

Hello hackers,

I'd like to discuss ways to improve the buildfarm experience for anyone who
are interested in using information which buildfarm gives to us.

Unless I'm missing something, as of now there are no means to determine
whether some concrete failure is known/investigated or fixed, how
frequently it occurs and so on... From my experience, it's not that
unbelievable that some failure occurred two years ago and lost in time was
an indication of e. g. a race condition still existing in the code/tests
and thus worth fixing. But without classifying/marking failures it's hard
to find such or other interesting failure among many others...

The first way to improve things I can imagine is to add two fields to the
buildfarm database: a link to the failure discussion (set when the failure
is investigated/reproduced and reported in -bugs or -hackers) and a commit
id/link (set when the failure is fixed). I understand that it requires
modifying the buildfarm code, and adding some UI to update these fields,
but it allows to add filters to see only unknown/non-investigated failures
in the buildfarm web interface later.

The second way is to create a wiki page, similar to "PostgreSQL 17 Open
Items", say, "Known buildfarm test failures" and fill it like below:
<url to failure1>
<url to failure2>
...
Useful info from the failure logs for reference
...
<link to -hackers thread>
---
This way is less invasive, but it would work well only if most of
interested people know of it/use it.
(I could start with the second approach, if you don't mind, and we'll see
how it works.)

Best regards,
Alexander

On Thu, May 23, 2024 at 4:30 PM Alexander Lakhin <exclusion@gmail.com> wrote:
>
> I'd like to discuss ways to improve the buildfarm experience for anyone who
> are interested in using information which buildfarm gives to us.
>
> Unless I'm missing something, as of now there are no means to determine
> whether some concrete failure is known/investigated or fixed, how
> frequently it occurs and so on... From my experience, it's not that
> unbelievable that some failure occurred two years ago and lost in time was
> an indication of e. g. a race condition still existing in the code/tests
> and thus worth fixing. But without classifying/marking failures it's hard
> to find such or other interesting failure among many others...
>
> The first way to improve things I can imagine is to add two fields to the
> buildfarm database: a link to the failure discussion (set when the failure
> is investigated/reproduced and reported in -bugs or -hackers) and a commit
> id/link (set when the failure is fixed). I understand that it requires
> modifying the buildfarm code, and adding some UI to update these fields,
> but it allows to add filters to see only unknown/non-investigated failures
> in the buildfarm web interface later.
>
> The second way is to create a wiki page, similar to "PostgreSQL 17 Open
> Items", say, "Known buildfarm test failures" and fill it like below:
> <url to failure1>
> <url to failure2>
> ...
> Useful info from the failure logs for reference
> ...
> <link to -hackers thread>
> ---
> This way is less invasive, but it would work well only if most of
> interested people know of it/use it.
> (I could start with the second approach, if you don't mind, and we'll see
> how it works.)
>

I feel it is a good idea to do something about this. It makes sense to
start with something simple and see how it works. I think this can
also help us whether we need to chase a particular BF failure
immediately after committing.

--
With Regards,
Amit Kapila.

On Thu, May 23, 2024 at 02:00:00PM +0300, Alexander Lakhin wrote:
> I'd like to discuss ways to improve the buildfarm experience for anyone who
> are interested in using information which buildfarm gives to us.
> 
> Unless I'm missing something, as of now there are no means to determine
> whether some concrete failure is known/investigated or fixed, how
> frequently it occurs and so on... From my experience, it's not that
> unbelievable that some failure occurred two years ago and lost in time was
> an indication of e. g. a race condition still existing in the code/tests
> and thus worth fixing. But without classifying/marking failures it's hard
> to find such or other interesting failure among many others...

I agree this is an area of difficulty consuming buildfarm results.  I have an
inefficient template for studying a failure, which your proposals would help:

**** grep recent -hackers for animal name
**** search the log for ~10 strings (e.g. "was terminated") to find the real indicator of where it failed
**** search mailing lists for that indicator
**** search buildfarm database for that indicator

> The first way to improve things I can imagine is to add two fields to the
> buildfarm database: a link to the failure discussion (set when the failure
> is investigated/reproduced and reported in -bugs or -hackers) and a commit
> id/link (set when the failure is fixed). I understand that it requires

I bet the hard part is getting data submissions, so I'd err on the side of
making this as easy as possible for submitters.  For example, accept free-form
text for quick notes, not only URLs and commit IDs.

> modifying the buildfarm code, and adding some UI to update these fields,
> but it allows to add filters to see only unknown/non-investigated failures
> in the buildfarm web interface later.
> 
> The second way is to create a wiki page, similar to "PostgreSQL 17 Open
> Items", say, "Known buildfarm test failures" and fill it like below:
> <url to failure1>
> <url to failure2>
> ...
> Useful info from the failure logs for reference
> ...
> <link to -hackers thread>
> ---
> This way is less invasive, but it would work well only if most of
> interested people know of it/use it.
> (I could start with the second approach, if you don't mind, and we'll see
> how it works.)

Certainly you doing (2) can only help, though it may help less than (1).


I recommend considering what the buildfarm server could discover and publish
on its own.  Examples:

- N members failed at the same step, in a related commit range.  Those members
  are now mostly green.  Defect probably got fixed quickly.

- Log contains the following lines that are highly correlated with failure.
  The following other reports, if any, also contained them.

Hello Amit and Noah,

24.05.2024 14:15, Amit Kapila wrote:
> I feel it is a good idea to do something about this. It makes sense to
> start with something simple and see how it works. I think this can
> also help us whether we need to chase a particular BF failure
> immediately after committing.

24.05.2024 23:00, Noah Misch wrote:

>
>> (I could start with the second approach, if you don't mind, and we'll see
>> how it works.)
> Certainly you doing (2) can only help, though it may help less than (1).

Thank you for paying attention to this!

I've created such page to accumulate information on test failures:
https://wiki.postgresql.org/wiki/Known_Buildfarm_Test_Failures

I've deliberately added a trivial issue with partition_split, which is
doomed to be fixed soon, to test the information workflow, and I'm going
to add a few other items in the coming days.

Please share your comments and suggestions, if any.

Best regards,
Alexander

On 2024-05-24 Fr 16:00, Noah Misch wrote:
> On Thu, May 23, 2024 at 02:00:00PM +0300, Alexander Lakhin wrote:
>> I'd like to discuss ways to improve the buildfarm experience for anyone who
>> are interested in using information which buildfarm gives to us.
>>
>> Unless I'm missing something, as of now there are no means to determine
>> whether some concrete failure is known/investigated or fixed, how
>> frequently it occurs and so on... From my experience, it's not that
>> unbelievable that some failure occurred two years ago and lost in time was
>> an indication of e. g. a race condition still existing in the code/tests
>> and thus worth fixing. But without classifying/marking failures it's hard
>> to find such or other interesting failure among many others...
> I agree this is an area of difficulty consuming buildfarm results.  I have an
> inefficient template for studying a failure, which your proposals would help:
>
> **** grep recent -hackers for animal name
> **** search the log for ~10 strings (e.g. "was terminated") to find the real indicator of where it failed
> **** search mailing lists for that indicator
> **** search buildfarm database for that indicator
>
>> The first way to improve things I can imagine is to add two fields to the
>> buildfarm database: a link to the failure discussion (set when the failure
>> is investigated/reproduced and reported in -bugs or -hackers) and a commit
>> id/link (set when the failure is fixed). I understand that it requires
> I bet the hard part is getting data submissions, so I'd err on the side of
> making this as easy as possible for submitters.  For example, accept free-form
> text for quick notes, not only URLs and commit IDs.
>
>> modifying the buildfarm code, and adding some UI to update these fields,
>> but it allows to add filters to see only unknown/non-investigated failures
>> in the buildfarm web interface later.
>>
>> The second way is to create a wiki page, similar to "PostgreSQL 17 Open
>> Items", say, "Known buildfarm test failures" and fill it like below:
>> <url to failure1>
>> <url to failure2>
>> ...
>> Useful info from the failure logs for reference
>> ...
>> <link to -hackers thread>
>> ---
>> This way is less invasive, but it would work well only if most of
>> interested people know of it/use it.
>> (I could start with the second approach, if you don't mind, and we'll see
>> how it works.)
> Certainly you doing (2) can only help, though it may help less than (1).
>
>
> I recommend considering what the buildfarm server could discover and publish
> on its own.  Examples:
>
> - N members failed at the same step, in a related commit range.  Those members
>    are now mostly green.  Defect probably got fixed quickly.
>
> - Log contains the following lines that are highly correlated with failure.
>    The following other reports, if any, also contained them.
>
>


I'm prepared to help, but also bear in mind that currently the only 
people who can submit notes are animal owners who can attach notes to 
their own animals. I'm not keen to allow general public submission of 
notes to the database. We already get lots of spam requests that we turn 
away.

If you have queries that you want canned we can look at that. Ditto 
extra database fields. Currently we don't have any processing that 
correlates different failures, but that's not inconceivable.


cheers


andrew

--
Andrew Dunstan
EDB: https://www.enterprisedb.com