Re: Proposal: remove obsolete hot-standby testing infrastructure

04.01.2022 18:33, Tom Lane wrote:
> Alexander Lakhin <exclusion@gmail.com> writes:
>> It's hardly that important, but we (Postgres Pro) run this test
>> regularly to check for primary-standby compatibility. It's useful when
>> checking binary packages from different minor versions. For example, we
>> setup postgresql-14.0 and postgresql-14.1 aside (renaming one
>> installation' directory and changing it's port) and perform the test.
>> What've found with it was e.g. incompatibility due to linkage of
>> different libicu versions (that was PgPro-only issue). I don't remember
>> whether we found something related to PostgreSQL itself, but we
>> definitely use this test and I'm not sure how to replace it in our setup
>> with a TAP test. On the other hand, testing binaries is not accustomed
>> in the community yet, so when such testing will be adopted, probably a
>> brand new set of tests should emerge.
> Oh, interesting.  I definitely concur that testing compatibility of
> different builds or minor versions is an important use-case.  And
> I concede that making src/test/recovery do it would be tricky and
> a bit out-of-scope.  But having said that, the hs_standby_* scripts
> seem like a poor fit for the job too.  AFAICS they don't really
> test any user data type except integer (so I'm surprised that they
> located an ICU incompatibility for you); and they spend a lot of
> effort on stuff that I doubt is relevant because it *is* covered
> by the TAP tests.
An ICU incompatibility was detected due to our invention [1] "default
collation" that is checked upon connection (before any query processing):
--- C:/tmp/.../src/test/regress/expected/hs_standby_check.out   
2021-10-14 04:07:38.000000000 +0200
+++ C:/tmp/.../src/test/regress/results/hs_standby_check.out   
2021-10-14 06:06:12.004043500 +0200
@@ -1,3 +1,6 @@
+WARNING:  collation "default" has version mismatch

+DETAIL:  The collation in the database was created using version
153.64, but the operating system provides version 153.14.

+HINT:  Check all objects affected by this collation and run ALTER
COLLATION pg_catalog."default" REFRESH VERSION

 --
 -- Hot Standby tests
 --

I admit that we decided to use this test mainly because it exists and
described in the documentation, not because it seemed very useful. It's
usage increased test coverage without a doubt, as it requires a rather
non-trivial setup (similar setups performed by TAP tests, but not with
pre-packaged binaries).
> If I were trying to test that topic using available spare parts,
> what I'd do is run the regular regression tests on the primary
> and see if the standby could track it.  Maybe pg_dump from both
> servers afterwards and see if the results match, a la the pg_upgrade
> test.  Bonus points for a script that could run some other pg_regress
> suite such as one of the contrib modules, because then you could
> check compatibility of those too.
Thanks for the idea! We certainly will implement something like that
when we start testing packages for v15. We've already learned to compare
dumps before/after minor upgrade, so we could reuse that logic for this
test too.
> I'm happy to keep the hs_standby_* scripts if there's a live use-case
> for them; but I don't see what they're doing for you that wouldn't be
> done better by other pg_regress suites.
Yes, I will not miss the test in case you will remove it. I just wanted
to mention that we use(d) it in our testing more or less successfully.

[1]
https://www.postgresql.org/message-id/37A534BE-CBF7-467C-B096-0AAD25091A9F%40yandex-team.ru

Best regards,
Alexander