[ANNOUNCE] == PostgreSQL Weekly News - September 24 2017 ==

== PostgreSQL Weekly News - September 24 2017 ==

PostgreSQL 10 Release Candidate 1 released.  Test!
https://www.postgresql.org/about/news/1783/

== PostgreSQL Product News ==

pg-wrapper 0.2.2, a wrapper for PHP's pgsql extension, released.
https://github.com/sad-spirit/pg-wrapper

brimbox 2.3.4, a web database interface and framework built atop PostgreSQL, released.
https://www.brimbox.com/brimbox-version-2-3-4-released/

pg_dirtyread 1.2, an extension that provides the ability to read dead but
unvacuumed rows from a relation, released.
https://github.com/ChristophBerg/pg_dirtyread

pg_builder 0.2.1, a PHP query builder for PostgreSQL, released.
https://github.com/sad-spirit/pg-builder

== PostgreSQL Jobs for September ==

http://archives.postgresql.org/pgsql-jobs/2017-09/

== PostgreSQL Local ==

PGDay.IT 2017 will take place October 13th, in Milan, Italy.
http://pgday.it

PostgreSQL Conference Europe 2017 will be held on October 24-27, 2017 in the
Warsaw Marriott Hotel, in Warsaw, Poland.
https://2017.pgconf.eu/

PGConf Local: Seattle will be held November 13 - 14, 2017.
https://www.pgconf.us/#Seattle2017

PGDay Australia 2017 will be held on November 17 in Melbourne.
http://2017.pgday.com.au/

2ndQuadrant PostgreSQL Conference 2017 (2Q PGConf, for short) will be hosted on
November 6th & 7th in New York City, and November 9th in Chicago.
http://www.2qpgconf.com/

PostgreSQL Session November 17th, 2017, in Paris, France.  The CfP is open until
September 30, 2017 at call-for-paper AT postgresql-sessions DOT org.  Details at
http://blog.dalibo.com/2017/08/29/cfp_pgsession9.html

PGConf Local: Austin will be held December 4 - 5, 2017. Call for Papers is
now open at https://www.pgconf.us/conferences/Austin2017

PGConf.ASIA 2017 will take place on December 4-6 2017 in Akihabara, Tokyo,
Japan.
http://www.pgconf.asia/EN/2017/

PGConf India 2018 will be on February 22-23, 2018 in Bengaluru, Karnataka.
Proposals are due via https://goo.gl/forms/F9hRjOIsaNasVOAz2 by October 31st, 2017.
http://pgconf.in/

== PostgreSQL in the News ==

Planet PostgreSQL: http://planet.postgresql.org/

PostgreSQL Weekly News is brought to you this week by David Fetter

Submit news and announcements by Sunday at 3:00pm EST5EDT.  Please send English
language ones to david@fetter.org, German language to pwn@pgug.de, Italian
language to pwn@itpug.org.

== Applied Patches ==

Peter Eisentraut pushed:

- Remove dead external links from documentation.
https://git.postgresql.org/pg/commitdiff/d31892e2105cf48d8430807d74d5fdf1434af541

- Update some dead external links in the documentation.
https://git.postgresql.org/pg/commitdiff/4b17c894293d0c3ed944da76aeb9bc2bb02a6db6

- Sync process names between ps and pg_stat_activity.  Remove gratuitous differences in the process names shown in
pg_stat_activity.backend_typeand the ps output.  Reviewed-by: Takayuki Tsunakawa <tsunakawa.takay@jp.fujitsu.com>
https://git.postgresql.org/pg/commitdiff/be87b70b6117609b7db0cd5e0cd96e7f569bdddb

- Fix compiler warning.  from gcc-7 -Wformat-truncation (via -Wall)
https://git.postgresql.org/pg/commitdiff/d42294fc00da4b97d04ddb4401b76295e8d86816

- Add basic TAP test setup for pg_upgrade.  The plan is to convert the current pg_upgrade test to the TAP framework.
Thiscommit just puts a basic TAP test in place so that we can see how the build farm behaves, since the build farm
clienthas some special knowledge of the pg_upgrade tests.  Author: Michael Paquier <michael.paquier@gmail.com>
https://git.postgresql.org/pg/commitdiff/f41e56c76e39f02bef7ba002c9de03d62b76de4d

- Fix build with !USE_WIDE_UPPER_LOWER.  The placement of the ifdef blocks in formatting.c was pretty bogus, so the
codefailed to compile if USE_WIDE_UPPER_LOWER was not defined.  Reported-by: Peter Geoghegan <pg@bowt.ie> Reported-by:
NoahMisch <noah@leadboat.com> https://git.postgresql.org/pg/commitdiff/e6023ee7fa73a2d9a2d7524f63584844b2291def
 

- Remove contrib/chkpass.  The recent addition of a test suite for this module revealed a few problems.  It uses a
crypt()method that is no longer considered secure and doesn't work anymore on some platforms.  Using a volatile input
functionviolates internal sanity check assumptions and leads to failures on the build farm.  So this module is neither
ausable security tool nor a good example for an extension.  No one wanted to argue for keeping or improving it, so
removeit.  Discussion: https://www.postgresql.org/message-id/5645b0d7-cc40-6ab5-c553-292a91091ee7%402ndquadrant.com
https://git.postgresql.org/pg/commitdiff/5d3cad564729f64d972c5c803ff34f0eb40bfd0c

- doc: Document commands that cannot be run in a transaction block.  Mainly covering the new CREATE SUBSCRIPTION and
DROPSUBSCRIPTION, but ALTER DATABASE SET TABLESPACE was also missing.
https://git.postgresql.org/pg/commitdiff/91ad8b416cee753eaa6f520ee2d21c2d41853381

- Revert "Add basic TAP test setup for pg_upgrade".  This reverts commit f41e56c76e39f02bef7ba002c9de03d62b76de4d.  The
buildfarm client would run the pg_upgrade tests twice, once as part of the existing pg_upgrade check run and once as
partof picking up all TAP tests by looking for "t" directories. Since the pg_upgrade tests are pretty slow, we will
needa better solution or possibly a build farm client change before we can proceed with this.
https://git.postgresql.org/pg/commitdiff/58ffe141eb37c3f027acd25c1fc6b36513bf9380

- Fix saving and restoring umask.  In two cases, we set a different umask for some piece of code and restore it
afterwards. But if the contained code errors out, the umask is not restored.  So add TRY/CATCH blocks to fix that.
https://git.postgresql.org/pg/commitdiff/aa6b7b72d9bcf967cbccd378de4bc5cef33d02f9

- Refactor new file permission handling.  The file handling functions from fd.c were called with a diverse mix of
notationsfor the file permissions when they were opening new files.  Almost all files created by the server should have
thesame permissions set.  So change the API so that e.g. OpenTransientFile() automatically uses the standard
permissionsset, and OpenTransientFilePerm() is a new function that takes an explicit permissions set for the few cases
whereit is needed.  This also saves an unnecessary argument for call sites that are just opening an existing file.
Whilewe're reviewing these APIs, get rid of the FileName typedef and use the standard const char * for the file name
andmode_t for the file mode.  This makes these functions match other file handling functions and removes an unnecessary
layerof mysteriousness. We can also get rid of a few casts that way.  Author: David Steele <david@pgmasters.net>
https://git.postgresql.org/pg/commitdiff/0c5803b450e0cc29b3527df3f352e6f18a038cc6

- Fix pg_basebackup test to original intent.  One test case was meant to check that pg_basebackup does not succeed when
aslot is specified with -S but WAL streaming is not selected, which used to require specifying -X stream.  Since -X
streamis the default in PostgreSQL 10, this test case no longer covers that meaning, but the pg_basebackup invocation
happenedto fail anyway for the unrelated reason that the specified replication slot does not exist.  To fix, move the
testcase to later in the file where the slot does exist, and add -X none to the invocation so that it covers the
originallymeant behavior. extracted from a patch by Michael Banck <michael.banck@credativ.de>
https://git.postgresql.org/pg/commitdiff/74ca8f9b9077017529fe658e445a11da296ac6ab

- doc: Expand user documentation on SCRAM.  Explain more about how the different password authentication methods and
thepassword_encryption settings relate to each other, give some upgrading advice, and set a better link from the
releasenotes.  Reviewed-by: Jeff Janes <jeff.janes@gmail.com>
https://git.postgresql.org/pg/commitdiff/9b31c72a9492880e657b68b1ed971dec3c361c95

- Allow ICU to use SortSupport on Windows with UTF-8.  There is no reason to ever prevent the use of SortSupport on
Windowswhen ICU locales are used.  We previously avoided SortSupport on Windows with UTF-8 server encoding and a non
C-localedue to restrictions in Windows' libc functionality.  This is now considered to be a restriction in one
platform'slibc collation provider, and not a more general platform restriction.  Reported-by: Peter Geoghegan
<pg@bowt.ie>https://git.postgresql.org/pg/commitdiff/6dda0998afc7d449145b9ba216844bdba7a817d6
 

Tom Lane pushed:

- Fix, or at least ameliorate, bugs in logicalrep_worker_launch().  If we failed to get a background worker slot, the
codejust walked away from the logicalrep-worker slot it already had, leaving that looking like the worker is still
startingup.  This led to an indefinite hang in subscription startup, as reported by Thomas Munro.  We must release the
sloton failure.  Also fix a thinko: we must capture the worker slot's generation before releasing LogicalRepWorkerLock
thefirst time, else testing to see if it's changed is pretty meaningless.  BTW, the CHECK_FOR_INTERRUPTS() in
WaitForReplicationWorkerAttachis a ticking time bomb, even without considering the possibility of elog(ERROR) in one of
theother functions it calls.  Really, this entire business needs a redesign with some actual thought about error
recovery. But for now I'm just band-aiding the case observed in testing.  Back-patch to v10 where this code was added.
Discussion:https://postgr.es/m/CAEepm=2bP3TBMFBArP6o20AZaRduWjMnjCjt22hSdnA-EvrtCw@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/3e1683d37e1d751eb2df9a5cb0507bebc6cf7d05

- Make DatumGetFoo/PG_GETARG_FOO/PG_RETURN_FOO macro names more consistent.  By project convention, these names should
include"P" when dealing with a pointer type; that is, if the result of a GETARG macro is of type FOO *, it should be
calledPG_GETARG_FOO_P not just PG_GETARG_FOO.  Some newer types such as JSONB and ranges had not followed the
convention,and a number of contrib modules hadn't gotten that memo either.  Rename the offending macros to improve
consistency. In passing, fix a few places that thought PG_DETOAST_DATUM() returns a Datum; it does not, it returns
"structvarlena *".  Applying DatumGetPointer to that happens not to cause any bad effects today, but it's formally
wrong. Also, adjust an ltree macro that was designed without any thought for what pgindent would do with it.  This is
allcosmetic and shouldn't have any impact on generated code.  Mark Dilger, some further tweaks by me Discussion:
https://postgr.es/m/EA5676F4-766F-4F38-8348-ECC7DB427C6A@gmail.com
https://git.postgresql.org/pg/commitdiff/4bd1994650fddf49e717e35f1930d62208845974

- Make ExplainOpenGroup and ExplainCloseGroup public.  Extensions with custom plan nodes might like to use these in
theirEXPLAIN output.  Hadi Moshayedi Discussion:
https://postgr.es/m/CA+_kT_dU-rHCN0u6pjA6bN5CZniMfD=-wVqPY4QLrKUY_uJq5w@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/66917bfaa7bb0b6bae52a5fe631a8b6443203f55

- Minor code-cleanliness improvements for btree.  Make the btree page-flags test macros (P_ISLEAF and friends) return
cleanboolean values, rather than values that might not fit in a bool.  Use them in a few places that were randomly
referencingthe flag bits directly.  In passing, change access/nbtree/'s only direct use of BUFFER_LOCK_SHARE to
BT_READ. (Some think we should go the other way, but as long as we have BT_READ/BT_WRITE, let's use them consistently.)
MasahikoSawada, reviewed by Doug Doole Discussion:
https://postgr.es/m/CAD21AoBmWPeN=WBB5Jvyz_Nt3rmW1ebUyAnk3ZbJP3RMXALJog@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/eb5c404b17752ca566947f12cb702438dcccdcb1

- Cache datatype-output-function lookup info across calls of concat().  Testing indicates this can save a third to a
halfof the runtime of the function. Pavel Stehule, reviewed by Alexander Kuzmenkov Discussion:
https://postgr.es/m/CAFj8pRAT62pRgjoHbgTfJUc2uLmeQ4saUj+yVJAEZUiMwNCmdg@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/ed22fb8b0091deea23747310fa7609079a96cf82

- Remove no-op GiST support functions in the core GiST opclasses.  The preceding patch allowed us to remove useless
GiSTsupport functions.  This patch actually does that for all the no-op cases in the core GiST code.  This buys us
whateverperformance gain is to be had, and more importantly exercises the preceding patch.  There remain no-op
functionsin the contrib GiST opclasses, but those will take more work to remove.  Discussion:
https://postgr.es/m/CAJEAwVELVx9gYscpE=Be6iJxvdW5unZ_LkcAaVNSeOwvdwtD=A@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/2d484f9b058035d41204f2eb8a0a8d2e8ee57b44

- Allow no-op GiST support functions to be omitted.  There are common use-cases in which the compress and/or decompress
functionscan be omitted, with the result being that we make no data transformation when storing or retrieving index
values. Previously, you had to provide a no-op function anyway, but this patch allows such opclass support functions to
beomitted.  Furthermore, if the compress function is omitted, then the core code knows that the stored representation
isthe same as the original data.  This means we can allow index-only scans without requiring a fetch function to be
providedeither. Previously you had to provide a no-op fetch function if you wanted IOS to work.  This reportedly
providesa small performance benefit in such cases, but IMO the real reason for doing it is just to reduce the amount of
uselessboilerplate code that has to be written for GiST opclasses.  Andrey Borodin, reviewed by Dmitriy Sarafannikov
Discussion:https://postgr.es/m/CAJEAwVELVx9gYscpE=Be6iJxvdW5unZ_LkcAaVNSeOwvdwtD=A@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/d3a4f89d8a3e500bd7c0b7a8a8a5ce1b47859128

- Fix erroneous documentation about noise word GROUP.  GRANT, REVOKE, and some allied commands allow the noise word
GROUPbefore a role name (cf. grantee production in gram.y).  This option does not exist elsewhere, but it had
nonethelesssnuck into the documentation for ALTER ROLE, ALTER USER, and CREATE SCHEMA.  Seems to be a copy-and-pasteo
incommit 31eae6028, which did expand the syntax choices here, but not in that way.  Back-patch to 9.5 where that came
in. Discussion: https://postgr.es/m/20170916123750.8885.66941@wrigleys.postgresql.org
https://git.postgresql.org/pg/commitdiff/36b564c648a044e42ca461466ae14d8588e6c5e2

- Fix instability in subscription regression test.  005_encoding.pl neglected to wait for the subscriber's initial
synchronizationto happen.  While we have not seen this fail in the buildfarm, it's pretty easy to demonstrate there's
anissue by hacking logicalrep_worker_launch() to fail most of the time. Michael Paquier Discussion:
https://postgr.es/m/27032.1505749806@sss.pgh.pa.us
https://git.postgresql.org/pg/commitdiff/4939488af9b86edfff9b981773cd388d361c5830

- Improve dubious memory management in pg_newlocale_from_collation(). pg_newlocale_from_collation() used malloc() and
strdup()directly, which is generally not per backend coding style, and it didn't bother to check for failure results,
butwould just SIGSEGV instead.  Also, if one of the numerous error checks in the middle of the function failed, the
already-allocatedmemory would be leaked permanently.  Admittedly, it's not a lot of memory, but it could build up if
thisfunction were called repeatedly for a bad collation. The first two problems are easily cured by palloc'ing in
TopMemoryContextinstead of calling libc directly.  We can fairly easily dodge the leakage problem for the struct
pg_locale_structby filling in a temporary variable and allocating permanent storage only once we reach the bottom of
thefunction. It's harder to get rid of the potential leakage for ICU's copy of the collcollate string, but at least
that'sonly allocated after most of the error checks; so live with that aspect.  Back-patch to v10 where this code came
in,with one or another of the ICU patches.
https://git.postgresql.org/pg/commitdiff/7b86c2ac9563ffd9b870cfd73a769431b7922e81

- Give a better error for duplicate entries in VACUUM/ANALYZE column list. Previously, the code didn't think about this
caseand would just try to analyze such a column twice.  That would fail at the point of inserting the second version of
thepg_statistic row, with obscure error messsages like "duplicate key value violates unique constraint" or "tuple
alreadyupdated by self", depending on context and PG version.  We could allow the case by ignoring duplicate column
specifications,but it seems better to reject it explicitly.  The bogus error messages seem like arguably a bug, so
back-patchto all supported versions.  Nathan Bossart, per a report from Michael Paquier, and whacked around a bit by
me. Discussion: https://postgr.es/m/E061A8E3-5E3D-494D-94F0-E8A9B312BBFC@amazon.com
https://git.postgresql.org/pg/commitdiff/71480501057fee9fa3649b072173ff10e2b842d0

- Revert "Fix bool/int type confusion".  This reverts commit 0ec2e908babfbfde83a3925680f06b16408739ff.  We'll use the
upstream(IANA) fix instead. https://git.postgresql.org/pg/commitdiff/a890432a872afc9ca2327573f3313fd994d17384
 

- Sync our copy of the timezone library with IANA tzcode master.  This patch absorbs a few unreleased fixes in the IANA
code. It corresponds to commit 2d8b944c1cec0808ac4f7a9ee1a463c28f9cd00a in https://github.com/eggert/tz. Non-cosmetic
changesinclude: TZDEFRULESTRING is updated to match current US DST practice, rather than what it was over ten years
ago. This only matters for interpretation of POSIX-style zone names (e.g., "EST5EDT"), and only if the timezone
databasedoesn't include either an exact match for the zone name or a "posixrules" entry.  The latter should not be true
inany current Postgres installation, but this could possibly matter when using --with-system-tzdata.  Get rid of a
nonportableuse of "++var" on a bool var. This is part of a larger fix that eliminates some vestigial support for
consecutiveleap seconds, and adds checks to the "zic" compiler that the data files do not specify that.  Remove a
coupleof ancient compatibility hacks. The IANA crew think these are obsolete, and I tend to agree.  But perhaps our
buildfarmwill think different.  Back-patch to all supported branches, in line with our policy that all branches should
beusing current IANA code.  Before v10, this includes application of current pgindent rules, to avoid whitespace
problemsin future back-patches.  Discussion: https://postgr.es/m/E1dsWhf-0000pT-F9@gemulon.postgresql.org
https://git.postgresql.org/pg/commitdiff/47f849a3c9005852926dca551d70ad8111f09f3a

- Assume wcstombs(), towlower(), and sibling functions are always present. These functions are required by SUS v2,
whichis our minimum baseline for Unix platforms, and are present on all interesting Windows versions as well.  Even our
oldestbuildfarm members have them.  Thus, we were not testing the "!USE_WIDE_UPPER_LOWER" code paths, which explains
whythe bug fixed in commit e6023ee7f escaped detection.  Per discussion, there seems to be no more real-world value in
maintainingthis option.  Hence, remove the configure-time tests for wcstombs() and towlower(), remove the
USE_WIDE_UPPER_LOWERsymbol, and remove all the !USE_WIDE_UPPER_LOWER code.  There's not actually all that much of the
latter,but simplifying the #if nests is a win in itself. Discussion:
https://postgr.es/m/20170921052928.GA188913@rfd.leadboat.com
https://git.postgresql.org/pg/commitdiff/85feb77aa09cda9ff3e12cf95c757c499dc25343

- Mop-up for commit 85feb77aa09cda9ff3e12cf95c757c499dc25343.  Adjust commentary in regc_pg_locale.c to remove mention
ofthe possibility of not having <wctype.h> functions, since we no longer consider that.  Eliminate duplicate code in
wparser_def.cby generalizing the p_iswhat macro to take a parameter saying what to return for non-ASCII chars in C
locale. (That's not really a consequence of the USE_WIDE_UPPER_LOWER-ectomy, but I noticed it while doing that.)
https://git.postgresql.org/pg/commitdiff/ed87e1980706975e7aa412bee200087774c5ff22

- Allow up to 3 "-P 1" reports per thread in pgbench run of 2 seconds.  There seems to be some considerable imprecision
inthe timing of -P progress reports.  Nominally each thread ought to produce 2 reports in this test, but about 10% of
thetime we only get one, and 1% of the time we get three, as per buildfarm results so far.  Pending further
investigation,treat the last case as a "pass".  (I, tgl, am suspicious that this still might not be lax enough, now
thatit's obvious that the behavior is load-dependent; but there's not yet buildfarm evidence to confirm that
suspicion.)Fabien Coelho Discussion: https://postgr.es/m/26654.1505232433@sss.pgh.pa.us
https://git.postgresql.org/pg/commitdiff/0f574a7afb5c998d19dc3d981e45cb10267286ed

- Ten-second timeout in 013_crash_restart.pl is not enough, let's try 60.  Per buildfarm member topminnow.
https://git.postgresql.org/pg/commitdiff/01c7d3ef85d4b0e1c52cc1a3542864f95f386f76

- Improve memory management in autovacuum.c.  Invoke vacuum(), as well as "work item" processing, in the PortalContext
thatdo_autovacuum() has manufactured, which will be reset before each such invocation.  This ensures cleanup of any
memoryleaked by these operations.  It also avoids the rather dangerous practice of calling vacuum() in a context that
vacuum()itself will destroy while it runs.  There's no known live bug there, but it's not hard to imagine introducing
oneif we leave it like this.  Tom Lane, reviewed by Michael Paquier and Alvaro Herrera Discussion:
https://postgr.es/m/13849.1506114543@sss.pgh.pa.us
https://git.postgresql.org/pg/commitdiff/335f3d04e4c8dd495c4dd30ab1049b6fe8f25052

- Fix bogus size calculation in strlist_to_textarray().  It's making an array of Datum, not an array of text *.  The
mistakeis harmless since those are currently the same size, but it's still wrong.
https://git.postgresql.org/pg/commitdiff/737639017c87d5a0a466e8676f1eadc61d775c78

- ... and the very same bug in publicationListToArray().  Sigh.
https://git.postgresql.org/pg/commitdiff/24541ffd788d56009126fff52b2341ada6c84245

- Remove pgbench "progress" test pending solution of its timing issues. Buildfarm member skink shows that this is even
moreflaky than I thought. There are probably some actual pgbench bugs here as well as a timing dependency.  But we
can'thave stuff this unstable in the buildfarm, it obscures other issues.
https://git.postgresql.org/pg/commitdiff/ad51c6fb5708342e603d12a730bbc4e663bd637e

- Fix assorted infelicities in new SetWALSegSize() function.  * Failure to check for malloc failure (ok, pretty
unlikelyhere, but that's not an excuse).  * Leakage of open fd on read error, and of malloc'd buffer always.  *
Incorrectassumption that a short read would set errno to zero.  * Failure to adhere to message style conventions (in
particular,not reporting errno where relevant; using "couldn't open" rather than "could not open" is not really in line
withproject style either).  * Missing newlines on some messages.  Coverity spotted the leak problems; I noticed the
restwhile fixing the leaks. https://git.postgresql.org/pg/commitdiff/8485a25a8c9a419ff3e0d30e43e4abd5e680cc65
 

Andres Freund pushed:

- Fix crash restart bug introduced in 8356753c212.  The bug was caused by not re-reading the control file during crash
recoveryrestarts, which lead to an attempt to pfree() shared memory contents. The fix is to re-read the control file,
whichseems good anyway.  It's unclear as of this moment, whether we want to keep the refactoring introduced in the
commitreferenced above, or come up with an alternative approach. But fixing the bug in the mean time seems like a good
idearegardless.  A followup commit will introduce regression test coverage for crash restarts.  Reported-By: Tom Lane
Discussion:https://postgr.es/m/14134.1505572349@sss.pgh.pa.us
https://git.postgresql.org/pg/commitdiff/ec9e05b3c392ba9587f283507459737684539574

- Fix uninitialized variable in dshash.c.  A bugfix for commit 8c0d7bafad36434cb08ac2c78e69ae72c194ca20.  The code
wouldhave crashed if hashtable->size_log2 ever had the same value as hashtable->control->size_log2 by coincidence.  Per
Valgrind. Author: Thomas Munro Reported-By: Tomas Vondra Discussion:
https://postgr.es/m/e72fb33c-4f31-f276-e972-263d9b59554d%402ndquadrant.com
https://git.postgresql.org/pg/commitdiff/0fb9e4ace5ce4d479d839a720f32b99fdc87f455

- Rearm statement_timeout after each executed query.  Previously statement_timeout, in the extended protocol, affected
allmessages till a Sync message.  For clients that pipeline/batch query execution that's problematic. Instead disable
timeoutafter each Execute message, and enable, if necessary, the timer in start_xact_command(). As that's done only for
Executeand not Parse / Bind, pipelining the latter two could still cause undesirable timeouts. But a survey of protocol
implementationsshows that all drivers issue Sync messages when preparing, and adding timeout rearming to both is fairly
expensivefor the common parse / bind / execute sequence.  Author: Tatsuo Ishii, editorialized by Andres Freund
Reviewed-By:Takayuki Tsunakawa, Andres Freund Discussion:
https://postgr.es/m/20170222.115044.1665674502985097185.t-ishii@sraoss.co.jp
https://git.postgresql.org/pg/commitdiff/f8e5f156b30efee5d0038b03e38735773abcb7ed

- Add test for postmaster crash restarts.  Given that I managed to break this... We probably should extend the tests to
alsocover other sub-processes dying, but that's something for later.  Author: Andres Freund Discussion:
https://postgr.es/m/20170917080752.rcmihzfmgbeuqjk2@alap3.anarazel.de
https://git.postgresql.org/pg/commitdiff/a1924a4ea29399111e5155532ca24c9c51d3c82d

- Speedup pgstat_report_activity by moving mb-aware truncation to read side. Previously multi-byte aware truncation was
doneon every pgstat_report_activity() call - proving to be a bottleneck for workloads with long query strings that
executequickly.  Instead move the truncation to the read side, which commonly is executed far less frequently. That's
possiblebecause all server encodings allow to determine the length of a multi-byte string from the first byte.  Rename
PgBackendStatus.st_activityto st_activity_raw so existing extension users of the field break - their code has to be
adjustedto use pgstat_clip_activity().  Author: Andres Freund Tested-By: Khuntal Ghosh Reviewed-By: Robert Haas, Tom
LaneDiscussion: https://postgr.es/m/20170912071948.pa7igbpkkkviecpz@alap3.anarazel.de
https://git.postgresql.org/pg/commitdiff/54b6cd589ac2f5635a42511236a5eb7299e2dcaf

- Avoid use of non-portable strnlen() in pgstat_clip_activity().  The use of strnlen rather than strlen was just
paranoia.Instead of giving up on the paranoia, just implement the safeguard differently. And add a comment explaining
whywe're careful.  Author: Andres Freund Discussion: https://postgr.es/m/E1duOkJ-0001Mc-U5@gemulon.postgresql.org
https://git.postgresql.org/pg/commitdiff/71edbb6f66f7139d6209334ef8734a122ba06b56

- s/NULL byte/NUL byte/ in comment refering to C string terminator. Reported-By: Robert Haas Discussion:
https://postgr.es/m/CA+Tgmoa+YBvWgFST2NVoeXjVSohEpK=vqnVCsoCkhTVVxfLcVQ@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/896537f078ba4d709ce754ecaff8350fd55bdfd8

- Make WAL segment size configurable at initdb time.  For performance reasons a larger segment size than the default
16MBcan be useful. A larger segment size has two main benefits: Firstly, in setups using archiving, it makes it easier
towrite scripts that can keep up with higher amounts of WAL, secondly, the WAL has to be written and synced to disk
lessfrequently.  But at the same time large segment size are disadvantageous for smaller databases. So far the segment
sizehad to be configured at compile time, often making it unrealistic to choose one fitting to a particularly load.
Thereforechange it to a initdb time setting.  This includes a breaking changes to the xlogreader.h API, which now
requiresthe current segment size to be configured.  For that and similar reasons a number of binaries had to be taught
howto recognize the current segment size.  Author: Beena Emerson, editorialized by Andres Freund Reviewed-By: Andres
Freund,David Steele, Kuntal Ghosh, Michael Paquier, Peter Eisentraut, Robert Hass, Tushar Ahuja Discussion:
https://postgr.es/m/CAOG9ApEAcQ--1ieKbhFzXSQPw_YLmepaa4hNdnY5+ZULpt81Mw@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/fc49e24fa69a15efacd5b8958115ed9c43c48f9a

- Make new crash restart test a bit more robust.  Add timeouts in case psql doesn't deliver the expected output, and
tryto cause the monitoring psql to be fully connected to a backend.  This isn't necessarily everything needed, but at
leastthe timeouts should reduce the pain for buildfarm owners. Author: Andres Freund Reported-By: Tom Lane, BF animals
prairiedogand calliphoridae Discussion: https://postgr.es/m/E1du6ZT-00043I-91@gemulon.postgresql.org
https://git.postgresql.org/pg/commitdiff/1910353675bd149e1020b29c0fae02538fc358cd

- Accept that server might not be able to send error in crash recovery test.  As it turns out we can't rely that the
script'smonitoring session is terminated with a proper error by the server, because the session might be terminated
whilealready trying to send data.  Also improve robustness and error reporting facilities of the test, developed while
debuggingthis issue. Discussion: https://postgr.es/m/20170920020038.kllxgilo7xzwmtto@alap3.anarazel.de
https://git.postgresql.org/pg/commitdiff/5ada1fcd0c30be1b0b793a802cf6da386a6c1925

- Fix s/intidb/initdb/ typo.  Reported-By: Michael Paquier Discussion:
https://postgr.es/m/CAB7nPqTfaKAYZ4wuUM-W8kc4VnXrxX1=5-a9i==VoUPTMFpsgg@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/f9583e86b4bfa8c4e4d83ab33e5dcdaeab5c45a1

- Expand expected output for recovery test even further.  I'd assumed that the backend being killed should be able to
getout an error message - but it turns out it's not guaranteed that it's not still sending a ready-for-query.  Really
needto do something about getting these error message to the client. Reported-By: Thomas Munro, Tom Lane Discussion:
https://postgr.es/m/CAEepm=0TE90nded+bNthP45_PEvGAAr=3gxhHJObL4xmOLtX0w@mail.gmail.com
https://postgr.es/m/14968.1506101414@sss.pgh.pa.us
https://git.postgresql.org/pg/commitdiff/8d926029e817d280b2376433e3aaa3895e1a7128

- Add inline murmurhash32(uint32) function.  The function already existed in tidbitmap.c but more users requiring fast
hashingof 32bit ints are coming up. Author: Andres Freund Discussion:
https://postgr.es/m/20170914061207.zxotvyopetm7lrrp@alap3.anarazel.de
https://git.postgresql.org/pg/commitdiff/791961f59b792fbd4f0a992d3ccab47298e79103

Andrew Dunstan pushed:

- Add citext_pattern_ops for citext contrib module.  This is similar to text_pattern_ops.  Alexey Chernyshov, reviewed
byJacob Champion. https://git.postgresql.org/pg/commitdiff/f2464997644c64b5dec93ab3c08305f48bfe14f1
 

- Set client encoding to UTF8 for the citext regression script.  Problem introduced with non-ascii characters in commit
f2464997644cand discovered on various buildfarm animals.
https://git.postgresql.org/pg/commitdiff/890faaf1957759c6e17fbcbfd16f7cabc4a59d07

- Disable multi-byte citext tests.  This reverts commit 890faaf1 which attempted unsuccessfully to deal with the
problem,and instead just comments out these tests like other similar tests elsewhere in the script.
https://git.postgresql.org/pg/commitdiff/d1687c6926819f023c78b353458950a303796aba

- Quieten warnings about unused variables.  These variables are only ever written to in assertion-enabled builds, and
thelatest Microsoft compilers complain about such variables in non-assertion-enabled builds.  Apparently they don't
worryso much about variables that are written to but not read from, so most of our PG_USED_FOR_ASSERTS_ONLY variables
don'tcause the problem.  Discussion: https://postgr.es/m/7800.1505950322@sss.pgh.pa.us
https://git.postgresql.org/pg/commitdiff/28ae524bbf865d23eb10f6ae1b996d59dcc30e4e

- Provide a test for variable existence in psql.  "\if :{?variable_name}" will be translated to "\if TRUE" if the
variableexists and "\if FALSE" otherwise. Thus it will be possible to execute code conditionally on the existence of
thevariable, regardless of its value.  Fabien Coelho, with some review by Robins Tharakan and some light text editing
byme.  Discussion: https://postgr.es/m/alpine.DEB.2.20.1708260835520.3627@lancre
https://git.postgresql.org/pg/commitdiff/d57c7a7c506276597af619bdb8c62fa5b592745a

Bruce Momjian pushed:

- doc:  add example of % substitution for connection URIs.  Reported-by: Zhou Digoal Discussion:
https://postgr.es/m/20170912133722.25637.91@wrigleys.postgresql.orgBackpatch-through: 10
https://git.postgresql.org/pg/commitdiff/d61f5bb7c444255b064a60df782907f7dddad61a

- docs:   re-add instructions on setting wal_level for rsync use.  This step was erroneously removed four days ago by
me. Reported-by: Magnus via IM Backpatch-through: 9.5
https://git.postgresql.org/pg/commitdiff/00210e3fb974ff2b9affc4d8f3b29f9cb3645a60

Magnus Hagander pushed:

- Mention need for --no-inc-recursive in rsync command.  Since rsync 3.0.0 (released in 2008), the default way to
enumeratechanges was changed in a way that makes it less likely that the hardlink sync mode works. Since the whole
pointof the documented procedure is for the hardlinks to work, change our docs to suggest using the backwards
compatibilityswitch. https://git.postgresql.org/pg/commitdiff/2c74e6c1dcc5002fa8b822e5757f6c95d899fb7a
 

Robert Haas pushed:

- Fix typo.  Thomas Munro Discussion:
http://postgr.es/m/CAEepm=2j-HAgnBUrAazwS0ry7Z_ihk+d7g+Ye3u99+6WbiGt_Q@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/7f3a3312abf34ea7e899046e326775612802764b

- Fix create_lateral_join_info to handle dead relations properly.  Commit 0a480502b092195a9b25a2f0f199a21d592a9c57
brokeit.  Report by Andreas Seltenreich.  Fix by Ashutosh Bapat.  Discussion:
http://postgr.es/m/874ls2vrnx.fsf@ansel.ydns.eu
https://git.postgresql.org/pg/commitdiff/57eebca03a9eb61eb18f8ea9db94775653f797d1

- Associate partitioning information with each RelOptInfo.  This is not used for anything yet, but it is necessary
infrastructurefor partition-wise join and for partition pruning without constraint exclusion.  Ashutosh Bapat, reviewed
byAmit Langote and with quite a few changes, mostly cosmetic, by me. Additional review and testing of this patch series
byAntonin Houska, Amit Khandekar, Rafia Sabih, Rajkumar Raghuwanshi, Thomas Munro, and Dilip Kumar. Discussion:
http://postgr.es/m/CAFjFpRfneFG3H+F6BaiXemMrKF+FY-POpx3Ocy+RiH3yBmXSNw@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/9140cf8269b0c4ae002b2748d93979d535891311

- hash: Implement page-at-a-time scan.  Commit 09cb5c0e7d6fbc9dee26dc429e4fc0f2a88e5272 added a similar optimization to
btreeback in 2006, but nobody bothered to implement the same thing for hash indexes, probably because they weren't
WAL-loggedand had lots of other performance problems as well.  As with the corresponding btree case, this eliminates
theproblem of potentially needing to refind our position within the page, and cuts down on pin/unpin traffic as well.
AshutoshSharma, reviewed by Alexander Korotkov, Jesper Pedersen, Amit Kapila, and me.  Some final edits to comments and
READMEby me.  Discussion: http://postgr.es/m/CAE9k0Pm3KTx93K8_5j6VMzG4h5F+SyknxUwXrN-zqSZ9X8ZS3w@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/7c75ef571579a3ad7a1d3ee909f11dba5e0b9440

- For wal_consistency_checking, mask page checksum as well as page LSN.  If the LSN is different, the checksum will be
different,too.  Ashwin Agrawal, reviewed by Michael Paquier and Kuntal Ghosh Discussion:
http://postgr.es/m/CALfoeis5iqrAU-+JAN+ZzXkpPr7+-0OAGv7QUHwFn=-wDy4o4Q@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/6a2fa09c0cba0e5a11854d733872ac18511f4c83

Álvaro Herrera pushed:

- Document further existing locks as wait events.  Reported-by: Jeremy Schneider Author: Michael Paquier Discussion:
https://postgr.es/m/CA+fnDAZaPCwfY8Lp-pfLnUGFAXRu1VfLyRgdup-L-kwcBj8MqQ@mail.gmail.com
https://git.postgresql.org/pg/commitdiff/885cab58115a5af9484926ddee8dca3dc0106c1e

- Test BRIN autosummarization.  There was no coverage for this code. Reported-by: Nikolay Shaplov, Tom Lane Discussion:
https://postgr.es/m/2700647.XEouBYNZic@x200mhttps://postgr.es/m/13849.1506114543@sss.pgh.pa.us
https://git.postgresql.org/pg/commitdiff/404ba54e8fd3036eee0f9241f68b17092ce734ee

== Pending Patches ==

Rushabh Lathia sent in another revision of a patch to fix an error that
manifested as GatherMerge failing to push a target list down.

Jesper Pedersen sent in another revision of a patch to implement hash
partitioning.

Oliver Ford sent in another revision of a patch to add Roman numeral conversion
to to_number.

Dilip Kumar sent in another revision of a patch to prune partitions more
efficiently in queries.

Ashutosh Bapat sent in a patch to skip "dead" relations in
create_lateral_join_info().

Jeevan Chalke sent in another revision of a patch to implement partition-wise
aggregation/grouping for declaratively partitioned tables.

Kyotaro HORIGUCHI sent in another revision of a patch to fix a race condition
between SELECT and ALTER TABLE NO INHERIT.

Noah Misch sent in a patch to change the release notes to do much less invasive
surgery on the information_schema.table_privileges view, replacing it rather
than re-creating the information schema whole cloth.

Michaël Paquier sent in another revision of a patch to remove
ALLOW_DANGEROUS_LO_FUNCTIONS for LO-related superuser checks, replace superuser
checks of large object import/export by ACL checks, remove ACL checks for large
objects when opening them.

Haribabu Kommi sent in another revision of a patch to add the support of storage
handler as an access method, add storage AM API hooks and related functions, add
storageam handler to the relation structure, add a tuple visibility function,
slot hooks, a tuple insert API, and scan functions to the storage AM, and remove
HeapScanDesc uses outside heap-specific code.

Rushabh Lathia sent in another revision of a patch to add parallel B-tree index
build sorting.

Jeff Janes sent in another revision of a patch to ensure that parallel paths
include target list cost.

Thomas Munro sent in another revision of a patch to make SERIALIZABLE isolation
work with parallel operations.

Masahiko Sawada sent in three more revisions of a patch to enable custom
initialization for pgbench.

Ashutosh Bapat sent in two more revisions of a patch to implement partition-wise
join for joins between (declaratively) partitioned tables.

Michaël Paquier sent in another revision of a patch to rewrite tests of
pg_upgrade as TAP tests.

Craig Ringer sent in a patch to add a utility class for value and identifier
quoting for use in TAP tests.

Vaishnavi Prabakaran sent in a patch to correct the debug log to reflect
the change of function name from lo_open to be_lo_open.

Amit Khandekar sent in another revision of a patch to implement parallel append.

Andres Freund sent in a patch to add the ability to force libpq to negotiate a
specific version of the protocol and add minimal v2 protocol regression tests
based on same.

Alexander Korotkov sent in two more revisions of a patch to add SP-GiST compress
methods, and GiST support for circles and polygons.

Masahiko Sawada sent in three revisions of a patch to fix an assertion failure
when the non-exclusive pg_stop_backup aborted.

Amul Sul sent in a patch to only handle oid sysattr, to reduce overhead in a
patch set intended to improve syscache/catcache performance.

Rajkumar Raghuwanshi sent in a patch to add some tests for partition-wise join
between declaratively partitioned tables.

Thomas Munro sent in a shell program which shows which blocks of SQL hit which
lines of C.

Peter Eisentraut sent in another revision of a patch to run only top-level
recursive lcov, have lcov exclude external files, add lcov --initial, add the
PostgreSQL version to coverage output, remove the coverage details view, run
coverage commands quietly, improve vpath support in the plperl build, support
coverage on vpath builds, and exclude flex-generated code from coverage testing.

Ashutosh Sharma sent in three more revisions of a patch to rewrite hash index
scan to work page at a time, remove redundant hash function _hash_step and do
some code cleanup, and improve the locking startegy during VACUUM in hash index
for regular tables.

Lucas Fittl sent in a patch to only skip query cancel itself when query cancel
holdoff count is positive.

Rafia Sabih sent in another revision of a patch to make gather faster.

Tom Lane sent in another revision of a patch to support arrays of domains.

Kyotaro HORIGUCHI sent in a patch to fix a misbehavior of hash indexes on
unlogged tables.

Julien Rouhaud sent in two more revisions of a patch to implement sorted append.

Nathan Bossart sent in three more revisions of a patch to enable specifying
multiple tables in VACUUM commands.

Nikita Glukhov sent in a patch to fix a circle bounding box inconsistency in the
GiST compress method and fix GiST ordering by distance for NULLs and NaNs.

Kyotaro HORIGUCHI sent in a patch to fix an issue where analyzeCTE is too strict
about typmods.

Amit Khandekar sent in another revision of a patch to enable UPDATEs of a
partition key which move the row to another partition.

Michaël Paquier sent in a patch to document new wait events.

Andres Freund sent in another revision of a patch to improve sys/catcache
performance.

Yura Sokolov sent in a patch to skip scanning a btree index if no pending
deleted pages exists.

Chen Huajun sent in another revision of a patch to make pg_rewind to not copy
useless WAL files.

Robert Haas sent in a patch to skip LSN checks during hash index scans.

Peter Eisentraut sent in a patch to expand the collation documentation and add
test cases for the new advanced ICU features.

Fabien COELHO sent in a patch to pgbench to use enum to store its meta commands.

Yura Sokolov sent in another revision of a patch to improve compactify_tuples.

Fabien COELHO sent in another revision of a patch to add server version
information to psql.

Tom Lane sent in a patch to make construct_[md_]array() handle empty arrays
correctly.

Andrew Dunstan sent in two more revisions of a patch to fix an infelicity in the
enum type.

Amit Kapila sent in another revision of a patch to change metapage usage for
B-tree and hash indexes.

Fabien COELHO sent in another revision of a patch to fix a pgbench regression
test failure.

Thomas Munro sent in a patch to improve the LDAP cleanup code in error paths,
log diagnostic messages if errors occur during LDAP auth, and add a regression
test to log an LDAP diagnostic message.

Mark Rofail sent in another revision of a patch to add foreign key arrays.

Tomas Vondra sent in another revision of a patch to make VACUUM and ANALYZE
agree on what reltuples means.

Erik Rijkers sent in a patch to improve some comments in prepunion.c.

Tomas Vondra sent in another revision of a patch to implement a generational
memory allocator.

Thomas Munro sent in another revision of a patch to use fallocate() to
preallocate "posix" DSM segments on Linux.


-- 
Sent via pgsql-announce mailing list (pgsql-announce@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-announce