== PostgreSQL Weekly News - October 02 2016 ==

== PostgreSQL Weekly News - October 02 2016 ==

PostgreSQL 9.6 Released!
https://www.postgresql.org/docs/current/static/release-9-6.html

== PostgreSQL Product News ==

pgAdmin4 1.0, a web- and native GUI control center for PostgreSQL, released.
https://www.pgadmin.org/

barman 2.0, a backup and recovery manager for PostgreSQL, released.
http://docs.pgbarman.org/release/2.0/

== PostgreSQL Local ==

Postgres Vision 2016 will be held October 11-13, 2016, in San Francisco.
http://postgresvision.com/

The Korean Users Group will hold its first PGDay event on October 15, 2016 in Seoul.
http://pgday.postgresql.kr/

PostgreSQL Conference Europe will take place in Tallin, Estonia, on
November 1-4, 2016.  The schedule has been published.
http://2016.pgconf.eu/registration/

PGDay Austin 2016, will take place on November 12, 2016.  Submission deadline
is September 21, 2016 by midnight CST.  Details and submission form at:
https://www.postgresql.us/events/2016/austin

PgConf Silicon Valley 2016 will be held on November 14-16, 2016.
http://www.pgconfsv.com/

CHAR(16) will take place in New York, December 6, 2016.
http://charconference.org/

PGDay.IT 2016 will take place in Prato on December the 13th 2016.
http://pgday.it

== PostgreSQL in the News ==

Planet PostgreSQL: http://planet.postgresql.org/

PostgreSQL Weekly News is brought to you this week by David Fetter

Submit news and announcements by Sunday at 3:00pm Pacific time.
Please send English language ones to david@fetter.org, German language
to pwn@pgug.de, Italian language to pwn@itpug.org.

== Applied Patches ==

Tom Lane pushed:

- Refer to OS X as "macOS", except for the port name which is still "darwin".
  We weren't terribly consistent about whether to call Apple's OS "OS X" or "Mac
  OS X", and the former is probably confusing to people who aren't Apple users.
  Now that Apple has rebranded it "macOS", follow their lead to establish a
  consistent naming pattern.  Also, avoid the use of the ancient project name
  "Darwin", except as the port code name which does not seem desirable to
  change.  (In short, this patch touches documentation and comments, but no
  actual code.) I didn't touch contrib/start-scripts/osx/, either.  I suspect
  those are obsolete and due for a rewrite, anyway.  I dithered about whether to
  apply this edit to old release notes, but those were responsible for quite a
  lot of the inconsistencies, so I ended up changing them too.  Anyway, Apple's
  being ahistorical about this, so why shouldn't we be?
  http://git.postgresql.org/pg/commitdiff/da6c4f6ca88df346573bdada2aa2544510bf167e

- Document has_type_privilege().  Evidently an oversight in commit 729205571.
  Back-patch to 9.2 where privileges for types were introduced.  Report:
  <20160922173517.8214.88959@wrigleys.postgresql.org>
  http://git.postgresql.org/pg/commitdiff/a4afb2b5c0b409bb175c20104b2ae9d47cf71be6

- Replace the built-in GIN array opclasses with a single polymorphic opclass.
  We had thirty different GIN array opclasses sharing the same operators and
  support functions.  That still didn't cover all the built-in types, nor did it
  cover arrays of extension-added types.  What we want is a single polymorphic
  opclass for "anyarray".  There were two missing features needed to make this
  possible: 1. We have to be able to declare the index storage type as
  ANYELEMENT when the opclass is declared to index ANYARRAY.  This just takes a
  few more lines in index_create().  Although this currently seems of use only
  for GIN, there's no reason to make index_create() restrict it to that.  2. We
  have to be able to identify the proper GIN compare function for the index
  storage type.  This patch proceeds by making the compare function optional in
  GIN opclass definitions, and specifying that the default btree comparison
  function for the index storage type will be looked up when the opclass omits
  it.  Again, that seems pretty generically useful.  Since the comparison
  function lookup is done in initGinState(), making use of the second feature
  adds an additional cache lookup to GIN index access setup.  It seems unlikely
  that that would be very noticeable given the other costs involved, but maybe
  at some point we should consider making GinState data persist longer than it
  now does --- we could keep it in the index relcache entry, perhaps.  Rather
  fortuitously, we don't seem to need to do anything to get this change to play
  nice with dump/reload or pg_upgrade scenarios: the new opclass definition is
  automatically selected to replace existing index definitions, and the on-disk
  data remains compatible.  Also, if a user has created a custom opclass
  definition for a non-builtin type, this doesn't break that, since CREATE INDEX
  will prefer an exact match to opcintype over a match to ANYARRAY.  However, if
  there's anyone out there with handwritten DDL that explicitly specifies
  _bool_ops or one of the other replaced opclass names, they'll need to adjust
  that.  Tom Lane, reviewed by Enrique Meneses Discussion:
  <14436.1470940379@sss.pgh.pa.us>
  http://git.postgresql.org/pg/commitdiff/fdc9186f7ed1ead827509584f3b763f8dc332c43

- Fix newly-introduced issues in pgbench.  The result of FD_ISSET() doesn't
  necessarily fit in a bool, though assigning it to one might accidentally work
  depending on platform and which socket FD number is being inquired of.
  Rewrite to test it with if(), rather than making any specific assumption about
  the result width, to match the way every other such call in PG is written.
  Don't break out of the input_mask-filling loop after finding the first client
  that we're waiting for results from.  That mostly breaks parallel query
  management.  Also, if we choose not to call select(), be sure to clear out any
  bits the mask-filling loop might have set, so that we don't accidentally call
  doCustom for clients we don't know have input.  Doing so would likely be
  harmless, but it's a waste of cycles and doesn't seem to be intended.  Make
  this_usec wide enough.  (Yeah, the value would usually fit in an int, but then
  why are we using int64 everywhere else?) Minor cosmetic adjustments, mostly
  comment improvements.  Problems introduced by commit 12788ae49.  The first
  issue was discovered by buildfarm testing, the others by code review.
  http://git.postgresql.org/pg/commitdiff/9779bda86c026e645773a3308f9169c7c0791f7c

- Improve contrib/cube's handling of zero-D cubes, infinities, and NaNs.  It's
  always been possible to create a zero-dimensional cube by converting from a
  zero-length float8 array, but cube_in failed to accept the '()' representation
  that cube_out produced for that case, resulting in a dump/reload hazard.  Make
  it accept the case.  Also fix a couple of other places that didn't behave
  sanely for zero-dimensional cubes: cube_size would produce 1.0 when surely the
  answer should be 0.0, and g_cube_distance risked a divide-by-zero failure.
  Likewise, it's always been possible to create cubes containing float8 infinity
  or NaN coordinate values, but cube_in couldn't parse such input, and cube_out
  produced platform-dependent spellings of the values.  Convert them to use
  float8in_internal and float8out_internal so that the behavior will be the same
  as for float8, as we recently did for the core geometric types (cf commit
  50861cd68).  As in that commit, I don't pretend that this patch fixes all
  insane corner-case behaviors that may exist for NaNs, but it's a step forward.
  (This change allows removal of the separate cube_1.out and cube_3.out
  expected-files, as the platform dependency that previously required them is
  now gone: an underflowing coordinate value will now produce an error not plus
  or minus zero.) Make errors from cube_in follow project conventions as to
  spelling ("invalid input syntax for cube" not "bad cube representation") and
  errcode (INVALID_TEXT_REPRESENTATION not SYNTAX_ERROR).  Also a few marginal
  code cleanups and comment improvements.  Tom Lane, reviewed by Amul Sul
  Discussion: <15085.1472494782@sss.pgh.pa.us>
  http://git.postgresql.org/pg/commitdiff/f31a931fade868d788ef4480c59753a2d5059246

- Redesign parallel dump/restore's wait-for-workers logic.  The
  ListenToWorkers/ReapWorkerStatus APIs were messy and hard to use.  Instead,
  make DispatchJobForTocEntry register a callback function that will take care
  of state cleanup, doing whatever had been done by the caller of
  ReapWorkerStatus in the old design.  (This callback is essentially just the
  old mark_work_done function in the restore case, and a trivial test for worker
  failure in the dump case.)  Then we can have ListenToWorkers call the callback
  immediately on receipt of a status message, and return the worker to WRKR_IDLE
  state; so the WRKR_FINISHED state goes away.  This allows us to design a
  unified wait-for-worker-messages loop: WaitForWorkers replaces
  EnsureIdleWorker and EnsureWorkersFinished as well as the mess in
  restore_toc_entries_parallel.  Also, we no longer need the fragile API spec
  that the caller of DispatchJobForTocEntry is responsible for ensuring there's
  an idle worker, since DispatchJobForTocEntry can just wait until there is one.
  In passing, I got rid of the ParallelArgs struct, which was a net negative in
  terms of notational verboseness, and didn't seem to be providing any
  noticeable amount of abstraction either.  Tom Lane, reviewed by Kevin Grittner
  Discussion: <1188.1464544443@sss.pgh.pa.us>
  http://git.postgresql.org/pg/commitdiff/b7b8cc0cfcf1c956b752f3e25894f9ad607583b7

- Rationalize parallel dump/restore's handling of worker cmd/status messages.
  The existing APIs for creating and parsing command and status messages are
  rather messy; for example, archive-format modules have to provide code for
  constructing command messages, which is entirely pointless since the code to
  read them is hard-wired in WaitForCommands() and hence no format-specific
  variation is actually possible.  But there's little foreseeable reason to need
  format-specific variation anyway.  The situation for status messages is no
  better; at least those are both constructed and parsed by format-specific
  code, but said code is quite redundant since there's no actual need for
  format-specific variation.  To add insult to injury, the first API involves
  returning pointers to static buffers, which is bad, while the second involves
  returning pointers to malloc'd strings, which is safer but randomly
  inconsistent.  Hence, get rid of the MasterStartParallelItem and
  MasterEndParallelItem APIs, and instead write centralized functions that
  construct and parse command and status messages.  If we ever do need more
  flexibility, these functions can be the standard implementations of
  format-specific callback methods, but that's a long way off if it ever
  happens.  Tom Lane, reviewed by Kevin Grittner Discussion:
  <17340.1464465717@sss.pgh.pa.us>
  http://git.postgresql.org/pg/commitdiff/fb03d08a89e81a68585f17fd8e7f21c618f4e851

- Make struct ParallelSlot private within pg_dump/parallel.c.  The only field of
  this struct that other files have any need to touch is the pointer to the
  TocEntry a worker is working on.  (Well, pg_backup_archiver.c is actually
  looking at workerStatus too, but that can be finessed by specifying that the
  TocEntry pointer is NULL for a non-busy worker.) Hence, move out the TocEntry
  pointers to a separate array within struct ParallelState, and then we can make
  struct ParallelSlot private.  I noted the possibility of this previously, but
  hadn't got round to actually doing it.  Discussion:
  <1188.1464544443@sss.pgh.pa.us>
  http://git.postgresql.org/pg/commitdiff/0109ab27609c0d58c1eddc6b799077d0968083de

- Disallow pushing volatile quals past set-returning functions.  Pushing an
  upper-level restriction clause into an unflattened subquery-in-FROM is okay
  when the subquery contains no SRFs in its targetlist, or when it does but the
  SRFs are unreferenced by the clause *and the clause is not volatile*.
  Otherwise, we're changing the number of times the clause is evaluated, which
  is bad for volatile quals, and possibly changing the result, since a volatile
  qual might succeed for some SRF output rows and not others despite not
  referencing any of the changing columns.  (Indeed, if the clause is something
  like "random() > 0.5", the user is probably expecting exactly that behavior.)
  We had most of these restrictions down, but not the one about the upper clause
  not being volatile.  Fix that, and add a regression test to illustrate the
  expected behavior.  Although this is definitely a bug, it doesn't seem like
  back-patch material, since possibly some users don't realize that the broken
  behavior is broken and are relying on what happens now.  Also, while the added
  test is quite cheap in the wake of commit a4c35ea1c, it would be much more
  expensive (or else messier) in older branches.  Per report from Tom van
  Tilburg.  Discussion:
  <CAP3PPDiucxYCNev52=YPVkrQAPVF1C5PFWnrQPT7iMzO1fiKFQ@mail.gmail.com>
  http://git.postgresql.org/pg/commitdiff/72daabc7a3e75788df862104b8f723513c2471ae

- Make to_timestamp() and to_date() range-check fields of their input.
  Historically, something like to_date('2009-06-40','YYYY-MM-DD') would return
  '2009-07-10' because there was no prohibition on out-of-range month or day
  numbers.  This has been widely panned, and it also turns out that Oracle
  throws an error in such cases.  Since these functions are nominally
  Oracle-compatibility features, let's change that.  There's no particular
  restriction on year (modulo the fact that the scanner may not believe that
  more than 4 digits are year digits, a matter to be addressed separately if at
  all).  But we now check month, day, hour, minute, second, and
  fractional-second fields, as well as day-of-year and second-of-day fields if
  those are used.  Currently, no checks are made on ISO-8601-style week numbers
  or day numbers; it's not very clear what the appropriate rules would be there,
  and they're probably so little used that it's not worth sweating over.  Artur
  Zakirov, reviewed by Amul Sul, further adjustments by me Discussion:
  <1873520224.1784572.1465833145330.JavaMail.yahoo@mail.yahoo.com> See-Also:
  <57786490.9010201@wars-nicht.de>
  http://git.postgresql.org/pg/commitdiff/d3cd36a133d96ad5578b6c10279b55fd5b538093

- Rationalize format-picture caching logic in formatting.c.  Add a validity flag
  to DCHCacheEntry and NUMCacheEntry entries, and do not set it true until after
  we've parsed the supplied format string.  This allows dealing with possible
  errors while parsing the format without the baroque hack that was there before
  (which only covered errors within NUMDesc_prepare, anyway).  We can get rid of
  the PG_TRY in NUMDesc_prepare, as well as last_NUMCacheEntry and
  NUM_cache_remove.  (Essentially, this reverts commit ff783fbae in favor of a
  less fragile solution; the problems with that approach are well illustrated by
  later hacking such as 55f927a46.) In passing, define the size of these caches
  as DCH_CACHE_ENTRIES not DCH_CACHE_FIELDS + 1 (whoever thought that was a good
  definition?) and likewise for the NUM cache.  Also const-ify format string
  parameters where convenient, and merge duplicated cache lookup logic.  This is
  primarily driven by a proposed patch from Artur Zakirov, which introduced some
  ereport's into format string parsing for the datetime case.  He proposed
  preventing the creation of invalid cache entries by parsing the format string
  first into a local-variable array, and then copying that to a cache entry.
  That seemed a bit ugly to me, and anyway randomly different from the way the
  identical problem had been solved for the numeric case.  Let's make the two
  sets of code more similar not less so.  I'm not sure whether we'll adopt the
  new error conditions Artur proposes, but this patch seems like good code
  cleanup and future-proofing in any case.  The existing code is critically (and
  undocumented-ly) dependent on no elog being thrown out of several nontrivial
  functions, which is trouble waiting to happen, though it doesn't seem to be
  actively broken today.  Discussion:
  <b2a39359-3282-b402-f4a3-057aae500ee7@postgrespro.ru>
  http://git.postgresql.org/pg/commitdiff/83bed06be4e808f3da30f99b6c91e9efda3961ad

- Allow contrib/file_fdw to read from a program, like COPY FROM PROGRAM.  This
  patch just exposes COPY's FROM PROGRAM option in contrib/file_fdw.  There
  don't seem to be any security issues with that that are any worse than what
  already exist with file_fdw and COPY; as in the existing cases, only
  superusers are allowed to control what gets executed.  A regression test case
  might be nice here, but choosing a 100% portable command to run is hard.  (We
  haven't got a test for COPY FROM PROGRAM itself, either.) Corey Huinker and
  Adam Gomaa, reviewed by Amit Langote Discussion:
  <CADkLM=dGDGmaEiZ=UDepzumWg-CVn7r8MHPjr2NArj8S3TsROQ@mail.gmail.com>
  http://git.postgresql.org/pg/commitdiff/8e91e12bc3af85ba2287866669268f6825d2cc03

- Fix multiple portability issues in pg_upgrade's rewriteVisibilityMap().  This
  is new code in 9.6, and evidently we missed out testing it as thoroughly as it
  should have been.  Bugs fixed here: 1. Use binary not text mode to open the
  files on Windows.  Before, if the visibility map chanced to contain two bytes
  that looked like \r\n, Windows' read() would convert that to \n, which both
  corrupts the map data and causes the file to look shorter than it should.
  Unless you were *very* unlucky and had an exact multiple of 8K such
  occurrences in each VM file, this would cause pg_upgrade to report a failure,
  though with a rather obscure error message.  2. The code for copying rebuilt
  bytes into the output was simply wrong.  It chanced to work okay on
  little-endian machines but would emit the bytes in the wrong order on
  big-endian, leading to silent corruption of the visibility map data.  3. The
  code was careless about alignment of the working buffers.  Given all three of
  an alignment-picky architecture, a compiler that chooses to put the
  new_vmbuf[] local variable at an odd starting address, and a checksum-enabled
  database, pg_upgrade would dump core.  Point one was reported by Thomas
  Kellerer, the other two detected by code-reading.  Point two is much the
  nastiest of these issues from an impact standpoint, though fortunately it
  affects only a minority of users.  The Windows issue will definitely bite
  people, but it seems quite unlikely that there would be undetected corruption
  from that.  In addition, I failed to resist the temptation to do some minor
  cosmetic adjustments, mostly improving the comments.  It would be a good idea
  to try to improve the error reporting here, but that seems like material for a
  separate patch.  -------
  http://git.postgresql.org/pg/commitdiff/5afcd2aa740f87fa6cec7c6693a4891b9df2b67d

- Improve error reporting in pg_upgrade's file copying/linking/rewriting.  The
  previous design for this had copyFile(), linkFile(), and
  rewriteVisibilityMap() returning strerror strings, with the caller producing
  one-size-fits-all error messages based on that.  This made it impossible to
  produce messages that described the failures with any degree of precision,
  especially not short-read problems since those don't set errno at all.  Since
  pg_upgrade has no intention of continuing after any error in this area, let's
  fix this by just letting these functions call pg_fatal() for themselves,
  making it easy for each point of failure to have a suitable error message.
  Taking this approach also allows dropping cleanup code that was unnecessary
  and was often rather sloppy about preserving errno.  To not lose relevant info
  that was reported before, pass in the schema name and table name of the
  current table so that they can be included in the error reports.  An
  additional problem was the use of getErrorText(), which was flat out wrong for
  all but a couple of call sites, because it unconditionally did
  "_dosmaperr(GetLastError())" on Windows.  That's only appropriate when
  reporting an error from a Windows-native API, which only a couple of the
  callers were actually doing.  Thus, even the reported strerror string would be
  unrelated to the actual failure in many cases on Windows.  To fix, get rid of
  getErrorText() altogether, and just have call sites do strerror(errno)
  instead, since that's the way all the rest of our frontend programs do it.
  Add back the _dosmaperr() calls in the two places where that's actually
  appropriate.  In passing, make assorted messages hew more closely to project
  style guidelines, notably by removing initial capitals in
  not-complete-sentence primary error messages.  (I didn't make any effort to
  clean up places I didn't have another reason to touch, though.) Per discussion
  of a report from Thomas Kellerer.  Back-patch to 9.6, but no further; given
  the relative infrequency of reports of problems here, it's not clear it's
  worth adapting the patch to older branches.  Patch by me, but with credit to
  Alvaro Herrera for spotting the issue with getErrorText's misuse of
  _dosmaperr().  Discussion: <nsjrbh$8li$1@blaine.gmane.org>
  http://git.postgresql.org/pg/commitdiff/f002ed2b8e45286fe73a36e119cba2016ea0de19

- Fix misplacement of submake-generated-headers prerequisites.  The sequence
  "configure; cd src/pl/plpython; make -j" failed due to trying to compile
  plpython's .o files before the generated headers finished building.  (This is
  an important real-world case, since it's the typical second step when building
  both plpython2 and plpython3.) This happens because the
  submake-generated-headers target is not placed in a way to make it a
  prerequisite to compiling the .o files.  Fix that.  Checking other uses of
  submake-generated-headers, I noted that the one attached to pg_regress was
  similarly misplaced; but it's actually not needed at all for pg_regress.o,
  rather regress.o, so move it to be a prerequisite of that.  Back-patch to 9.6
  where submake-generated-headers was introduced (by commit 548af97fc).  It's
  not immediately clear to me why the previous coding didn't have the same
  issue; but since we've not had field reports of plpython make failing, leave
  it alone in the older branches.  Pavel Raiskup and Tom Lane Discussion:
  <1925924.izSMJEZO3x@unused-4-107.brq.redhat.com>
  http://git.postgresql.org/pg/commitdiff/7107d58ec5a3c45967e77525809612a5f89b97f3

- Fix misstatement in comment in Makefile.shlib.  There is no need for "all:
  all-lib" to be placed before inclusion of Makefile.shlib.  Makefile.global is
  what ensures that "all" is the default target, and we already document that
  that has to be included first.  Per comment from Pavel Raiskup.  Discussion:
  <1925924.izSMJEZO3x@unused-4-107.brq.redhat.com>
  http://git.postgresql.org/pg/commitdiff/ea046f08d1bcee56c3bedfa16a05c38a0515b41d

- Copy-editing for contrib/pg_visibility documentation.  Add omitted names for
  some function parameters.  Fix some minor grammatical issues.
  http://git.postgresql.org/pg/commitdiff/33596edf09516a7cab65914e16cfd6adf9fc55d1

- Fix bugs in contrib/pg_visibility.  collect_corrupt_items() failed to
  initialize tuple.t_self.  While HeapTupleSatisfiesVacuum() doesn't actually
  use that value, it does Assert that it's valid, so that the code would dump
  core if ip_posid chanced to be zero.  (That's somewhat unlikely, which
  probably explains how this got missed.  In any case it wouldn't matter for
  field use.) Also, collect_corrupt_items was returning the wrong TIDs, that is
  the contents of t_ctid rather than the tuple's own location.  This would be
  the same thing in simple cases, but it could be wrong if, for example, a past
  update attempt had been rolled back, leaving a live tuple whose t_ctid doesn't
  point at itself.  Also, in pg_visibility(), guard against trying to read a
  page past the end of the rel.  The VM code handles inquiries beyond the end of
  the map by silently returning zeroes, and it seems like we should do the same
  thing here.  I ran into the assertion failure while using pg_visibility to
  check pg_upgrade's behavior, and then noted the other problems while reading
  the code.  Report: <29043.1475288648@sss.pgh.pa.us>
  http://git.postgresql.org/pg/commitdiff/9a109452da1b923e183f20fcf5516984d448ece9

- Do ClosePostmasterPorts() earlier in SubPostmasterMain().  In standard Unix
  builds, postmaster child processes do ClosePostmasterPorts immediately after
  InitPostmasterChild, that is almost immediately after being spawned.  This is
  important because we don't want children holding open the postmaster's end of
  the postmaster death watch pipe.  However, in EXEC_BACKEND builds,
  SubPostmasterMain was postponing this responsibility significantly, in order
  to make it slightly more convenient to pass the right flag value to
  ClosePostmasterPorts.  This is bad, particularly seeing that
  process_shared_preload_libraries() might invoke nearly-arbitrary code.
  Rearrange so that we do it as soon as we've fetched the socket FDs via
  read_backend_variables().  Also move the comment explaining about
  randomize_va_space to before the call of PGSharedMemoryReAttach, which is
  where it's relevant.  The old placement was appropriate when the reattach
  happened inside CreateSharedMemoryAndSemaphores, but that was a long time ago.
  Back-patch to 9.3; the patch doesn't apply cleanly before that, and it doesn't
  seem worth a lot of effort given that we've had no actual field complaints
  traceable to this.  Discussion: <4157.1475178360@sss.pgh.pa.us>
  http://git.postgresql.org/pg/commitdiff/3b90e38c5d592ea8ec8236287dd5c749fc041728

- Avoid leaking FDs after an fsync failure.  Fixes errors introduced in commit
  bc34223bc, as detected by Coverity.  In passing, report ENOSPC for a short
  write while padding a new wal file in open_walfile, make certain that
  close_walfile closes walfile in all cases, and improve a couple of comments.
  Michael Paquier and Tom Lane
  http://git.postgresql.org/pg/commitdiff/728ceba938dfadb204a4854bb76ae3b11b635401

- Add ALTER EXTENSION ADD/DROP ACCESS METHOD, and use it in pg_upgrade.  Without
  this, an extension containing an access method is not properly dumped/restored
  during pg_upgrade --- the AM ends up not being a member of the extension after
  upgrading.  Another oversight in commit 473b93287, reported by Andrew Dunstan.
  Report: <f7ac29f3-515c-2a44-21c5-ec925053265f@dunslane.net>
  http://git.postgresql.org/pg/commitdiff/e8bdee2770ff52aab208bc6f8965a4a01979d0aa

Peter Eisentraut pushed:

- Fix some typos in comment.
  http://git.postgresql.org/pg/commitdiff/440c8d1bbc8c62d225ab7c3e30e0a7db2639cd0f

- Fix CRC check handling in get_controlfile.  The previous patch broke this by
  returning NULL for a failed CRC check, which pg_controldata would then try to
  read.  Fix by returning the result of the CRC check in a separate argument.
  Michael Paquier and myself
  http://git.postgresql.org/pg/commitdiff/e79e6c4da152154544913b38354b98d07b65e411

- Remove dead line of code.
  http://git.postgresql.org/pg/commitdiff/967ed9205bfcd89c1dde1e85735b32b404721399

- Exclude additional directories in pg_basebackup.  The list of files and
  directories that pg_basebackup excludes from the backup was somewhat
  incomplete and unorganized.  Change that with having the exclusion driven from
  tables.  Clean up some code around it.  Also document the exclusions in more
  detail so that users of pg_start_backup can make use of it as well.  The
  contents of these directories are now excluded from the backup: pg_dynshmem,
  pg_notify, pg_serial, pg_snapshots, pg_subtrans Also fix a bug that a
  pg_repl_slot or pg_stat_tmp being a symlink would cause a corrupt tar header
  to be created.  Now such symlinks are included in the backup as empty
  directories.  Bug found by Ashutosh Sharma <ashu.coek88@gmail.com>.  From:
  David Steele <david@pgmasters.net> Reviewed-by: Michael Paquier
  <michael.paquier@gmail.com>
  http://git.postgresql.org/pg/commitdiff/6ad8ac6026287e3ccbc4d606b6ab6116ccc0eec8

- pg_basebackup pg_receivexlog: Issue fsync more carefully.  Several places
  weren't careful about fsyncing in the way.  See 1d4a0ab1 and 606e0f98 for
  details about required fsyncs.  This adds a couple of functions in src/common/
  that have an equivalent in the backend: durable_rename(), fsync_parent_path()
  From: Michael Paquier <michael.paquier@gmail.com>
  http://git.postgresql.org/pg/commitdiff/bc34223bc1e2c51dff2007b3d3bd492a09b5a491

- Switch pg_basebackup commands in Postgres.pm to use --nosync.  On slow
  machines, this greatly reduces the I/O pressure induced by the tests.  From:
  Michael Paquier <michael.paquier@gmail.com>
  http://git.postgresql.org/pg/commitdiff/728a3e73e96173bb0b0b2afc0da061991f9e3d5d

- pg_basebackup: Add --nosync option.  This is useful for testing, similar to
  initdb's --nosync.  From: Michael Paquier <michael.paquier@gmail.com>
  http://git.postgresql.org/pg/commitdiff/6ed2d8584cc680a2d6898480de74a57cd96176b5

- Move fsync routines of initdb into src/common/.  The intention is to used
  those in other utilities such as pg_basebackup and pg_receivexlog.  From:
  Michael Paquier <michael.paquier@gmail.com>
  http://git.postgresql.org/pg/commitdiff/bf5bb2e85b6492c7245b9446efaf43d52a98db13

- Fix compiler warnings.  This was missed in
  bf5bb2e85b6492c7245b9446efaf43d52a98db13, because the code is only visible
  under PG_FLUSH_DATA_WORKS.
  http://git.postgresql.org/pg/commitdiff/f2af8dc5ba8801c126d877a52076d441cac78646

- Add missing include files to configure tests.  atoi() needs stdlib.h strcmp()
  needs string.h Reviewed-by: Heikki Linnakangas <hlinnaka@iki.fi> Reviewed-by:
  Thomas Munro <thomas.munro@enterprisedb.com>
  http://git.postgresql.org/pg/commitdiff/a5da81359dbddfee49ea7800b43d99c204b4d31f

- Remove unnecessary prototypes.  Prototypes for functions implementing
  V1-callable functions are no longer necessary.  Reviewed-by: Heikki
  Linnakangas <hlinnaka@iki.fi> Reviewed-by: Thomas Munro
  <thomas.munro@enterprisedb.com>
  http://git.postgresql.org/pg/commitdiff/0665023b4435a469e42289d7065c436967a022b6

- Use return instead of exit() in configure.  Using exit() requires stdlib.h,
  which is not included.  Use return instead.  Also add return type for main().
  Reviewed-by: Heikki Linnakangas <hlinnaka@iki.fi> Reviewed-by: Thomas Munro
  <thomas.munro@enterprisedb.com>
  http://git.postgresql.org/pg/commitdiff/1c0cf52b39ca3a9a79661129cff918dc000a55eb

- Fix use of offsetof().  Using offsetof() with a run-time computed argument is
  not allowed in either C or C++.  Apparently, gcc allows it, but g++ doesn't.
  Reviewed-by: Heikki Linnakangas <hlinnaka@iki.fi> Reviewed-by: Thomas Munro
  <thomas.munro@enterprisedb.com>
  http://git.postgresql.org/pg/commitdiff/f1a469c9f1cfeab9f9c7d4a5d3e75892e7b6f60c

- Fix breakage in previous change.
  http://git.postgresql.org/pg/commitdiff/cd03890d0b5d9cbcfca169195d71f1693f84cafa

- Separate enum from struct.  Otherwise the enum symbols are not visible outside
  the struct in C++.  Reviewed-by: Thomas Munro <thomas.munro@enterprisedb.com>
  http://git.postgresql.org/pg/commitdiff/330b48b94b53bcbbc490f952d6798d5ab637721a

- Set log_line_prefix and application name in test drivers.  Before pg_regress
  runs psql, set the application name to the test name.  Similarly, set the
  application name to the test file name in the TAP tests.  Also, set a default
  log_line_prefix that show the application name, as well as the PID and a time
  stamp.  That way, the server log output can be correlated to the test input
  files, making debugging a bit easier.
  http://git.postgresql.org/pg/commitdiff/a4327296df7366ecc657b706a9b5e87aa921311a

Álvaro Herrera pushed:

- Include <sys/select.h> where needed.  <sys/select.h> is required by
  POSIX.1-2001 to get the prototype of select(2), but nearly no systems enforce
  that because older standards let you get away with including some other
  headers.  Recent OpenBSD hacking has removed that frail touch of friendliness,
  however, which broke some compiles; fix all the way back to 9.1 by adding the
  required standard.  Only vacuumdb.c was reported to fail, but it seems easier
  to fix the whole lot in a fell swoop.  Per bug #14334 by Sean Farrell.
  http://git.postgresql.org/pg/commitdiff/51c3e9fade76c12e4aa37bffdf800bbf74fb3fb1

- Silence compiler warnings.  Reported by Peter Eisentraut.  Coding suggested by
  Tom Lane.
  http://git.postgresql.org/pg/commitdiff/b82d5a2c7c9dd1a01041feef2834db27a46fb2d3

Heikki Linnakangas pushed:

- Refactor script execution state machine in pgbench.  The doCustom() function
  had grown into quite a mess. Rewrite it, in a more explicit state machine
  style, for readability.  This also fixes one minor bug: if a script consisted
  entirely of meta commands, doCustom() never returned to the caller, so
  progress reports with the -P option were not printed. I don't want to
  backpatch this refactoring, and the bug is quite insignificant, so only commit
  this to master, and leave the bug unfixed in back-branches.  Review and
  original bug report by Fabien Coelho.  Discussion:
  <alpine.DEB.2.20.1607090850120.3412@sto>
  http://git.postgresql.org/pg/commitdiff/12788ae49e1933f463bc59a6efe46c4a01701b76

- Turn password_encryption GUC into an enum.  This makes the parameter easier to
  extend, to support other password-based authentication protocols than MD5.
  (SCRAM is being worked on.) The GUC still accepts on/off as aliases for "md5"
  and "plain", although we may want to remove those once we actually add support
  for another password hash type.  Michael Paquier, reviewed by David Steele,
  with some further edits by me.  Discussion:
  <CAB7nPqSMXU35g=W9X74HVeQp0uvgJxvYOuA4A-A3M+0wfEBv-w@mail.gmail.com>
  http://git.postgresql.org/pg/commitdiff/babe05bc2b781eb3eb84a18d7010d08277e2e399

- Don't bother to lock bufmgr partitions in pg_buffercache.  That makes the view
  a lot less disruptive to use on a production system.  Without the locks, you
  don't get a consistent snapshot across all buffers, but that's OK. It wasn't a
  very useful guarantee in practice.  Ivan Kartyshov, reviewed by Tomas Vondra
  and Robert Haas.  Discusssion:
  <f9d6cab2-73a7-7a84-55a8-07dcb8516ae5@postgrespro.ru>
  http://git.postgresql.org/pg/commitdiff/6e654546fb61f62cc982d0c8f62241b3b30e7ef8

Robert Haas pushed:

- Fix dangling pointer problem in ReorderBufferSerializeChange.  Commit
  3fe3511d05127cc024b221040db2eeb352e7d716 introduced a new case into this
  function, but neglected to ensure that the "ondisk" pointer got updated after
  a possible reallocation as the code does in other cases.  Stas Kelvich, per
  diagnosis by Konstantin Knizhnik.
  http://git.postgresql.org/pg/commitdiff/308985b0b404a5891a1a629f38cc46c2b2dcb4be

- worker_spi: Call pgstat_report_stat.  Without this, statistics changes
  accumulated by the worker never get reported to the stats collector, which is
  bad.  Julien Rouhaud
  http://git.postgresql.org/pg/commitdiff/4929704acb3b4cdde8e890af977cdacaf1ac2ccb

Stephen Frost pushed:

- Remove superuser checks in pgstattuple.  Now that we track initial privileges
  on extension objects and changes to those permissions, we can drop the
  superuser() checks from the various functions which are part of the
  pgstattuple extension and rely on the GRANT system to control access to those
  functions.  Since a pg_upgrade will preserve the version of the extension
  which existed prior to the upgrade, we can't simply modify the existing
  functions but instead need to create new functions which remove the checks and
  update the SQL-level functions to use the new functions (and to REVOKE EXECUTE
  rights on those functions from PUBLIC).  Thanks to Tom and Andres for adding
  support for extensions to follow update paths (see: 40b449a), allowing this
  patch to be much smaller since no new base version script needed to be
  included.  Approach suggested by Noah.  Reviewed by Michael Paquier.
  http://git.postgresql.org/pg/commitdiff/fd321a1dfd64d30bf1652ea6b39b654304f68ae4

Magnus Hagander pushed:

- Retry opening new segments in pg_xlogdump --follow.  There is a small window
  between when the server closes out the existing segment and the new one is
  created. Put a loop around the open call in this case to make sure we wait for
  the new file to actually appear.
  http://git.postgresql.org/pg/commitdiff/3d39244e6e7374febff59eba338f7a82f696a618

== Pending Patches ==

Takayuki Tsunakawa sent in two revisions of a patch to implement huge_pages on
Windows.

Michaël Paquier and Heikki Linnakangas traded patch flocks to implement SCRAM
password auth.

Pavel Stěhule sent in another revision of a patch to add a \setfileref command
to psql.

Pavel Stěhule sent in another revision of a patch to add xmltable().

Vitaly Burovoy sent in a patch to detect supported SET parameters when
pg_restore is run.

Vinayak Pokale support prepared transactions with foreign servers.

Peter Eisentraut and Michaël Paquier traded patches to fix CRC check handling in
get_controlfile.

Stephen Frost and Jeevan Chalke traded patches to add support for restrictive
RLS policies.

Jesper Pedersen and Peter Eisentraut traded patches to add hash index support
to pageinspect.

Kyotaro HORIGUCHI and Michaël Paquier traded patches to track LSN on lightly
loaded systems to reduce the number of checkpoints.

Tomas Vondra sent in another revision of a patch to add two slab-like memory
allocators.

Fabien COELHO sent in three more revisions of a patch to add more operators and
functions to pgbench.

Fabien COELHO sent in another revision of a patch to enable storing select
results into variables.

Rajkumar Raghuwanshi sent in a patch to add test cases for partition-wise joins
on declarative partitions.

Daniel Gustafsson sent in a patch to make flex/bison checks stricter in Git
trees.

Mark Dilger sent in a patch to stop casting away const in comparators.

Mark Dilger sent in a patch to  stop casting away const in regex.

Mark Dilger sent in a patch to avoid casting away const when using the FileName
typedef.

Takayuki Tsunakawa sent in a patch to allow SIGQUIT to prompt a fast shutdown of
the stats collector without leaving partly-written stats files behind.

Mark Dilger sent in a WIP patch to const-ify xlog_outdesc() and
rm_redo_error_callback().

David Steele and Michaël Paquier traded patches to exclude additional
directories in pg_basebackup.

Amit Langote sent in three more revisions of a patch to implement declarative
partitions.

Michaël Paquier sent in four more revisions of a patch to track wait event for
latches.

Victor Wagner sent in two more revisions of a patch to implement failover at the
libpq connect level.

Mark Dilger sent in a patch to tidy up some handling of pointers in
rangetypes_spgist.c, tsgistidx.c, and tsrank.c.

David Cramer and Heikki Linnakangas traded patches to add support for
multi-dimensional arrays to PL/PythonU.

Mithun Cy sent in another revision of a patch to cache hash index meta pages.

Emre Hasegeli sent in another revision of a patch to fix floating point
comparison inconsistencies in the geometric types.

Ivan Kartyshov sent in another revision of a patch to make pg_buffercache more
efficient with large shared memory.

Etsuro Fujita sent in two more revisions of a patch to push down more full joins
to the PostgreSQL FDW.

David Fetter and Thomas Munro traded patches to add a hook and corresponding
GUCs to disable UPDATEs and DELETEs that lack a WHERE clause.

Michaël Paquier sent in another revision of a patch to rename pg_xlog to pg_wal
and rename pg_clog to pg_transaction.

Heikki Linnakangas sent in two more revisions of a patch to change the way use
memory for larger read buffers in logtape.c rather than pre-reading tuples into
SortTuple slots during merge.

Magnus Hagander sent in two more revisions of a patch to enable pg_basebackup to
stream xlog to tar.

Kyotaro HORIGUCHI sent in another revision of a patch to fix wal level minimal.

Ashutosh Bapat sent in another revision of a patch to fix an issue where
altering foreign table was not invalidating prepare statement execution plans
that it should have.

Jeevan Chalke sent in another revision of a patch to enable pushing aggregates
to a FDW.

Aleksander Alekseev sent in a patch to rename md5Salt to pwsalt.

Corey Huinker sent in a PoC patch to make a copy() set-returning function.

Stephen Frost sent in another revision of a patch to fix some infelicities
between COPY and RLS.

Andres Freund sent in another revision of a patch to add a macro-customizable
hashtable for bitmapscan and aggregation performance.

Daniel Vérité sent in another revision of a patch to fix some pg_dump / copy
bugs with long lines.

Dmitry Dolgov sent in another revision of a patch to add a generic type
subscription.

Emre Hasegeli sent in a patch to add <@, @>, <<@, and @>> operator symbols for
the inet datatype to replace <<=, >>=, <<, and >>.

Christoph Berg sent in a patch to ensure that the default log_line prefix is not
empty.