== PostgreSQL Weekly News - May 18 2014 ==

== PostgreSQL Weekly News - May 18 2014 ==

PostgreSQL 9.4 Beta 1 released.  TEST!
http://www.postgresql.org/developer/beta

PostgreSQL Conference Europe 2014 will be October 21-24 at Hotel
Miguel Angel in Madrid, Spain.  The CfP is open for talks in English
and Spanish.
http://2014.pgconf.eu/

== PostgreSQL Product News ==

pg_catcheck, a catalog integrity checker for PostgreSQL and Postgres
Plus Advanced Server, released.
https://github.com/EnterpriseDB/pg_catcheck

PG Commander 1.3, a GUI made specifically for PostgreSQL on OSX, released.
https://eggerapps.at/pgcommander/

psqlODBC 09.03.0300 released.
http://www.postgresql.org/ftp/odbc/versions/

psycopg2 2.5.3, a Python connector for PostgreSQL, released.
http://initd.org/psycopg/articles/2014/05/13/psycopg-253-released/

== PostgreSQL Jobs for May ==

http://archives.postgresql.org/pgsql-jobs/2014-05/threads.php

== PostgreSQL Local ==

PGCon 2014, the world-wide developer conference for PostgreSQL, will
be in Ottawa, Ontario, Canada May 20-24, 2014.
http://www.pgcon.org/2014/

Char(14) and PGday UK will be held July 8 and 9, 2014.
http://www.char14.info
http://postgresqlusergroup.org.uk/

The sixth PGDay Cubano be held on 13 and 14 October 2014 in Habana.
https://postgresql.uci.cu/?p=380

Postgres Open 2014 will be in Chicago, IL, USA, September 17-19.  The
CfP is open!
http://postgresopen.org/2014/callforpapers/

PostgreSQL Conference Europe 2014 will be held on October 21-24 in
Madrid, Spain, at the Hotel Miguel Angel.
http://2014.pgconf.eu/

== PostgreSQL in the News ==

Planet PostgreSQL: http://planet.postgresql.org/

PostgreSQL Weekly News is brought to you this week by David Fetter

Submit news and announcements by Sunday at 3:00pm Pacific time.
Please send English language ones to david@fetter.org, German language
to pwn@pgug.de, Italian language to pwn@itpug.org.  Spanish language
to pwn@arpug.com.ar.

== Applied Patches ==

Heikki Linnakangas pushed:

- Free PQresult on error in pg_receivexlog.  The leak is fairly small
  and rare, but a leak nevertheless.  Per Coverity report. Backpatch
  to 9.2, where pg_receivexlog was added.  pg_basebackup shares the
  code, but it always exits on error, so there is no real leak.
  http://git.postgresql.org/pg/commitdiff/c890b488063e23f12b5ab01c9a6d254a034fcea2

- Initialize padding bytes in btree_gist varbit support.  The code
  expands a varbit gist leaf key to a node key by copying the bit data
  twice in a varlen datum, as both the lower and upper key. The lower
  key was expanded to INTALIGN size, but the padding bytes were not
  initialized.  That's a problem because when the lower/upper keys are
  compared, the padding bytes are used compared too, when the values
  are otherwise equal. That could lead to incorrect query results.
  REINDEX is advised for any btree_gist indexes on bit or bit varying
  data type, to fix any garbage padding bytes on disk.  Per Valgrind,
  reported by Andres Freund. Backpatch to all supported versions.
  http://git.postgresql.org/pg/commitdiff/540ac7cea919623f691b20892ccc50e5e33b5009

- Fix harmless access to uninitialized memory.  When cache
  invalidations arrive while ri_LoadConstraintInfo() is busy filling a
  new cache entry, InvalidateConstraintCacheCallBack() compares the -
  not yet initialized - oidHashValue field with the to-be-invalidated
  hash value. To fix, check whether the entry is already marked as
  invalid.  Andres Freund
  http://git.postgresql.org/pg/commitdiff/f35aef415aa755c4e99f8c0ef83f9d14dbc48bb4

- Silence warnings about redefining popen on Mingw-w64.  Mingw-w64
  headers map popen/pclose to _popen and _pclose, but we want to use
  our popen wrapper rather than the Mingw-w64. #undef the Mingw's
  version.
  http://git.postgresql.org/pg/commitdiff/a82a17475d8a8a91df5fd1448b0f93772f590ecd

- Misc message style and doc fixes.  Euler Taveira
  http://git.postgresql.org/pg/commitdiff/ff810b4928882bfdc4ebe1ce603c47830aba3132

- Fix race condition in preparing a transaction for two-phase commit.
  To lock a prepared transaction's shared memory entry, we used to
  mark it with the XID of the backend. When the XID was no longer
  active according to the proc array, the entry was implicitly
  considered as not locked anymore. However, when preparing a
  transaction, the backend's proc array entry was cleared before
  transfering the locks (and some other state) to the prepared
  transaction's dummy PGPROC entry, so there was a window where
  another backend could finish the transaction before it was in fact
  fully prepared.  To fix, rewrite the locking mechanism of global
  transaction entries.  Instead of an XID, just have simple
  locked-or-not flag in each entry (we store the locking backend's
  backend id rather than a simple boolean, but that's just for
  debugging purposes). The backend is responsible for explicitly
  unlocking the entry, and to make sure that that happens, install a
  callback to unlock it on abort or process exit.  Backpatch to all
  supported versions.
  http://git.postgresql.org/pg/commitdiff/bb38fb0d43c8d7ff54072bfd8bd63154e536b384

- Handle duplicate XIDs in txid_snapshot.  The proc array can contain
  duplicate XIDs, when a transaction is just being prepared for
  two-phase commit. To cope, remove any duplicates in
  txid_current_snapshot(). Also ignore duplicates in the input
  functions, so that if e.g. you have an old pg_dump file that already
  contains duplicates, it will be accepted.  Report and fix by Jan
  Wieck. Backpatch to all supported versions.
  http://git.postgresql.org/pg/commitdiff/8f9b9590d79fc1fc1ad08b207401acfdbb0bfac7

- Fix a couple of bugs in pg_recvlogical output to stdout.  Don't
  close stdout on SIGHUP. Also, when a SIGHUP is received, close the
  file immediately, rather than only after receiving some more data
  from the server. Rename a variable, to avoid mentally dealing with
  double negatives (not unsynced means synced).
  http://git.postgresql.org/pg/commitdiff/00c26b6a60e7ceed29ddae34b0a69fe945c08f81

- Initialize tsId and dbId fields in WAL record of COMMIT PREPARED.
  Commit dd428c79 added dbId and tsId to the xl_xact_commit struct but
  missed that prepared transaction commits reuse that struct. Fix
  that.  Because those fields were left unitialized, replaying a
  commit prepared WAL record in a hot standby node would fail to
  remove the relcache init file.  That can lead to "could not open
  file" errors on the standby. Relcache init file only needs to be
  removed when a system table/index is rewritten in the transaction
  using two phase commit, so that should be rare in practice. In HEAD,
  the incorrect dbId/tsId values are also used for filtering in
  logical replication code, causing the transaction to always be
  filtered out.  Analysis and fix by Andres Freund. Backpatch to 9.0
  where hot standby was introduced.
  http://git.postgresql.org/pg/commitdiff/07a4a93a0e35a778c77ffbbbc18de29e859e18f0

- Open output file before sleeping in pg_recvlogical.  Let's complain
  about e.g an invalid path or permission problem sooner rather than
  later. Before this patch, we would only try to open the output file
  after receiving the first decoded message from the server.
  http://git.postgresql.org/pg/commitdiff/e7873b74d9c1bb7a325960f52043db6de88e6e66

- Fix thinko in logical decoding of commit-prepared records.  The
  decoding of prepared transaction commits accidentally used the XID
  of the transaction performing the COMMIT PREPARED, not the XID of
  the prepared transaction. Before bb38fb0d43c8d that lead to those
  transactions not being decoded, afterwards to a assertion failure.
  http://git.postgresql.org/pg/commitdiff/03e2b1017c0d360cae0b4d0a8bf5d6924536c834

- Fix test_decoding test case's check that slot has been dropped.
  pg_stat_replication shows connected replication clients. The ddl
  test case never has any replication clients connected, so querying
  pg_stat_replication is pointless. To check that a slot has been
  dropped correctly, query pg_replication_slots instead.  Andres
  Freund
  http://git.postgresql.org/pg/commitdiff/afd0fcbc5d421454e7d55d949cca0721c9c135c3

- Add test case for logical decoding of prepared transactions.  Andres
  Freund
  http://git.postgresql.org/pg/commitdiff/d900e192a3ef2dd73fae4522c35aadce6d72a534

- Update README, we don't do post-recovery cleanup actions anymore.
  transam/README explained how B-tree incomplete splits were tracked
  and fixed after recovery, as an example of handling complex actions
  that need multiple WAL records, but that's not how it works anymore.
  Explain the new paradigm.
  http://git.postgresql.org/pg/commitdiff/a3655dd4a5cee3917a7d1766e07e36013e7e8835

Fujii Masao pushed:

- Fix typo in test_shm_mq document.  Amit Langote
  http://git.postgresql.org/pg/commitdiff/a3632771bb66eec74c7e73c0d7a5f8cdd6719be2

Tom Lane pushed:

- Ignore config.pl and buildenv.pl in src/tools/msvc.  config.pl and
  buildenv.pl can be used to customize build settings when using MSVC.
  They should never get committed into the common source tree.
  Back-patch to 9.0; it looks like the rules were different in 8.4.
  Michael Paquier
  http://git.postgresql.org/pg/commitdiff/73011f35eca7f4a3973289d5813648abbc1074a0

- Be more wary in choice of timezone names to test make_timestamptz
  with.  America/Metlakatla hasn't been in the IANA database all that
  long, so some installations might not have it.  It does seem
  worthwhile to test with a fractional-minute GMT offset, but we can
  get that from almost any pre-1900 date; I chose Europe/Paris, whose
  LMT offset from Greenwich should be pretty darn well established.
  Also, assuming that Mars/Mons_Olympus will never be in the IANA
  database seems less than future-proof, so let's use a more fanciful
  location for the bad-zone-name check.  Per complaint from Christoph
  Berg.
  http://git.postgresql.org/pg/commitdiff/66b737cd9a72e10df4de1867ae7b675bc48d478c

- Make initdb throw error for bad locale values.  Historically we've
  printed a complaint for a bad locale setting, but then fallen back
  to the environment default.  Per discussion, this is not such a
  great idea, because rectifying an erroneous locale choice
  post-initdb (perhaps long after data has been loaded) could be
  enormously expensive.  Better to complain and give the user a chance
  to double-check things.  The behavior was particularly bad if the
  bad setting came from environment variables rather than a bogus
  command-line switch: in that case not only was there a fallback to
  C/SQL_ASCII, but the printed complaint was quite unhelpful.  It's
  hard to be entirely sure what variables setlocale looked at, but we
  can at least give a hint where the problem might be.  Per a
  complaint from Tomas Vondra.
  http://git.postgresql.org/pg/commitdiff/31a263237fa3eb6f36d58335fbdb82bab3136db3

- Code review for recent changes in relcache.c.  rd_replidindex should
  be managed the same as rd_oidindex, and rd_keyattr and rd_idattr
  should be managed like rd_indexattr.  Omissions in this area meant
  that the bitmapsets computed for rd_keyattr and rd_idattr would be
  leaked during any relcache flush, resulting in a slow but permanent
  leak in CacheMemoryContext.  There was also a tiny probability of
  relcache entry corruption if we ran out of memory at just the wrong
  point in RelationGetIndexAttrBitmap.  Otherwise, the fields were not
  zeroed where expected, which would not bother the code any AFAICS
  but could greatly confuse anyone examining the relcache entry while
  debugging.  Also, create an API function RelationGetReplicaIndex
  rather than letting non-relcache code be intimate with the
  mechanisms underlying caching of that value (we won't even mention
  the memory leak there).  Also, fix a relcache flush hazard
  identified by Andres Freund: RelationGetIndexAttrBitmap must not
  assume that rd_replidindex stays valid across index_open.  The
  aspects of this involving rd_keyattr date back to 9.3, so back-patch
  those changes.
  http://git.postgresql.org/pg/commitdiff/b23b0f5588d2e2f15edff66e072e339a8c9616a7

- Fix version check for pg_upgrade line type crosscheck.  Per
  buildfarm.
  http://git.postgresql.org/pg/commitdiff/4456763c78d2a94c42e67ce788fbe94a07aa12dc

- In initdb, ensure stdout/stderr buffering behavior is what we
  expect.  Since this program may print to either stdout or stderr,
  the relative ordering of its messages depends on the buffering
  behavior of those files.  Force stdout to be line-buffered and
  stderr to be unbuffered, ensuring that the behavior will match
  standard Unix interactive behavior, even when stdout and stderr are
  rerouted to a file.  Per complaint from Tomas Vondra.  The
  particular case he pointed out is new in HEAD, but issues of the
  same sort could arise in any branch with other error messages, so
  back-patch to all branches.  I'm unsure whether we might not want to
  do this in other client programs as well.  For the moment, just fix
  initdb.
  http://git.postgresql.org/pg/commitdiff/2dc4f011fd61501cce507be78c39a2677690d44b

- Fix unportable setvbuf() usage in initdb.  In yesterday's commit
  2dc4f011fd61501cce507be78c39a2677690d44b, I tried to force buffering
  of stdout/stderr in initdb to be what it is by default when the
  program is run interactively on Unix (since that's how most manual
  testing is done).  This tripped over the fact that Windows doesn't
  support _IOLBF mode.  We dealt with that a long time ago in
  syslogger.c by falling back to unbuffered mode on Windows.  Export
  that solution in port.h and use it in initdb.  Back-patch to 8.4,
  like the previous commit.
  http://git.postgresql.org/pg/commitdiff/f62d417825d6a8e26768a130a5f4c9e454d9dbe4

- Fix valgrind warning for btree_gist indexes on macaddr.  The macaddr
  opclass stores two macaddr structs (each of size 6) in an index
  column that's declared as being of type gbtreekey16, ie 16 bytes.
  In the original coding this led to passing a palloc'd value of size
  12 to the index insertion code, so that data would be fetched past
  the end of the allocated value during index tuple construction.
  This makes valgrind unhappy.  In principle it could result in a
  SIGSEGV, though with the current implementation of palloc there's no
  risk since the 12-byte request size would be rounded up to 16 bytes
  anyway.  To fix, add a field to struct gbtree_ninfo showing the
  declared size of the index datums, and use that in the palloc
  requests; and use palloc0 to be sure that any wasted bytes are
  cleanly initialized.  Per report from Andres Freund.  No back-patch
  since there's no current risk of a real problem.
  http://git.postgresql.org/pg/commitdiff/82bbb60c30dbff0633da34387ccab58d843379b5

- Fix a second cause of undersized pallocs for btree_gist indexes on
  macaddr.  gbt_macad_union also allocated 12-byte structs where we
  really need 16.  Per report from Andres Freund.  No back-patch since
  there's no current risk of a real problem.
  http://git.postgresql.org/pg/commitdiff/39586bc1e971c136c4487b26eb7334dbbe23b102

- Suppress some more valgrind whining about btree_gist.  A couple of
  functions didn't bother to zero out pad bytes in datums that would
  ultimately go to disk.  Harmless, but valgrind doesn't know that.
  http://git.postgresql.org/pg/commitdiff/af215d8190e6ab170c02c24afd1be81f5a147481

- Make sure chr(int) can't create invalid UTF8 sequences.  Several
  years ago we changed chr(int) so that if the database encoding is
  UTF8, it would interpret its argument as a Unicode code point and
  expand it into the appropriate multibyte sequence.  However, we
  weren't sufficiently careful about checking validity of the input.
  According to RFC3629, UTF8 disallows code points above U+10FFFF
  (note that the predecessor standard RFC2279 was more liberal).
  Also, both versions of the UTF8 spec agree that Unicode
  surrogate-pair codes should never appear in UTF8.  Because our
  encoding validity checks follow RFC3629, our failure to enforce
  these restrictions in chr() means it could be used to produce text
  strings that will be rejected when the database is dumped and
  reloaded.  To ensure consistency with the input functions, let's
  actually apply pg_utf8_islegal() to the proposed output of chr().
  Per discussion, this seems like too much of a behavioral change to
  back-patch, but it's not too late to squeeze it into 9.4.
  http://git.postgresql.org/pg/commitdiff/7894ac5004d5ec158f46296540d3cf49386d30e4

- Fix unaligned accesses in DecodeUpdate().  The xl_heap_header_len
  structures in an XLOG_HEAP_UPDATE record aren't necessarily aligned
  adequately.  The regular replay function for these records is aware
  of that, but decode.c didn't get the memo.  I'm not sure why the
  buildfarm failed to catch this; the test_decoding test certainly
  blows up real good on my old HPPA box.  Also, I'm pretty sure that
  the address arithmetic was wrong for the case of
  XLOG_HEAP_CONTAINS_OLD and not XLOG_HEAP_CONTAINS_NEW_TUPLE, though
  this apparently can't happen when logical decoding is active.
  http://git.postgresql.org/pg/commitdiff/6c42b2b10af3f717030966b9f05867f3e065becc

- Fix a bunch of functions that were declared static then defined
  not-static.  Per testing with a compiler that whines about this.
  http://git.postgresql.org/pg/commitdiff/c1907f0cc49e38df9853b7547c9afce5204e4784

- Make fallback implementation of pg_memory_barrier() work.  The
  fallback implementation involves acquiring and releasing a spinlock
  variable that is otherwise unreferenced --- not even to the extent
  of initializing it.  This accidentally fails to fail on platforms
  where spinlocks should be initialized to zeroes, but elsewhere it
  results in a "stuck spinlock" failure during startup.  I griped
  about this last July, and put in a hack that worked for gcc on HPPA,
  but didn't get around to fixing the general case.  Per the
  discussion back then, the best thing to do seems to be to initialize
  dummy_spinlock in main.c.
  http://git.postgresql.org/pg/commitdiff/44cd47c1d49655c5dd9648bde8e267617c3735b4

- Fix two ancient memory-leak bugs in relcache.c.
  RelationCacheInsert() ignored the possibility that
  hash_search(HASH_ENTER) might find a hashtable entry already present
  for the same OID.  However, that can in fact occur during recursive
  relcache load scenarios.  When it did happen, we overwrote the
  pointer to the pre-existing Relation, causing a session-lifespan
  leakage of that entire structure.  As far as is known, the
  pre-existing Relation would always have reference count zero by the
  time we arrive back at the outer insertion, so add code that deletes
  the pre-existing Relation if so.  If by some chance its refcount is
  positive, elog a WARNING and allow the pre-existing Relation to be
  leaked as before.  Also, AttrDefaultFetch() was sloppy about leaking
  the cstring form of the pg_attrdef.adbin value it's copying into the
  relcache structure.  This is only a query-lifespan leakage, and
  normally not very significant, but it adds up during CLOBBER_CACHE
  testing.  These bugs are of very ancient vintage, but I'll refrain
  from back-patching since there's no evidence that these leaks amount
  to anything in ordinary usage.
  http://git.postgresql.org/pg/commitdiff/078b2ed291c758e7125d72c3a235f128d40a232b

- Ooops, I broke initdb with that last patch.  That's what I get for
  not fully retesting the final version of the patch.  The
  replace_allowed cross-check needs an additional special case for
  bootstrapping.
  http://git.postgresql.org/pg/commitdiff/0c19aaba22c4d904f7259ef11f7d6db7b5dd36a0

Bruce Momjian pushed:

- doc: 9.4 release note adjustments.  Report by Nicolas Barbier,
  Tatsuo Ishii, MauMau
  http://git.postgresql.org/pg/commitdiff/6de9a6c015d543c41933171cd6cf5d357597f19f

- docs:  use structfield instead of structname in 9.4 release notes.
  Where appropriate
  http://git.postgresql.org/pg/commitdiff/9af8bd51a10edaf454c6b5246d79eded73528fd2

- docs:  9.4 release notes adjustments.  Patch by Andres Freund,
  slight adjustments by me
  http://git.postgresql.org/pg/commitdiff/062f53518927f9bfe1820578ce79d3180b1aa2ca

- doc:  auto-updatable view adjustments for 9.4 release notes.  Report
  by Dean Rasheed
  http://git.postgresql.org/pg/commitdiff/e513087f6401a64b0d9f0b20cd2a51a7938ac645

- docs:  mention windows quoting change in 9.4 release notes.  Report
  by Heikki Linnakangas
  http://git.postgresql.org/pg/commitdiff/eb6144bb447ecbb036bfa08da85940cd50d49410

- doc: fix typo in 9.4 release note comments.  Patch by Sergey
  Muraviov
  http://git.postgresql.org/pg/commitdiff/eefd90c4a95449f02b568d5370cfe6bcd1bf13e9

- doc:  fix 9.4 release notes typo.  Report by Dean Rasheed
  http://git.postgresql.org/pg/commitdiff/ac53295d667e7727d7b70ddf11d82c067870501f

- pg_upgrade:  error out on 'line' data type usage.  The data type
  internal format changed in 9.4.  Also mention this in the 9.4
  release notes.
  http://git.postgresql.org/pg/commitdiff/bb797b640449d67f722d27660c718b68a692f8f9

- pg_update:  slight major version adjustment
  http://git.postgresql.org/pg/commitdiff/2e6602dfc25d1f8e8a2e1df48c96440b02aa19c1

Noah Misch pushed:

- Add Valgrind suppression for reorderbuffer padding bytes.  Andres
  Freund
  http://git.postgresql.org/pg/commitdiff/5a90ac29c44f0dabf1e3a7b0443ba772cf206e2e

Peter Eisentraut pushed:

- pg_ctl: Write error messages to stderr
  http://git.postgresql.org/pg/commitdiff/c424c049189f758920b1a3bb961bc05c93a900df

- doc: Clarify what files pg_basebackup omits from data directory
  http://git.postgresql.org/pg/commitdiff/4773c70aa4c938c367ee179f7301f70a4efabc0e

- Fix whitespace
  http://git.postgresql.org/pg/commitdiff/2f8e68bd7ea3523977567369056c4b3e67e44fdc

== Rejected Patches (for now) ==

No one was disappointed this week :-)

== Pending Patches ==

Kaigai Kouhei sent in two revisions of a patch to fix an error in the
regression tests for SEPgsql.

Thomas Munro and Craig Ringer traded patches to implement SKIP LOCKED
DATA.

Sergey Muraviov sent in three more revisions of a patch to fix psql's
wrapping in extended mode with the default pager.

Amit Kapila sent in two revisions of a patch to help scale shared
buffer eviction by reducing contention around BufFreelistLock.

Emre Hasegeli sent in another revision of a patch to add a selectivity
estimation for inet operators.

Jeff Janes sent in a patch to fix pg_dump warnings in the MinGW build.

Jeff Janes sent in a patch to fix an issue with autovacuum scheduling
starvation and frenzy.

Craig Ringer sent in a patch to fix an issue where dynamic background
workers with BGW_NEVER_RESTART worker restarted on FatalError.

Steve Singer and Andres Freund traded patches to fix an issue where
CREATE REPLICATION SLOT would fail on a timeout.

Andres Freund sent in two revisions of a patch to fix an issue where
%d in log_line_prefix doesn't work for bg/autovacuum workers.

Heikki Linnakangas sent in another revision of a patch to change the
WAL format with corresponding changes to the API.

Marko Kreen sent in a patch to clarify the crypto documentation for
SSL.

Heikki Linnakangas sent in a patch to fix a checksum error in recovery
with btree index.

David Rowley sent in another revision of a patch to allow join
removals for more join types.

Fabrízio de Royes Mello sent in a patch to fix an issue where psql's
\db+ does not have a size column.

Fabrízio de Royes Mello sent in a patch to fix a wrong name in the
documentation of the pg_stat_replication view.