Обсуждение: pg_control is missing a field for LOBLKSIZE
I just chanced to notice that if someone were to change the value for
LOBLKSIZE and recompile, there'd be nothing to stop him from starting
that postmaster against an existing database, even though it would
completely misinterpret and mangle any data in pg_largeobject.
I think there ought to be a guard for that, for exactly the same reasons
that we check TOAST_MAX_CHUNK_SIZE: correct interpretation of on-disk
data requires that this value match the original database configuration.
Obviously it's too late to do anything about this in existing branches,
but I propose to add a field to pg_control after we branch off 9.4.
regards, tom lane
On 06/04/2014 10:03 AM, Tom Lane wrote: > I just chanced to notice that if someone were to change the value for > LOBLKSIZE and recompile, there'd be nothing to stop him from starting > that postmaster against an existing database, even though it would > completely misinterpret and mangle any data in pg_largeobject. > > I think there ought to be a guard for that, for exactly the same reasons > that we check TOAST_MAX_CHUNK_SIZE: correct interpretation of on-disk > data requires that this value match the original database configuration. > > Obviously it's too late to do anything about this in existing branches, > but I propose to add a field to pg_control after we branch off 9.4. > > If we did an initdb-requiring change for 9.4 could we piggy-back this onto it? cheers andrew
On 2014-06-04 10:25:07 -0400, Andrew Dunstan wrote: > If we did an initdb-requiring change for 9.4 could we piggy-back this onto > it? Do you know of a problem requiring that? Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training & Services
* Andrew Dunstan (andrew@dunslane.net) wrote:
> On 06/04/2014 10:03 AM, Tom Lane wrote:
> >I just chanced to notice that if someone were to change the value for
> >LOBLKSIZE and recompile, there'd be nothing to stop him from starting
> >that postmaster against an existing database, even though it would
> >completely misinterpret and mangle any data in pg_largeobject.
> >
> >I think there ought to be a guard for that, for exactly the same reasons
> >that we check TOAST_MAX_CHUNK_SIZE: correct interpretation of on-disk
> >data requires that this value match the original database configuration.
> >
> >Obviously it's too late to do anything about this in existing branches,
> >but I propose to add a field to pg_control after we branch off 9.4.
> >
> >
>
> If we did an initdb-requiring change for 9.4 could we piggy-back
> this onto it?
I was thinking more-or-less the same thing...
Then again, I've never heard of a field complaint regarding this, so
pehraps it's not worth it.
Thanks,
Stephen
On 06/04/2014 10:27 AM, Andres Freund wrote: > On 2014-06-04 10:25:07 -0400, Andrew Dunstan wrote: >> If we did an initdb-requiring change for 9.4 could we piggy-back this onto >> it? > Do you know of a problem requiring that? > No, just thinking ahead. cheers andrew
Stephen Frost <sfrost@snowman.net> writes:
> * Andrew Dunstan (andrew@dunslane.net) wrote:
>> On 06/04/2014 10:03 AM, Tom Lane wrote:
>>> I just chanced to notice that if someone were to change the value for
>>> LOBLKSIZE and recompile, there'd be nothing to stop him from starting
>>> that postmaster against an existing database, even though it would
>>> completely misinterpret and mangle any data in pg_largeobject.
> Then again, I've never heard of a field complaint regarding this, so
> pehraps it's not worth it.
I've not heard one either, but there was just somebody asking in
pgsql-general about changing LOBLKSIZE, so he's going to be at risk.
That's not a big enough sample size to make me panic about getting a
hasty fix into 9.4, but I do think we should fix this going forward.
regards, tom lane
* Tom Lane (tgl@sss.pgh.pa.us) wrote: > Stephen Frost <sfrost@snowman.net> writes: > > * Andrew Dunstan (andrew@dunslane.net) wrote: > >> On 06/04/2014 10:03 AM, Tom Lane wrote: > >>> I just chanced to notice that if someone were to change the value for > >>> LOBLKSIZE and recompile, there'd be nothing to stop him from starting > >>> that postmaster against an existing database, even though it would > >>> completely misinterpret and mangle any data in pg_largeobject. > > > Then again, I've never heard of a field complaint regarding this, so > > pehraps it's not worth it. > > I've not heard one either, but there was just somebody asking in > pgsql-general about changing LOBLKSIZE, so he's going to be at risk. > That's not a big enough sample size to make me panic about getting a > hasty fix into 9.4, but I do think we should fix this going forward. Agreed. Thanks, Stephen
On 2014-06-04 10:03:09 -0400, Tom Lane wrote: > I just chanced to notice that if someone were to change the value for > LOBLKSIZE and recompile, there'd be nothing to stop him from starting > that postmaster against an existing database, even though it would > completely misinterpret and mangle any data in pg_largeobject. > > I think there ought to be a guard for that, for exactly the same reasons > that we check TOAST_MAX_CHUNK_SIZE: correct interpretation of on-disk > data requires that this value match the original database configuration. > > Obviously it's too late to do anything about this in existing branches, > but I propose to add a field to pg_control after we branch off 9.4. Btw, I had wondered before if we shouldn't also add sizeof(long) to pg_control to catch cases where a database is copied between a LLP64 (64bit windows) and an LP64 (nearly every other 64bit system) system. I have my doubts that we're completely clean about the size difference. Not to speak of extension datatypes. Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/PostgreSQL Development, 24x7 Support, Training & Services
Andres Freund <andres@2ndquadrant.com> writes:
> Btw, I had wondered before if we shouldn't also add sizeof(long) to
> pg_control to catch cases where a database is copied between a LLP64
> (64bit windows) and an LP64 (nearly every other 64bit system) system. I
> have my doubts that we're completely clean about the size
> difference. Not to speak of extension datatypes.
I don't believe that this is necessary. It's certainly true that some
in-memory structures will be laid out differently, but not on-disk.
regards, tom lane
Stephen Frost <sfrost@snowman.net> writes:
> * Tom Lane (tgl@sss.pgh.pa.us) wrote:
>> I've not heard one either, but there was just somebody asking in
>> pgsql-general about changing LOBLKSIZE, so he's going to be at risk.
>> That's not a big enough sample size to make me panic about getting a
>> hasty fix into 9.4, but I do think we should fix this going forward.
> Agreed.
BTW, just comparing the handling of TOAST_MAX_CHUNK_SIZE and LOBLKSIZE,
I noticed that the tuptoaster.c functions are reasonably paranoid about
checking that toast chunks are the expected size, but the large object
functions are not: the latter have either no check at all, or just an
Assert that the size is not more than expected. So we could provide at
least a partial guard against a wrong LOBLKSIZE configuration by making
all the large-object functions throw elog(ERROR) if the length of a LO
chunk is more than LOBLKSIZE. Unfortunately, length *less* than LOBLKSIZE
is an expected case, so this would only help in one direction. Still,
it'd be an easy and back-patchable change that would provide at least some
defense, so I'm thinking of doing it.
regards, tom lane
On Wed, Jun 4, 2014 at 1:21 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote: > Stephen Frost <sfrost@snowman.net> writes: >> * Tom Lane (tgl@sss.pgh.pa.us) wrote: >>> I've not heard one either, but there was just somebody asking in >>> pgsql-general about changing LOBLKSIZE, so he's going to be at risk. >>> That's not a big enough sample size to make me panic about getting a >>> hasty fix into 9.4, but I do think we should fix this going forward. > >> Agreed. > > BTW, just comparing the handling of TOAST_MAX_CHUNK_SIZE and LOBLKSIZE, > I noticed that the tuptoaster.c functions are reasonably paranoid about > checking that toast chunks are the expected size, but the large object > functions are not: the latter have either no check at all, or just an > Assert that the size is not more than expected. So we could provide at > least a partial guard against a wrong LOBLKSIZE configuration by making > all the large-object functions throw elog(ERROR) if the length of a LO > chunk is more than LOBLKSIZE. Unfortunately, length *less* than LOBLKSIZE > is an expected case, so this would only help in one direction. Still, > it'd be an easy and back-patchable change that would provide at least some > defense, so I'm thinking of doing it. This seems like a pretty weak argument for adding run-time overhead. Maybe it can be justified on the grounds that it would catch corrupted TOAST data, but I've never heard of anyone changing LOBLKSIZE and see no reason to get agitated about it. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
Robert Haas <robertmhaas@gmail.com> writes: > On Wed, Jun 4, 2014 at 1:21 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote: >> BTW, just comparing the handling of TOAST_MAX_CHUNK_SIZE and LOBLKSIZE, >> I noticed that the tuptoaster.c functions are reasonably paranoid about >> checking that toast chunks are the expected size, but the large object >> functions are not: the latter have either no check at all, or just an >> Assert that the size is not more than expected. So we could provide at >> least a partial guard against a wrong LOBLKSIZE configuration by making >> all the large-object functions throw elog(ERROR) if the length of a LO >> chunk is more than LOBLKSIZE. Unfortunately, length *less* than LOBLKSIZE >> is an expected case, so this would only help in one direction. Still, >> it'd be an easy and back-patchable change that would provide at least some >> defense, so I'm thinking of doing it. > This seems like a pretty weak argument for adding run-time overhead. > Maybe it can be justified on the grounds that it would catch corrupted > TOAST data, but I've never heard of anyone changing LOBLKSIZE and see > no reason to get agitated about it. One if-test per fetched tuple hardly seems likely to add measurable overhead. As for "never heard of", see today's thread in pgsql-general: http://www.postgresql.org/message-id/flat/CAGou9Mg=9qPYTdh18NDO3LTJtwQN8uRdTwABfkcyMRUt6D_fJw@mail.gmail.com There was a similar gripe a few months ago: http://www.postgresql.org/message-id/CACg6vWXy_84ShCCXzNCsz9xLfWnx5sZvQRU6aNcrR0c3XW1Bxg@mail.gmail.com There are at least two places in inv_api.c where we have "Assert(pagelen <= LOBLKSIZE)" that is protecting a subsequent memcpy into a local variable of size LOBLKSIZE, so that the only thing standing between us and a stack-smash security issue that's trivially exploitable in production builds is that on-disk data conforms to our expectation about LOBLKSIZE. I think it's definitely worth promoting these checks to regular runtime-if-test-and-elog. regards, tom lane
* Tom Lane (tgl@sss.pgh.pa.us) wrote:
> There are at least two places in inv_api.c where we have
> "Assert(pagelen <= LOBLKSIZE)" that is protecting a subsequent memcpy
> into a local variable of size LOBLKSIZE, so that the only thing standing
> between us and a stack-smash security issue that's trivially exploitable
> in production builds is that on-disk data conforms to our expectation
> about LOBLKSIZE. I think it's definitely worth promoting these checks
> to regular runtime-if-test-and-elog.
Agreed. Promoting that to a run-time check seems well worth it to me.
Thanks,
Stephen
Stephen Frost <sfrost@snowman.net> writes:
> * Tom Lane (tgl@sss.pgh.pa.us) wrote:
>> There are at least two places in inv_api.c where we have
>> "Assert(pagelen <= LOBLKSIZE)" that is protecting a subsequent memcpy
>> into a local variable of size LOBLKSIZE, so that the only thing standing
>> between us and a stack-smash security issue that's trivially exploitable
>> in production builds is that on-disk data conforms to our expectation
>> about LOBLKSIZE. I think it's definitely worth promoting these checks
>> to regular runtime-if-test-and-elog.
> Agreed. Promoting that to a run-time check seems well worth it to me.
Here's a draft patch for this. Barring objections I'll commit the whole
thing to HEAD, and the inv_api.c changes to the back branches as well.
regards, tom lane
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index d675560..a61878e 100644
*** a/src/backend/access/transam/xlog.c
--- b/src/backend/access/transam/xlog.c
***************
*** 49,54 ****
--- 49,55 ----
#include "storage/bufmgr.h"
#include "storage/fd.h"
#include "storage/ipc.h"
+ #include "storage/large_object.h"
#include "storage/latch.h"
#include "storage/pmsignal.h"
#include "storage/predicate.h"
*************** WriteControlFile(void)
*** 4352,4357 ****
--- 4353,4359 ----
ControlFile->indexMaxKeys = INDEX_MAX_KEYS;
ControlFile->toast_max_chunk_size = TOAST_MAX_CHUNK_SIZE;
+ ControlFile->loblksize = LOBLKSIZE;
#ifdef HAVE_INT64_TIMESTAMP
ControlFile->enableIntTimes = true;
*************** ReadControlFile(void)
*** 4545,4550 ****
--- 4547,4559 ----
" but the server was compiled with TOAST_MAX_CHUNK_SIZE %d.",
ControlFile->toast_max_chunk_size, (int) TOAST_MAX_CHUNK_SIZE),
errhint("It looks like you need to recompile or initdb.")));
+ if (ControlFile->loblksize != LOBLKSIZE)
+ ereport(FATAL,
+ (errmsg("database files are incompatible with server"),
+ errdetail("The database cluster was initialized with LOBLKSIZE %d,"
+ " but the server was compiled with LOBLKSIZE %d.",
+ ControlFile->loblksize, (int) LOBLKSIZE),
+ errhint("It looks like you need to recompile or initdb.")));
#ifdef HAVE_INT64_TIMESTAMP
if (ControlFile->enableIntTimes != true)
diff --git a/src/backend/storage/large_object/inv_api.c b/src/backend/storage/large_object/inv_api.c
index 57ec1c2..0918142 100644
*** a/src/backend/storage/large_object/inv_api.c
--- b/src/backend/storage/large_object/inv_api.c
*************** myLargeObjectExists(Oid loid, Snapshot s
*** 173,185 ****
}
! static int32
! getbytealen(bytea *data)
{
! Assert(!VARATT_IS_EXTENDED(data));
! if (VARSIZE(data) < VARHDRSZ)
! elog(ERROR, "invalid VARSIZE(data)");
! return (VARSIZE(data) - VARHDRSZ);
}
--- 173,210 ----
}
! /*
! * Extract data field from a pg_largeobject tuple, detoasting if needed
! * and verifying that the length is sane. Returns data pointer (a bytea *),
! * data length, and an indication of whether to pfree the data pointer.
! */
! static void
! getdatafield(Form_pg_largeobject tuple,
! bytea **pdatafield,
! int *plen,
! bool *pfreeit)
{
! bytea *datafield;
! int len;
! bool freeit;
!
! datafield = &(tuple->data); /* see note at top of file */
! freeit = false;
! if (VARATT_IS_EXTENDED(datafield))
! {
! datafield = (bytea *)
! heap_tuple_untoast_attr((struct varlena *) datafield);
! freeit = true;
! }
! len = VARSIZE(datafield) - VARHDRSZ;
! if (len < 0 || len > LOBLKSIZE)
! ereport(ERROR,
! (errcode(ERRCODE_DATA_CORRUPTED),
! errmsg("pg_largeobject entry for OID %u, page %d has invalid data field size %d",
! tuple->loid, tuple->pageno, len)));
! *pdatafield = datafield;
! *plen = len;
! *pfreeit = freeit;
}
*************** inv_getsize(LargeObjectDesc *obj_desc)
*** 366,385 ****
{
Form_pg_largeobject data;
bytea *datafield;
bool pfreeit;
if (HeapTupleHasNulls(tuple)) /* paranoia */
elog(ERROR, "null field found in pg_largeobject");
data = (Form_pg_largeobject) GETSTRUCT(tuple);
! datafield = &(data->data); /* see note at top of file */
! pfreeit = false;
! if (VARATT_IS_EXTENDED(datafield))
! {
! datafield = (bytea *)
! heap_tuple_untoast_attr((struct varlena *) datafield);
! pfreeit = true;
! }
! lastbyte = (uint64) data->pageno * LOBLKSIZE + getbytealen(datafield);
if (pfreeit)
pfree(datafield);
}
--- 391,404 ----
{
Form_pg_largeobject data;
bytea *datafield;
+ int len;
bool pfreeit;
if (HeapTupleHasNulls(tuple)) /* paranoia */
elog(ERROR, "null field found in pg_largeobject");
data = (Form_pg_largeobject) GETSTRUCT(tuple);
! getdatafield(data, &datafield, &len, &pfreeit);
! lastbyte = (uint64) data->pageno * LOBLKSIZE + len;
if (pfreeit)
pfree(datafield);
}
*************** inv_read(LargeObjectDesc *obj_desc, char
*** 506,520 ****
off = (int) (obj_desc->offset - pageoff);
Assert(off >= 0 && off < LOBLKSIZE);
! datafield = &(data->data); /* see note at top of file */
! pfreeit = false;
! if (VARATT_IS_EXTENDED(datafield))
! {
! datafield = (bytea *)
! heap_tuple_untoast_attr((struct varlena *) datafield);
! pfreeit = true;
! }
! len = getbytealen(datafield);
if (len > off)
{
n = len - off;
--- 525,531 ----
off = (int) (obj_desc->offset - pageoff);
Assert(off >= 0 && off < LOBLKSIZE);
! getdatafield(data, &datafield, &len, &pfreeit);
if (len > off)
{
n = len - off;
*************** inv_write(LargeObjectDesc *obj_desc, con
*** 630,645 ****
*
* First, load old data into workbuf
*/
! datafield = &(olddata->data); /* see note at top of file */
! pfreeit = false;
! if (VARATT_IS_EXTENDED(datafield))
! {
! datafield = (bytea *)
! heap_tuple_untoast_attr((struct varlena *) datafield);
! pfreeit = true;
! }
! len = getbytealen(datafield);
! Assert(len <= LOBLKSIZE);
memcpy(workb, VARDATA(datafield), len);
if (pfreeit)
pfree(datafield);
--- 641,647 ----
*
* First, load old data into workbuf
*/
! getdatafield(olddata, &datafield, &len, &pfreeit);
memcpy(workb, VARDATA(datafield), len);
if (pfreeit)
pfree(datafield);
*************** inv_truncate(LargeObjectDesc *obj_desc,
*** 815,833 ****
if (olddata != NULL && olddata->pageno == pageno)
{
/* First, load old data into workbuf */
! bytea *datafield = &(olddata->data); /* see note at top of
! * file */
! bool pfreeit = false;
int pagelen;
! if (VARATT_IS_EXTENDED(datafield))
! {
! datafield = (bytea *)
! heap_tuple_untoast_attr((struct varlena *) datafield);
! pfreeit = true;
! }
! pagelen = getbytealen(datafield);
! Assert(pagelen <= LOBLKSIZE);
memcpy(workb, VARDATA(datafield), pagelen);
if (pfreeit)
pfree(datafield);
--- 817,827 ----
if (olddata != NULL && olddata->pageno == pageno)
{
/* First, load old data into workbuf */
! bytea *datafield;
int pagelen;
+ bool pfreeit;
! getdatafield(olddata, &datafield, &pagelen, &pfreeit);
memcpy(workb, VARDATA(datafield), pagelen);
if (pfreeit)
pfree(datafield);
diff --git a/src/bin/pg_controldata/pg_controldata.c b/src/bin/pg_controldata/pg_controldata.c
index 24e2116..f815024 100644
*** a/src/bin/pg_controldata/pg_controldata.c
--- b/src/bin/pg_controldata/pg_controldata.c
*************** main(int argc, char *argv[])
*** 287,292 ****
--- 287,294 ----
ControlFile.indexMaxKeys);
printf(_("Maximum size of a TOAST chunk: %u\n"),
ControlFile.toast_max_chunk_size);
+ printf(_("Size of a large-object chunk: %u\n"),
+ ControlFile.loblksize);
printf(_("Date/time type storage: %s\n"),
(ControlFile.enableIntTimes ? _("64-bit integers") : _("floating-point numbers")));
printf(_("Float4 argument passing: %s\n"),
diff --git a/src/bin/pg_resetxlog/pg_resetxlog.c b/src/bin/pg_resetxlog/pg_resetxlog.c
index d11280e..915a1ed 100644
*** a/src/bin/pg_resetxlog/pg_resetxlog.c
--- b/src/bin/pg_resetxlog/pg_resetxlog.c
***************
*** 52,57 ****
--- 52,58 ----
#include "catalog/catversion.h"
#include "catalog/pg_control.h"
#include "common/fe_memutils.h"
+ #include "storage/large_object.h"
#include "pg_getopt.h"
*************** GuessControlValues(void)
*** 536,541 ****
--- 537,543 ----
ControlFile.nameDataLen = NAMEDATALEN;
ControlFile.indexMaxKeys = INDEX_MAX_KEYS;
ControlFile.toast_max_chunk_size = TOAST_MAX_CHUNK_SIZE;
+ ControlFile.loblksize = LOBLKSIZE;
#ifdef HAVE_INT64_TIMESTAMP
ControlFile.enableIntTimes = true;
#else
*************** PrintControlValues(bool guessed)
*** 620,625 ****
--- 622,629 ----
ControlFile.indexMaxKeys);
printf(_("Maximum size of a TOAST chunk: %u\n"),
ControlFile.toast_max_chunk_size);
+ printf(_("Size of a large-object chunk: %u\n"),
+ ControlFile.loblksize);
printf(_("Date/time type storage: %s\n"),
(ControlFile.enableIntTimes ? _("64-bit integers") : _("floating-point numbers")));
printf(_("Float4 argument passing: %s\n"),
diff --git a/src/include/catalog/pg_control.h b/src/include/catalog/pg_control.h
index 4579eb6..ba79d25 100644
*** a/src/include/catalog/pg_control.h
--- b/src/include/catalog/pg_control.h
***************
*** 21,27 ****
/* Version identifier for this pg_control format */
! #define PG_CONTROL_VERSION 941
/*
* Body of CheckPoint XLOG records. This is declared here because we keep
--- 21,27 ----
/* Version identifier for this pg_control format */
! #define PG_CONTROL_VERSION 942
/*
* Body of CheckPoint XLOG records. This is declared here because we keep
*************** typedef struct ControlFileData
*** 207,212 ****
--- 207,213 ----
uint32 indexMaxKeys; /* max number of columns in an index */
uint32 toast_max_chunk_size; /* chunk size in TOAST tables */
+ uint32 loblksize; /* chunk size in pg_largeobject */
/* flag indicating internal format of timestamp, interval, time */
bool enableIntTimes; /* int64 storage enabled? */
diff --git a/src/include/storage/large_object.h b/src/include/storage/large_object.h
index 0d81a4b..30438a9 100644
*** a/src/include/storage/large_object.h
--- b/src/include/storage/large_object.h
*************** typedef struct LargeObjectDesc
*** 66,71 ****
--- 66,73 ----
* Also, it seems to be a smart move to make the page size be a power of 2,
* since clients will often be written to send data in power-of-2 blocks.
* This avoids unnecessary tuple updates caused by partial-page writes.
+ *
+ * NB: Changing LOBLKSIZE requires an initdb.
*/
#define LOBLKSIZE (BLCKSZ / 4)
On Wed, Jun 4, 2014 at 06:57:31PM -0400, Tom Lane wrote: > Stephen Frost <sfrost@snowman.net> writes: > > * Tom Lane (tgl@sss.pgh.pa.us) wrote: > >> There are at least two places in inv_api.c where we have > >> "Assert(pagelen <= LOBLKSIZE)" that is protecting a subsequent memcpy > >> into a local variable of size LOBLKSIZE, so that the only thing standing > >> between us and a stack-smash security issue that's trivially exploitable > >> in production builds is that on-disk data conforms to our expectation > >> about LOBLKSIZE. I think it's definitely worth promoting these checks > >> to regular runtime-if-test-and-elog. > > > Agreed. Promoting that to a run-time check seems well worth it to me. > > Here's a draft patch for this. Barring objections I'll commit the whole > thing to HEAD, and the inv_api.c changes to the back branches as well. Uh, I think pg_upgrade needs to check that they match too. -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + Everyone has their own god. +
Bruce Momjian <bruce@momjian.us> writes:
> Uh, I think pg_upgrade needs to check that they match too.
Possibly. What do you think it should do when examining a pg_control
version that lacks the field?
regards, tom lane
On Tue, Jun 17, 2014 at 12:28:46PM -0400, Tom Lane wrote: > Bruce Momjian <bruce@momjian.us> writes: > > Uh, I think pg_upgrade needs to check that they match too. > > Possibly. What do you think it should do when examining a pg_control > version that lacks the field? Good question. I have existing cases where fields were removed, but not ones that were added. As we have no way to query the old cluster's value for LOBLKSIZE, I think I will just add code to compare them if they _both_ exist. -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + Everyone has their own god. +
Bruce Momjian wrote: > On Tue, Jun 17, 2014 at 12:28:46PM -0400, Tom Lane wrote: > > Bruce Momjian <bruce@momjian.us> writes: > > > Uh, I think pg_upgrade needs to check that they match too. > > > > Possibly. What do you think it should do when examining a pg_control > > version that lacks the field? > > Good question. I have existing cases where fields were removed, but not > ones that were added. As we have no way to query the old cluster's > value for LOBLKSIZE, I think I will just add code to compare them if > they _both_ exist. Can't you compare it to the historic default value? I mean, add an assumption that people thus far has never tweaked it. -- Álvaro Herrera http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services
On Tue, Jun 17, 2014 at 03:55:02PM -0400, Alvaro Herrera wrote: > Bruce Momjian wrote: > > On Tue, Jun 17, 2014 at 12:28:46PM -0400, Tom Lane wrote: > > > Bruce Momjian <bruce@momjian.us> writes: > > > > Uh, I think pg_upgrade needs to check that they match too. > > > > > > Possibly. What do you think it should do when examining a pg_control > > > version that lacks the field? > > > > Good question. I have existing cases where fields were removed, but not > > ones that were added. As we have no way to query the old cluster's > > value for LOBLKSIZE, I think I will just add code to compare them if > > they _both_ exist. > > Can't you compare it to the historic default value? I mean, add an > assumption that people thus far has never tweaked it. Well, if they did tweak it, then they would be unable to use pg_upgrade because it would complain about a mismatch if they actually matched the old and new servers. -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + Everyone has their own god. +
Bruce Momjian <bruce@momjian.us> writes:
> On Tue, Jun 17, 2014 at 03:55:02PM -0400, Alvaro Herrera wrote:
>> Can't you compare it to the historic default value? I mean, add an
>> assumption that people thus far has never tweaked it.
> Well, if they did tweak it, then they would be unable to use pg_upgrade
> because it would complain about a mismatch if they actually matched the
> old and new servers.
What about comparing to the symbolic value LOBLKSIZE? This would make
pg_upgrade assume that the old installation had been tweaked the same
as in its own build. This ends up being the same as what you said,
ie, effectively no comparison ... but it might be less complicated to
code/understand.
regards, tom lane
On Tue, Jun 17, 2014 at 07:12:16PM -0400, Tom Lane wrote: > Bruce Momjian <bruce@momjian.us> writes: > > On Tue, Jun 17, 2014 at 03:55:02PM -0400, Alvaro Herrera wrote: > >> Can't you compare it to the historic default value? I mean, add an > >> assumption that people thus far has never tweaked it. > > > Well, if they did tweak it, then they would be unable to use pg_upgrade > > because it would complain about a mismatch if they actually matched the > > old and new servers. > > What about comparing to the symbolic value LOBLKSIZE? This would make > pg_upgrade assume that the old installation had been tweaked the same > as in its own build. This ends up being the same as what you said, > ie, effectively no comparison ... but it might be less complicated to > code/understand. OK, assume the compiled-in default is the value for an old cluster that has no value --- yeah, I could do that. -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + Everyone has their own god. +
On Tue, Jun 17, 2014 at 8:46 PM, Bruce Momjian <bruce@momjian.us> wrote: > On Tue, Jun 17, 2014 at 07:12:16PM -0400, Tom Lane wrote: >> Bruce Momjian <bruce@momjian.us> writes: >> > On Tue, Jun 17, 2014 at 03:55:02PM -0400, Alvaro Herrera wrote: >> >> Can't you compare it to the historic default value? I mean, add an >> >> assumption that people thus far has never tweaked it. >> >> > Well, if they did tweak it, then they would be unable to use pg_upgrade >> > because it would complain about a mismatch if they actually matched the >> > old and new servers. >> >> What about comparing to the symbolic value LOBLKSIZE? This would make >> pg_upgrade assume that the old installation had been tweaked the same >> as in its own build. This ends up being the same as what you said, >> ie, effectively no comparison ... but it might be less complicated to >> code/understand. > > OK, assume the compiled-in default is the value for an old cluster that > has no value --- yeah, I could do that. I'm not really sure why this is better than Bruce's original proposal, though. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
Robert Haas <robertmhaas@gmail.com> writes:
> On Tue, Jun 17, 2014 at 8:46 PM, Bruce Momjian <bruce@momjian.us> wrote:
>> On Tue, Jun 17, 2014 at 07:12:16PM -0400, Tom Lane wrote:
>>> What about comparing to the symbolic value LOBLKSIZE? This would make
>>> pg_upgrade assume that the old installation had been tweaked the same
>>> as in its own build. This ends up being the same as what you said,
>>> ie, effectively no comparison ... but it might be less complicated to
>>> code/understand.
>> OK, assume the compiled-in default is the value for an old cluster that
>> has no value --- yeah, I could do that.
> I'm not really sure why this is better than Bruce's original proposal, though.
The net behavior would be the same, but I thought it might be easier to
code by thinking of it this way. Or maybe it wouldn't --- it's just a
suggestion.
regards, tom lane
On Wed, Jun 18, 2014 at 2:55 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote: > Robert Haas <robertmhaas@gmail.com> writes: >> On Tue, Jun 17, 2014 at 8:46 PM, Bruce Momjian <bruce@momjian.us> wrote: >>> On Tue, Jun 17, 2014 at 07:12:16PM -0400, Tom Lane wrote: >>>> What about comparing to the symbolic value LOBLKSIZE? This would make >>>> pg_upgrade assume that the old installation had been tweaked the same >>>> as in its own build. This ends up being the same as what you said, >>>> ie, effectively no comparison ... but it might be less complicated to >>>> code/understand. > >>> OK, assume the compiled-in default is the value for an old cluster that >>> has no value --- yeah, I could do that. > >> I'm not really sure why this is better than Bruce's original proposal, though. > > The net behavior would be the same, but I thought it might be easier to > code by thinking of it this way. Or maybe it wouldn't --- it's just a > suggestion. Well, the difference is that if we just don't check it, there can never be an error. Basically, it's the user's job to DTRT. If we check it against some semi-arbitrary value, we'll catch the case where the old cluster was modified with a custom setting and the new one was not - but couldn't we also get false positives under obscure circumstances? -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
Robert Haas <robertmhaas@gmail.com> writes:
> On Wed, Jun 18, 2014 at 2:55 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> The net behavior would be the same, but I thought it might be easier to
>> code by thinking of it this way. Or maybe it wouldn't --- it's just a
>> suggestion.
> Well, the difference is that if we just don't check it, there can
> never be an error. Basically, it's the user's job to DTRT. If we
> check it against some semi-arbitrary value, we'll catch the case where
> the old cluster was modified with a custom setting and the new one was
> not - but couldn't we also get false positives under obscure
> circumstances?
Huh? What we'd be checking is the LOBLKSIZE compiled into pg_upgrade
versus that stored into pg_control by the new postmaster. If those
are different, then pg_upgrade didn't come from the same build as the
new postmaster, which is already a pretty hazardous situation (especially
if the user is fooling with low-level stuff like this).
regards, tom lane
On Wed, Jun 18, 2014 at 3:28 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote: > Robert Haas <robertmhaas@gmail.com> writes: >> On Wed, Jun 18, 2014 at 2:55 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote: >>> The net behavior would be the same, but I thought it might be easier to >>> code by thinking of it this way. Or maybe it wouldn't --- it's just a >>> suggestion. > >> Well, the difference is that if we just don't check it, there can >> never be an error. Basically, it's the user's job to DTRT. If we >> check it against some semi-arbitrary value, we'll catch the case where >> the old cluster was modified with a custom setting and the new one was >> not - but couldn't we also get false positives under obscure >> circumstances? > > Huh? What we'd be checking is the LOBLKSIZE compiled into pg_upgrade > versus that stored into pg_control by the new postmaster. If those > are different, then pg_upgrade didn't come from the same build as the > new postmaster, which is already a pretty hazardous situation (especially > if the user is fooling with low-level stuff like this). OK, I agree that checking that wouldn't hurt anything. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
On Tue, Jun 17, 2014 at 08:46:00PM -0400, Bruce Momjian wrote: > On Tue, Jun 17, 2014 at 07:12:16PM -0400, Tom Lane wrote: > > Bruce Momjian <bruce@momjian.us> writes: > > > On Tue, Jun 17, 2014 at 03:55:02PM -0400, Alvaro Herrera wrote: > > >> Can't you compare it to the historic default value? I mean, add an > > >> assumption that people thus far has never tweaked it. > > > > > Well, if they did tweak it, then they would be unable to use pg_upgrade > > > because it would complain about a mismatch if they actually matched the > > > old and new servers. > > > > What about comparing to the symbolic value LOBLKSIZE? This would make > > pg_upgrade assume that the old installation had been tweaked the same > > as in its own build. This ends up being the same as what you said, > > ie, effectively no comparison ... but it might be less complicated to > > code/understand. > > OK, assume the compiled-in default is the value for an old cluster that > has no value --- yeah, I could do that. Turns out I already had values that could be missing in the old cluster, so I just used the same format for this, rather than testing for LOBLKSIZE. Attached patch applied. -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + Everyone has their own god. +