Обсуждение: Enhancement to pg_dump

Hi,<br /><br />I'm very new to hacking postgresql but am using on a very big site (<a
href="http://ojp.nationalrail.co.uk">http://ojp.nationalrail.co.uk</a>).One of the issues that we have is moving data
froma live database to a reports one. I've hacked an extra option to pg_dump to delete from tables rather than dropping
them.<br/><br />Once I'm happy with it (I'm a bit rusty at C!), do I post the patch here?<br /><br />Thanks<br /><br
/>Rob<br/><br /> 

On Tue, Nov 25, 2008 at 8:39 PM, Rob Kirkbride <rob.kirkbride@gmail.com> wrote:
> Hi,
>
> I'm very new to hacking postgresql but am using on a very big site
> (http://ojp.nationalrail.co.uk). One of the issues that we have is moving
> data from a live database to a reports one. I've hacked an extra option to
> pg_dump to delete from tables rather than dropping them.

National Rail use Postgres for their journey planner? Cool :-)

> Once I'm happy with it (I'm a bit rusty at C!), do I post the patch here?

Yes (and please add details to
http://wiki.postgresql.org/wiki/CommitFestOpen so it doesn't get
lost), but please note that we're in the middle of the final phase of
the development cycle at the moment, so new patches are unlikely to be
looked at for at least a couple of months.


-- 
Dave Page
EnterpriseDB UK:   http://www.enterprisedb.com

Dave,

Ok thanks. Yes, we've got over 1/2 billion rows in one of our tables which is interesting!

Will post back soon.

Rob

"Rob Kirkbride" <rob.kirkbride@gmail.com> writes:

> Once I'm happy with it (I'm a bit rusty at C!), do I post the patch here?

I would say you should post *before* you have a patch you're happy with. As
soon as you have a specific plan of what you want to do it's best to post an
outline of it. That way you at least have a chance of avoiding wasting work in
the wrong direction.

Sometimes things don't really work out that way -- sometimes the plan sounds
good and it only becomes apparent there's a better way later -- but you're
best off getting the best chance you can.

Incidentally, I don't know exactly what the use case you're trying to cover
here is but you should consider looking at TRUNCATE instead of DELETE if
you're really deleting all the records in the table and can accept locking the
table.

--  Gregory Stark EnterpriseDB          http://www.enterprisedb.com Ask me about EnterpriseDB's Slony Replication
support!

OK thanks for the advice.

What I'm trying to overcome is where we've got a long report running and the process that is taking data from the main database cannot complete because of the drop table. I believe a DELETE (and possibly TRUNCATE?) doesn't need an exclusive lock on the table and therefore can continue.

I've introduced a --delete-not-drop option which simply does a DELETE FROM % rather than 'DROP and then CREATE'.

I hope this sounds sensible and I haven't missed something - I'm still learning!

Rob

Rob Kirkbride wrote:
> I've introduced a --delete-not-drop option which simply does a DELETE FROM %
> rather than 'DROP and then CREATE'.

Beware foreign-keys slowing you - TRUNCATE all relevant tables should be
the fastest method if possible.

> I hope this sounds sensible and I haven't missed something - I'm still
> learning!

Have you considered restoring to a completely different database
(report1/report2) and just switching between them?

--  Richard Huxton Archonet Ltd

Richard,

Yes, I've changed it use TRUNCATE rather than DELETE and it's working 
well for us now.

The switching of the database is a good idea - thanks. Unfortunately, 
we've not got enough disk space currently to do that, but if we get 
problems in the future that will definitely be something we'll consider.

Rob

Richard Huxton wrote:
> Rob Kirkbride wrote:
>   
>> I've introduced a --delete-not-drop option which simply does a DELETE FROM %
>> rather than 'DROP and then CREATE'.
>>     
>
> Beware foreign-keys slowing you - TRUNCATE all relevant tables should be
> the fastest method if possible.
>
>   
>> I hope this sounds sensible and I haven't missed something - I'm still
>> learning!
>>     
>
> Have you considered restoring to a completely different database
> (report1/report2) and just switching between them?
>
>   

Rob Kirkbride <rob.kirkbride@gmail.com> writes:

> Richard,
>
> Yes, I've changed it use TRUNCATE rather than DELETE and it's working well for
> us now.

I'm a bit surprised actually as it sounded like you were aiming to avoid the
table lock. A TRUNCATE does require an exclusive lock on the table. It still
has advantages over DROP in that there is no window when the table does not
exist and any existing references to the table from views or functions will
continue to function.


--  Gregory Stark EnterpriseDB          http://www.enterprisedb.com Ask me about EnterpriseDB's RemoteDBA services!

I must admit I've not read up on the various locks that are set so that's a good point. Is there a good reference for me to read and understand these?

I'm guessing though that a delete from and then an insert never requires an exclusive lock, what about adding/deleting constraints?

Rob

"Rob Kirkbride" <rob.kirkbride@gmail.com> writes:

> I must admit I've not read up on the various locks that are set so that's a
> good point. Is there a good reference for me to read and understand these?
>
> I'm guessing though that a delete from and then an insert never requires an
> exclusive lock, what about adding/deleting constraints?

There is documentation

http://www.postgresql.org/docs/8.3/static/explicit-locking.html

However I found it very confusing when I was first learning. It's not really
the documentation's fault either, there are just a lot of different lock
levels with a lot of different combinations possible.

All DML, even selects, take a table-level shared lock on the tables involved
which blocks the tables from being dropped or truncated while the query is
running.

DELETE and UPDATE (and SELECT FOR UPDATE) take exclusive row-level locks. A
SELECT can read the old version of the record but another UPDATE will block
until your transaction finishes so it can update the most recent version. But
an update which doesn't need to look at that record won't be affected at all.

TRUNCATE and DROP take exclusive table-level locks which blocks anyone else
from even selecting from the table. It also means they can't proceed until all
queries which have already started reading the table finish.

DROP is still a lot heavier than TRUNCATE because it also has to drop (or
search for and throw an error) anything else dependent on the table. triggers,
views, etc.

--  Gregory Stark EnterpriseDB          http://www.enterprisedb.com Ask me about EnterpriseDB's 24x7 Postgres support!

Gregory Stark wrote:
> There is documentation
>
> http://www.postgresql.org/docs/8.3/static/explicit-locking.html
>
> However I found it very confusing when I was first learning. It's not really
> the documentation's fault either, there are just a lot of different lock
> levels with a lot of different combinations possible.
>
> All DML, even selects, take a table-level shared lock on the tables involved
> which blocks the tables from being dropped or truncated while the query is
> running.
>
> DELETE and UPDATE (and SELECT FOR UPDATE) take exclusive row-level locks. A
> SELECT can read the old version of the record but another UPDATE will block
> until your transaction finishes so it can update the most recent version. But
> an update which doesn't need to look at that record won't be affected at all.
>
> TRUNCATE and DROP take exclusive table-level locks which blocks anyone else
> from even selecting from the table. It also means they can't proceed until all
> queries which have already started reading the table finish.
>
> DROP is still a lot heavier than TRUNCATE because it also has to drop (or
> search for and throw an error) anything else dependent on the table. triggers,
> views, etc.
>
>   

Thanks for that  - it's very useful. As you say I believe the 
documentation is pretty good, it's just that we're not dealing in simple 
issues here.

I definitely think I should do a delete rather than a truncate (or drop) 
in my case.


Regards

Rob