Обсуждение: xlogdump
There's an xlogdump project on pgfoundry. However it suffers from perennial bitrot as it has to maintain its own table of xlog record types and code to decode each xlog record type. Earlier I modified xlogdump to generate a CSV loadable data set so I could do some basic analysis and see what types of operations are generating the most wal traffic. But I found it had bitrotted and needed some attention to bring it up to date. Again now I wanted to repeat that analysis to measure the effect HOT has had on WAL traffic. And again now I find it has bitrotted, not least because of HOT of course... I think this module should be rewritten to depend more closely on the Postgres source files. What I'm doing now is making an SRF in the style of the pageinspect module which will read an arbitrary wal file and generate records directly. This has a big disadvantage compared to the original approach, namely that you need a functioning Postgres instance of the same version to dissect wal records. But it also has a big advantage, namely that it will always be in sync. It will just use the same RmgrTable to find the rm_name and call the rm_desc method to decode the record. The result might not be quite as or dense as the rm_desc function is meant for debugging messages. We could address that sometime with a new method if we wanted to. I'm thinking of actually dropping it directly into the pageinspect contrib module. It's not quite an exact fit but it doesn't seem to deserve it's own contrib module and it's likely to suffer the same bitrot problem if it lives in pgfoundry. Incidentally I would like to call xlog.c:RecordIsValid() which is currently a static function. Any objection to exporting it? It doesn't depend on any external xlog.c state. -- Gregory Stark EnterpriseDB http://www.enterprisedb.com Ask me about EnterpriseDB's On-Demand Production Tuning
Gregory Stark wrote: > There's an xlogdump project on pgfoundry. However it suffers from perennial > bitrot as it has to maintain its own table of xlog record types and code to > decode each xlog record type. > > ... > > I think this module should be rewritten to depend more closely on the Postgres > source files. What I'm doing now is making an SRF in the style of the > pageinspect module which will read an arbitrary wal file and generate records > directly. > > This has a big disadvantage compared to the original approach, namely that you > need a functioning Postgres instance of the same version to dissect wal > records. > > But it also has a big advantage, namely that it will always be in sync. It > will just use the same RmgrTable to find the rm_name and call the rm_desc > method to decode the record. The result might not be quite as or dense as the > rm_desc function is meant for debugging messages. We could address that > sometime with a new method if we wanted to. Would it still be possible to compile it as a stand-alone program, using the backend source files? It would be a hack, we just went through some effort to clean up references to server private header files from ecpg and initdb, but it feels a lot nicer to use as a standalone program than requiring a running postgres instance. How much infrastructure would you need to call rm_name and rm_desc from a standalone program? palloc and friends, I presume, What else? > I'm thinking of actually dropping it directly into the pageinspect contrib > module. It's not quite an exact fit but it doesn't seem to deserve it's own > contrib module and it's likely to suffer the same bitrot problem if it lives > in pgfoundry. I'd vote for pgfoundry or a new contrib module. It shouldn't suffer from bitrot as easily as what's there now. That was the whole point of switching over to the new approach, right? -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com
Heikki Linnakangas wrote: > Gregory Stark wrote: >> There's an xlogdump project on pgfoundry. However it suffers from >> perennial >> bitrot as it has to maintain its own table of xlog record types and >> code to >> decode each xlog record type. >> >> ... >> >> I think this module should be rewritten to depend more closely on the >> Postgres >> source files. What I'm doing now is making an SRF in the style of the >> pageinspect module which will read an arbitrary wal file and generate >> records >> directly. >> >> This has a big disadvantage compared to the original approach, namely >> that you >> need a functioning Postgres instance of the same version to dissect wal >> records. >> >> But it also has a big advantage, namely that it will always be in >> sync. It >> will just use the same RmgrTable to find the rm_name and call the rm_desc >> method to decode the record. The result might not be quite as or dense >> as the >> rm_desc function is meant for debugging messages. We could address that >> sometime with a new method if we wanted to. > > Would it still be possible to compile it as a stand-alone program, using > the backend source files? It would be a hack, we just went through some > effort to clean up references to server private header files from ecpg > and initdb, but it feels a lot nicer to use as a standalone program than > requiring a running postgres instance. > > How much infrastructure would you need to call rm_name and rm_desc from > a standalone program? palloc and friends, I presume, What else? Do you actually need palloc and friends, or just "something named palloc"? We already have some stuff in src/port that deals with using palloc calls in routines used in frontend programs... //Magnus
--On Freitag, November 02, 2007 10:54:45 +0000 Gregory Stark <stark@enterprisedb.com> wrote: > I think this module should be rewritten to depend more closely on the > Postgres source files. What I'm doing now is making an SRF in the style > of the pageinspect module which will read an arbitrary wal file and > generate records directly. Isn't that already partially implemented in xlogviewer, hosted on the same project: <http://cvs.pgfoundry.org/cgi-bin/cvsweb.cgi/xlogviewer/xlogviewer/ The last time i've used it it had some problems with exhausted memory, but it implements a SRF returning WAL records from a given log segment file. -- Thanks Bernd
On Fri, 2007-11-02 at 10:54 +0000, Gregory Stark wrote: > Incidentally I would like to call xlog.c:RecordIsValid() which is currently a > static function. Any objection to exporting it? It doesn't depend on any > external xlog.c state. You'll have some fun with that because most of the stuff in xlog.c depends upon module-level state. We should change that, but I'd like to see Florian's code go in first then refactor things. If you change those programs too much we'll need completely separate programs for each release, which might be a bad thing. -- Simon Riggs 2ndQuadrant http://www.2ndQuadrant.com
Added to TODO: * Move pgfoundry's xlogdump to /contrib and have it rely more closely on the WAL backend code http://archives.postgresql.org/pgsql-hackers/2007-11/msg00035.php --------------------------------------------------------------------------- Gregory Stark wrote: > > There's an xlogdump project on pgfoundry. However it suffers from perennial > bitrot as it has to maintain its own table of xlog record types and code to > decode each xlog record type. > > Earlier I modified xlogdump to generate a CSV loadable data set so I could do > some basic analysis and see what types of operations are generating the most > wal traffic. But I found it had bitrotted and needed some attention to bring > it up to date. > > Again now I wanted to repeat that analysis to measure the effect HOT has had > on WAL traffic. And again now I find it has bitrotted, not least because of > HOT of course... > > I think this module should be rewritten to depend more closely on the Postgres > source files. What I'm doing now is making an SRF in the style of the > pageinspect module which will read an arbitrary wal file and generate records > directly. > > This has a big disadvantage compared to the original approach, namely that you > need a functioning Postgres instance of the same version to dissect wal > records. > > But it also has a big advantage, namely that it will always be in sync. It > will just use the same RmgrTable to find the rm_name and call the rm_desc > method to decode the record. The result might not be quite as or dense as the > rm_desc function is meant for debugging messages. We could address that > sometime with a new method if we wanted to. > > I'm thinking of actually dropping it directly into the pageinspect contrib > module. It's not quite an exact fit but it doesn't seem to deserve it's own > contrib module and it's likely to suffer the same bitrot problem if it lives > in pgfoundry. > > Incidentally I would like to call xlog.c:RecordIsValid() which is currently a > static function. Any objection to exporting it? It doesn't depend on any > external xlog.c state. > > -- > Gregory Stark > EnterpriseDB http://www.enterprisedb.com > Ask me about EnterpriseDB's On-Demand Production Tuning > > ---------------------------(end of broadcast)--------------------------- > TIP 1: if posting/reading through Usenet, please send an appropriate > subscribe-nomail command to majordomo@postgresql.org so that your > message can get through to the mailing list cleanly -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://postgres.enterprisedb.com + If your life is a hard drive, Christ can be your backup. +