Обсуждение: Read db files directly

Поиск
Список
Период
Сортировка

Read db files directly

От
"Shen, Mingzuo"
Дата:
Hello,
 
I see some OS files (50+ GB total) but cannot see anything in psql -l.
There seem to be some big tables there.
I wonder if there are some tools that would dump the tables in text.
or, how to make PostgreSQL "connect" to these files again?
Thanks in advance!
 
psql -V
psql (PostgreSQL) 7.2.2
 
the original DBA is no longer "available".
 
 
-rw-------    1 postgres postgres 932921344 May 14 20:43 1076888071
-rw-------    1 postgres postgres     8192 May 14 20:43 1076911034
-rw-------    1 postgres postgres        0 Feb 13  2004 1076911036
-rw-------    1 postgres postgres     8192 Feb 13  2004 1076911038
-rw-------    1 postgres postgres    32768 Jul 22 14:47 1247
-rw-------    1 postgres postgres   360448 Jul 22 14:47 1249
-rw-------    1 postgres postgres   253952 Feb 13  2004 1255
-rw-------    1 postgres postgres    65536 Jul 22 14:51 1259
-rw-------    1 postgres postgres 1073741824 Jun  8  2005 1338809055
-rw-------    1 postgres postgres 1073741824 Jun 11  2005 1338809055.1
-rw-------    1 postgres postgres 1073741824 Jul 11  2005 1338809055.10
-rw-------    1 postgres postgres 837844992 May 14 20:51 1338809055.11
-rw-------    1 postgres postgres 1073741824 Jun 15  2005 1338809055.2
-rw-------    1 postgres postgres 1073741824 Jun 18  2005 1338809055.3
-rw-------    1 postgres postgres 1073741824 Jun 21  2005 1338809055.4
-rw-------    1 postgres postgres 1073741824 Jun 23  2005 1338809055.5
-rw-------    1 postgres postgres 1073741824 Jun 27  2005 1338809055.6
-rw-------    1 postgres postgres 1073741824 Jun 29  2005 1338809055.7
-rw-------    1 postgres postgres 1073741824 Jul  5  2005 1338809055.8
-rw-------    1 postgres postgres 1073741824 Jul  7  2005 1338809055.9
-rw-------    1 postgres postgres 160129024 May 14 20:51 1338809163
-rw-------    1 postgres postgres 140402688 May 14 20:51 1338809165
-rw-------    1 postgres postgres 1073741824 Feb 13  2004 1411100549
-rw-------    1 postgres postgres 1073741824 Feb 13  2004 1411100549.1
-rw-------    1 postgres postgres 139853824 Feb 13  2004 1411100549.2
-rw-------    1 postgres postgres 26640384 Feb 13  2004 1519546462
-rw-------    1 postgres postgres 616554496 Feb 13  2004 1527610042
-rw-------    1 postgres postgres   180224 Feb 13  2004 1527676219
-rw-------    1 postgres postgres 1073741824 Feb 13  2004 1558949218
-rw-------    1 postgres postgres 1073741824 Feb 13  2004 1558949218.1
-rw-------    1 postgres postgres 367534080 Feb 13  2004 1558949218.2
-rw-------    1 postgres postgres 1014079488 Feb 13  2004 1558955870
-rw-------    1 postgres postgres 78544896 May 14 20:55 1560852916
-rw-------    1 postgres postgres  3121152 Apr  5  2004 1573041194
-rw-------    1 postgres postgres  6012928 Apr  6  2004 1573041225
-rw-------    1 postgres postgres   958464 Feb 13  2004 1573041235
-rw-------    1 postgres postgres  1843200 Feb 13  2004 1573041254
-rw-------    1 postgres postgres  7020544 Feb 13  2004 1600656072
-rw-------    1 postgres postgres 11509760 Feb 13  2004 1600927397
-rw-------    1 postgres postgres 43499520 Dec 16  2004 1600949827
-rw-------    1 postgres postgres 1073741824 Feb 13  2004 1637104512
-rw-------    1 postgres postgres 1073741824 Feb 13  2004 1637104512.1
-rw-------    1 postgres postgres 1073741824 Feb 13  2004 1637104512.2
-rw-------    1 postgres postgres 1073741824 Feb 13  2004 1637104512.3
-rw-------    1 postgres postgres 1073741824 Feb 13  2004 1637104512.4
-rw-------    1 postgres postgres 563847168 Feb 13  2004 1637104512.5
-rw-------    1 postgres postgres 1073741824 Feb 13  2004 1637110994
-rw-------    1 postgres postgres 841973760 Feb 13  2004 1637110994.1
-rw-------    1 postgres postgres        0 Feb 13  2004 16384
-rw-------    1 postgres postgres        0 Feb 13  2004 16386
-rw-------    1 postgres postgres        0 Feb 13  2004 16388
-rw-------    1 postgres postgres    16384 Jul 22 14:51 16390
-rw-------    1 postgres postgres    81920 Feb 13  2004 16392
-rw-------    1 postgres postgres     8192 Feb 13  2004 16394
-rw-------    1 postgres postgres     8192 Feb 13  2004 16396
-rw-------    1 postgres postgres    16384 Feb 13  2004 16398
-rw-------    1 postgres postgres     8192 Feb 13  2004 16400
-rw-------    1 postgres postgres     8192 Feb 13  2004 16402
-rw-------    1 postgres postgres        0 Feb 13  2004 16404
-rw-------    1 postgres postgres     8192 Feb 13  2004 16406
-rw-------    1 postgres postgres   270336 May 14 20:42 16408
-rw-------    1 postgres postgres    32768 Feb 13  2004 16410
-rw-------    1 postgres postgres     8192 Feb 13  2004 16412
-rw-------    1 postgres postgres        0 Feb 13  2004 16414
-rw-------    1 postgres postgres    98304 Feb 13  2004 16416
-rw-------    1 postgres postgres    16384 Feb 13  2004 16418
-rw-------    1 postgres postgres    16384 Feb 13  2004 16419
-rw-------    1 postgres postgres    16384 Feb 13  2004 16420
-rw-------    1 postgres postgres    16384 Feb 13  2004 16421
-rw-------    1 postgres postgres    16384 Feb 13  2004 16422
-rw-------    1 postgres postgres    16384 Feb 13  2004 16423
-rw-------    1 postgres postgres    16384 Feb 13  2004 16424
-rw-------    1 postgres postgres     8192 Feb 13  2004 16425
-rw-------    1 postgres postgres   303104 Jul 22 14:51 16426
-rw-------    1 postgres postgres   131072 Jul 22 14:51 16427
-rw-------    1 postgres postgres    16384 Jul 22 14:51 16428
-rw-------    1 postgres postgres    49152 Jul 22 14:51 16429
-rw-------    1 postgres postgres    57344 Feb 13  2004 16432
-rw-------    1 postgres postgres    16384 May 14 20:42 16435
-rw-------    1 postgres postgres    16384 May 14 20:42 16436
-rw-------    1 postgres postgres     8192 Feb 13  2004 16437
-rw-------    1 postgres postgres    16384 Feb 13  2004 16438
-rw-------    1 postgres postgres    16384 Feb 13  2004 16439
-rw-------    1 postgres postgres     8192 Feb 13  2004 16440
-rw-------    1 postgres postgres    16384 Feb 13  2004 16441
-rw-------    1 postgres postgres    16384 Feb 13  2004 16442
-rw-------    1 postgres postgres    32768 Feb 13  2004 16443
-rw-------    1 postgres postgres    65536 Feb 13  2004 16444
-rw-------    1 postgres postgres    49152 Feb 13  2004 16445
-rw-------    1 postgres postgres   237568 Feb 13  2004 16446
-rw-------    1 postgres postgres     8192 Feb 13  2004 16447
-rw-------    1 postgres postgres    16384 Feb 13  2004 16448
-rw-------    1 postgres postgres    16384 Feb 13  2004 16449
-rw-------    1 postgres postgres    32768 Mar 25  2004 16452
-rw-------    1 postgres postgres    16384 Feb 13  2004 16453
-rw-------    1 postgres postgres    16384 Feb 13  2004 16454
-rw-------    1 postgres postgres    16384 Feb 13  2004 16455
-rw-------    1 postgres postgres    16384 Feb 13  2004 16456
-rw-------    1 postgres postgres    16384 Jul 22 14:51 16457
-rw-------    1 postgres postgres    32768 Jul 22 14:51 16458
-rw-------    1 postgres postgres        0 Feb 13  2004 16460
-rw-------    1 postgres postgres     8192 Feb 13  2004 16462
-rw-------    1 postgres postgres        0 Feb 13  2004 16463
-rw-------    1 postgres postgres     8192 Feb 13  2004 16465
-rw-------    1 postgres postgres        0 Feb 13  2004 16466
-rw-------    1 postgres postgres     8192 Feb 13  2004 16468
-rw-------    1 postgres postgres        0 Feb 13  2004 16469
-rw-------    1 postgres postgres     8192 Feb 13  2004 16471
-rw-------    1 postgres postgres    16384 Feb 13  2004 16472
-rw-------    1 postgres postgres    16384 Feb 13  2004 16474
-rw-------    1 postgres postgres        0 Feb 13  2004 16475
-rw-------    1 postgres postgres     8192 Feb 13  2004 16477
-rw-------    1 postgres postgres 1073741824 May 24  2004 1715778033
-rw-------    1 postgres postgres 1073741824 May 24  2004 1715778033.1
-rw-------    1 postgres postgres 211279872 May 14 21:15 1715778033.10
-rw-------    1 postgres postgres 1073741824 May 24  2004 1715778033.2
-rw-------    1 postgres postgres 1073741824 May 24  2004 1715778033.3
-rw-------    1 postgres postgres 1073741824 May 24  2004 1715778033.4
-rw-------    1 postgres postgres 1073741824 May 24  2004 1715778033.5
-rw-------    1 postgres postgres 1073741824 May 24  2004 1715778033.6
-rw-------    1 postgres postgres 1073741824 May 24  2004 1715778033.7
-rw-------    1 postgres postgres 1073741824 May 14 21:13 1715778033.8
-rw-------    1 postgres postgres 1073741824 May 14 21:15 1715778033.9
-rw-------    1 postgres postgres 1073741824 Feb 13  2004 1829259384
-rw-------    1 postgres postgres 1073741824 Feb 13  2004 1829259384.1
-rw-------    1 postgres postgres 247996416 Feb 13  2004 1829259384.2
-rw-------    1 postgres postgres 730816512 Feb 13  2004 1837299515
-rw-------    1 postgres postgres 75956224 Feb 13  2004 1934262674
-rw-------    1 postgres postgres 1026228224 May 14 21:07 2140124652
-rw-------    1 postgres postgres 196026368 May 14 21:06 2149234748
-rw-------    1 postgres postgres 171352064 May 14 21:07 2149300741
-rw-------    1 postgres postgres     8192 Mar  5  2004 2156705278
-rw-------    1 postgres postgres   434176 Mar  5  2004 2156715705
-rw-------    1 postgres postgres 355672064 Mar 15  2004 2253565510
-rw-------    1 postgres postgres     8192 May 24  2004 3475308772
-rw-------    1 postgres postgres 105922560 May 24  2004 3475503001
-rw-------    1 postgres postgres 22298624 May 24  2004 3476856707
-rw-------    1 postgres postgres 1073741824 May 24  2004 3476895975
-rw-------    1 postgres postgres 1073741824 May 24  2004 3476895975.1
-rw-------    1 postgres postgres 1073741824 May 24  2004 3476895975.2
-rw-------    1 postgres postgres 1073741824 May 24  2004 3476895975.3
-rw-------    1 postgres postgres 1073741824 May 24  2004 3476895975.4
-rw-------    1 postgres postgres 1073741824 May 24  2004 3476895975.5
-rw-------    1 postgres postgres 1030725632 May 24  2004 3476895975.6
-rw-------    1 postgres postgres 383729664 May 27  2004 3625785642
-rw-------    1 postgres postgres 1073741824 Jun  2  2004 3667355595
-rw-------    1 postgres postgres 1073741824 Jun  2  2004 3667355595.1
-rw-------    1 postgres postgres 1073741824 Jun  4  2004 3667355595.2
-rw-------    1 postgres postgres 1073741824 Jun  4  2004 3667355595.3
-rw-------    1 postgres postgres 1073741824 Jun  4  2004 3667355595.4
-rw-------    1 postgres postgres 835903488 May 14 21:25 3667355595.5
-rw-------    1 postgres postgres     1800 May 15 21:44 pg_internal.init
drwx------    2 postgres postgres     4096 Jul 16 20:40 pgsql_tmp
-rw-------    1 postgres postgres        4 Feb 13  2004 PG_VERSION

Re: Read db files directly

От
"Aaron Bono"
Дата:
On 7/22/06, Shen, Mingzuo <mshen@bcm.edu> wrote:
Hello,
 
I see some OS files (50+ GB total) but cannot see anything in psql -l.
There seem to be some big tables there.
I wonder if there are some tools that would dump the tables in text.
or, how to make PostgreSQL "connect" to these files again?

You can dump the whole database contents with pg_dump.  This dumps your whole database with data into a text file.  Is this what you mean?

==================================================================
   Aaron Bono
   Aranya Software Technologies, Inc.
   http://www.aranya.com
==================================================================

Re: Read db files directly

От
Mingzuo Shen
Дата:
Thanks Aaron.

No. pg_dump, psql do not "see" those files.
As I said, psql -l shows "nothing".
pg_dumpall outputs nothing.

strings 3476895975

for example, does show a lot of text,
some of them seem to be actual data
that I am interested in extracting.

If I create a new db and create new tables
in the new db, then they work as usual.



--- Aaron Bono <postgresql@aranya.com> wrote:

> On 7/22/06, Shen, Mingzuo <mshen@bcm.edu> wrote:
> >
> >  Hello,
> >
> > I see some OS files (50+ GB total) but cannot see
> anything in psql -l.
> > There seem to be some big tables there.
> > I wonder if there are some tools that would dump
> the tables in text.
> > or, how to make PostgreSQL "connect" to these
> files again?
> >
>
> You can dump the whole database contents with
> pg_dump.  This dumps your
> whole database with data into a text file.  Is this
> what you mean?
>
>
==================================================================
>    Aaron Bono
>    Aranya Software Technologies, Inc.
>    http://www.aranya.com
>
==================================================================
>


Re: Read db files directly

От
Scott Marlowe
Дата:
Can you get postgresql to startup?  Are you trying to use pg_ctl to
start it or /etc/init.d/postgresql scripts?

Basically, you need the database up and running first.  When you "psql
and pg_dump don't see those files" what errors are you actually getting?

On Fri, 2006-07-28 at 14:23, Mingzuo Shen wrote:
> Thanks Aaron.
>
> No. pg_dump, psql do not "see" those files.
> As I said, psql -l shows "nothing".
> pg_dumpall outputs nothing.
>
> strings 3476895975
>
> for example, does show a lot of text,
> some of them seem to be actual data
> that I am interested in extracting.
>
> If I create a new db and create new tables
> in the new db, then they work as usual.
>
>
>
> --- Aaron Bono <postgresql@aranya.com> wrote:
>
> > On 7/22/06, Shen, Mingzuo <mshen@bcm.edu> wrote:
> > >
> > >  Hello,
> > >
> > > I see some OS files (50+ GB total) but cannot see
> > anything in psql -l.
> > > There seem to be some big tables there.
> > > I wonder if there are some tools that would dump
> > the tables in text.
> > > or, how to make PostgreSQL "connect" to these
> > files again?
> > >
> >
> > You can dump the whole database contents with
> > pg_dump.  This dumps your
> > whole database with data into a text file.  Is this
> > what you mean?
> >
> >
> ==================================================================
> >    Aaron Bono
> >    Aranya Software Technologies, Inc.
> >    http://www.aranya.com
> >
> ==================================================================
> >
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 5: don't forget to increase your free space map settings

Re: Read db files directly

От
Mingzuo Shen
Дата:
Thanks Scott.

Well, PostgreSQL runs just fine.
As I said, I can create new db, new tables, etc.
everything works just fine for new stuff.

when I say psql shows nothing,
I really mean "nothing".
No error messages.

Of course if I create a new db, psql -l shows my
new db.

psql -l
       List of databases
  Name  |  Owner   | Encoding
--------+----------+-----------
 testdb | postgres | SQL_ASCII
(1 row)


What I am curious about is, what are
in those 50 GB worth of files.
"strings" shows me there are some data that
I might be interested in saving before I erase
this pg instance.


--- Scott Marlowe <smarlowe@g2switchworks.com> wrote:

> Can you get postgresql to startup?  Are you trying
> to use pg_ctl to
> start it or /etc/init.d/postgresql scripts?
>
> Basically, you need the database up and running
> first.  When you "psql
> and pg_dump don't see those files" what errors are
> you actually getting?
>
> On Fri, 2006-07-28 at 14:23, Mingzuo Shen wrote:
> > Thanks Aaron.
> >
> > No. pg_dump, psql do not "see" those files.
> > As I said, psql -l shows "nothing".
> > pg_dumpall outputs nothing.
> >
> > strings 3476895975
> >
> > for example, does show a lot of text,
> > some of them seem to be actual data
> > that I am interested in extracting.
> >
> > If I create a new db and create new tables
> > in the new db, then they work as usual.
> >
> >
> >
> > --- Aaron Bono <postgresql@aranya.com> wrote:
> >
> > > On 7/22/06, Shen, Mingzuo <mshen@bcm.edu> wrote:
> > > >
> > > >  Hello,
> > > >
> > > > I see some OS files (50+ GB total) but cannot
> see
> > > anything in psql -l.
> > > > There seem to be some big tables there.
> > > > I wonder if there are some tools that would
> dump
> > > the tables in text.
> > > > or, how to make PostgreSQL "connect" to these
> > > files again?
> > > >
> > >
> > > You can dump the whole database contents with
> > > pg_dump.  This dumps your
> > > whole database with data into a text file.  Is
> this
> > > what you mean?
> > >
> > >
> >
>
==================================================================
> > >    Aaron Bono
> > >    Aranya Software Technologies, Inc.
> > >    http://www.aranya.com
> > >
> >
>
==================================================================
> > >
> >
> >
> > ---------------------------(end of
> broadcast)---------------------------
> > TIP 5: don't forget to increase your free space
> map settings
>


Re: Read db files directly

От
Scott Marlowe
Дата:
It sounds like the current postgresql is running in one directory, and
you're looking in another directory.  If you can see how postgresql was
started, does it have a -D switch that shows the directory?  My guess is
you could chmod 000 the master directory you're looking at right now and
postgresql could still startup, because it's not where you think it is.

On Fri, 2006-07-28 at 14:43, Mingzuo Shen wrote:
> Thanks Scott.
>
> Well, PostgreSQL runs just fine.
> As I said, I can create new db, new tables, etc.
> everything works just fine for new stuff.
>
> when I say psql shows nothing,
> I really mean "nothing".
> No error messages.
>
> Of course if I create a new db, psql -l shows my
> new db.
>
> psql -l
>        List of databases
>   Name  |  Owner   | Encoding
> --------+----------+-----------
>  testdb | postgres | SQL_ASCII
> (1 row)
>
>
> What I am curious about is, what are
> in those 50 GB worth of files.
> "strings" shows me there are some data that
> I might be interested in saving before I erase
> this pg instance.
>
>
> --- Scott Marlowe <smarlowe@g2switchworks.com> wrote:
>
> > Can you get postgresql to startup?  Are you trying
> > to use pg_ctl to
> > start it or /etc/init.d/postgresql scripts?
> >
> > Basically, you need the database up and running
> > first.  When you "psql
> > and pg_dump don't see those files" what errors are
> > you actually getting?
> >
> > On Fri, 2006-07-28 at 14:23, Mingzuo Shen wrote:
> > > Thanks Aaron.
> > >
> > > No. pg_dump, psql do not "see" those files.
> > > As I said, psql -l shows "nothing".
> > > pg_dumpall outputs nothing.
> > >
> > > strings 3476895975
> > >
> > > for example, does show a lot of text,
> > > some of them seem to be actual data
> > > that I am interested in extracting.
> > >
> > > If I create a new db and create new tables
> > > in the new db, then they work as usual.
> > >
> > >
> > >
> > > --- Aaron Bono <postgresql@aranya.com> wrote:
> > >
> > > > On 7/22/06, Shen, Mingzuo <mshen@bcm.edu> wrote:
> > > > >
> > > > >  Hello,
> > > > >
> > > > > I see some OS files (50+ GB total) but cannot
> > see
> > > > anything in psql -l.
> > > > > There seem to be some big tables there.
> > > > > I wonder if there are some tools that would
> > dump
> > > > the tables in text.
> > > > > or, how to make PostgreSQL "connect" to these
> > > > files again?
> > > > >
> > > >
> > > > You can dump the whole database contents with
> > > > pg_dump.  This dumps your
> > > > whole database with data into a text file.  Is
> > this
> > > > what you mean?
> > > >
> > > >
> > >
> >
> ==================================================================
> > > >    Aaron Bono
> > > >    Aranya Software Technologies, Inc.
> > > >    http://www.aranya.com
> > > >
> > >
> >
> ==================================================================
> > > >
> > >
> > >
> > > ---------------------------(end of
> > broadcast)---------------------------
> > > TIP 5: don't forget to increase your free space
> > map settings
> >
>

Re: Read db files directly

От
Tom Lane
Дата:
Mingzuo Shen <mzshen@yahoo.com> writes:
> when I say psql shows nothing,
> I really mean "nothing".
> No error messages.

> Of course if I create a new db, psql -l shows my
> new db.

> psql -l
>        List of databases
>   Name  |  Owner   | Encoding
> --------+----------+-----------
>  testdb | postgres | SQL_ASCII
> (1 row)

This sounds like transaction XID wraparound in pg_database.  What
vacuuming policy have you been following?  It'd possibly work to
do "vacuum pg_database" to fix it, but if so you've got other
problems ...

            regards, tom lane

Re: Read db files directly

От
Mingzuo Shen
Дата:
Thanks Scott.
That is a much clearer way of putting it.
That old PostgreSQL runs just fine,
in one place, but I have 50 GB of files in
another place. PostgreSQL is not reading it.
How can I persuade this PostgreSQL,
or any PostgeSQL, to read that 50 GB of files.
Or any independent tool to read the files.

Tom Lane mentioned "vacuum".
If only I knew the database name,
I could try "psql dbname".
But I don't know the database name either.
I did run "vacuum" in my new testdb.

Yeah. I guess the previous DBA put those files
on a different file system,
and then forgot about them, probably with
good reason. But as I said, the previous DBA
is no longer available.

Imagine I send just those files to you,
and you try to get some text out of them.
I do not have the SQL used to create
the tables, no table structures.



--- Scott Marlowe <smarlowe@g2switchworks.com> wrote:

> It sounds like the current postgresql is running in
> one directory, and
> you're looking in another directory.  If you can see
> how postgresql was
> started, does it have a -D switch that shows the
> directory?  My guess is
> you could chmod 000 the master directory you're
> looking at right now and
> postgresql could still startup, because it's not
> where you think it is.


Re: Read db files directly

От
Jeff Frost
Дата:
So what are the two locations in question?  Is one /var/lib/pgsql/data and
another one /usr/local/pgsql/data by chance?

You can start another instance of postmaster in that directory by using:

pg_ctl -D <path to data directory> start

example:

pg_ctl -D /usr/local/pgsql/data start

If you do a ps -ef | grep data, you should probably see something like:

postgres 20991     1  4 13:33 pts/13   00:00:00 /usr/bin/postmaster -p 5432 -D
/var/lib/pgsql/data

which would tell you that the current instance of postgres is running in
/var/lib/pgsql/data and you need to start the other one up to see what's in
the other location.

On Fri, 28 Jul 2006, Mingzuo Shen wrote:

> Thanks Scott.
> That is a much clearer way of putting it.
> That old PostgreSQL runs just fine,
> in one place, but I have 50 GB of files in
> another place. PostgreSQL is not reading it.
> How can I persuade this PostgreSQL,
> or any PostgeSQL, to read that 50 GB of files.
> Or any independent tool to read the files.
>
> Tom Lane mentioned "vacuum".
> If only I knew the database name,
> I could try "psql dbname".
> But I don't know the database name either.
> I did run "vacuum" in my new testdb.
>
> Yeah. I guess the previous DBA put those files
> on a different file system,
> and then forgot about them, probably with
> good reason. But as I said, the previous DBA
> is no longer available.
>
> Imagine I send just those files to you,
> and you try to get some text out of them.
> I do not have the SQL used to create
> the tables, no table structures.
>
>
>
> --- Scott Marlowe <smarlowe@g2switchworks.com> wrote:
>
>> It sounds like the current postgresql is running in
>> one directory, and
>> you're looking in another directory.  If you can see
>> how postgresql was
>> started, does it have a -D switch that shows the
>> directory?  My guess is
>> you could chmod 000 the master directory you're
>> looking at right now and
>> postgresql could still startup, because it's not
>> where you think it is.
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 3: Have you checked our extensive FAQ?
>
>               http://www.postgresql.org/docs/faq
>
>

--
Jeff Frost, Owner     <jeff@frostconsultingllc.com>
Frost Consulting, LLC     http://www.frostconsultingllc.com/
Phone: 650-780-7908    FAX: 650-649-1954

Re: Read db files directly

От
Mingzuo Shen
Дата:
Thanks Jeff!

No, cannot do that.
Because the folder has only the "oid" files.
Don't know that to call them.
All file names are numbers.
Except the three following:
pg_internal.init
pgsql_tmp (empty folder)
PG_VERSION

/seagate400/1061329089 is the actual location
of those 50 GB worth of files.
I have a symlink like so:
/var/lib/pgsql/data/base/1061329089
->
/seagate400/1061329089
Restarted many times.



--- Jeff Frost <jeff@frostconsultingllc.com> wrote:

> So what are the two locations in question?  Is one
> /var/lib/pgsql/data and
> another one /usr/local/pgsql/data by chance?
>
> You can start another instance of postmaster in that
> directory by using:
>
> pg_ctl -D <path to data directory> start
>
> example:
>
> pg_ctl -D /usr/local/pgsql/data start
>
> If you do a ps -ef | grep data, you should probably
> see something like:
>
> postgres 20991     1  4 13:33 pts/13   00:00:00
> /usr/bin/postmaster -p 5432 -D
> /var/lib/pgsql/data
>
> which would tell you that the current instance of
> postgres is running in
> /var/lib/pgsql/data and you need to start the other
> one up to see what's in
> the other location.
>
> On Fri, 28 Jul 2006, Mingzuo Shen wrote:
>
> > Thanks Scott.
> > That is a much clearer way of putting it.
> > That old PostgreSQL runs just fine,
> > in one place, but I have 50 GB of files in
> > another place. PostgreSQL is not reading it.
> > How can I persuade this PostgreSQL,
> > or any PostgeSQL, to read that 50 GB of files.
> > Or any independent tool to read the files.
> >
> > Tom Lane mentioned "vacuum".
> > If only I knew the database name,
> > I could try "psql dbname".
> > But I don't know the database name either.
> > I did run "vacuum" in my new testdb.
> >
> > Yeah. I guess the previous DBA put those files
> > on a different file system,
> > and then forgot about them, probably with
> > good reason. But as I said, the previous DBA
> > is no longer available.
> >
> > Imagine I send just those files to you,
> > and you try to get some text out of them.
> > I do not have the SQL used to create
> > the tables, no table structures.
> >
> >
> >
> > --- Scott Marlowe <smarlowe@g2switchworks.com>
> wrote:
> >
> >> It sounds like the current postgresql is running
> in
> >> one directory, and
> >> you're looking in another directory.  If you can
> see
> >> how postgresql was
> >> started, does it have a -D switch that shows the
> >> directory?  My guess is
> >> you could chmod 000 the master directory you're
> >> looking at right now and
> >> postgresql could still startup, because it's not
> >> where you think it is.
> >
> >
> > ---------------------------(end of
> broadcast)---------------------------
> > TIP 3: Have you checked our extensive FAQ?
> >
> >               http://www.postgresql.org/docs/faq
> >
> >
>
> --
> Jeff Frost, Owner     <jeff@frostconsultingllc.com>
> Frost Consulting, LLC
> http://www.frostconsultingllc.com/
> Phone: 650-780-7908    FAX: 650-649-1954
>


Re: Read db files directly

От
Jeff Frost
Дата:
So, is the working postmaster running in /var/lib/pgsql/data?

Have you taken Tom's advice and run 'VACUUM pg_database' ?  Or even just
'vacuumdb -av' ?

On Fri, 28 Jul 2006, Mingzuo Shen wrote:

> Thanks Jeff!
>
> No, cannot do that.
> Because the folder has only the "oid" files.
> Don't know that to call them.
> All file names are numbers.
> Except the three following:
> pg_internal.init
> pgsql_tmp (empty folder)
> PG_VERSION
>
> /seagate400/1061329089 is the actual location
> of those 50 GB worth of files.
> I have a symlink like so:
> /var/lib/pgsql/data/base/1061329089
> ->
> /seagate400/1061329089
> Restarted many times.
>
>
>
> --- Jeff Frost <jeff@frostconsultingllc.com> wrote:
>
>> So what are the two locations in question?  Is one
>> /var/lib/pgsql/data and
>> another one /usr/local/pgsql/data by chance?
>>
>> You can start another instance of postmaster in that
>> directory by using:
>>
>> pg_ctl -D <path to data directory> start
>>
>> example:
>>
>> pg_ctl -D /usr/local/pgsql/data start
>>
>> If you do a ps -ef | grep data, you should probably
>> see something like:
>>
>> postgres 20991     1  4 13:33 pts/13   00:00:00
>> /usr/bin/postmaster -p 5432 -D
>> /var/lib/pgsql/data
>>
>> which would tell you that the current instance of
>> postgres is running in
>> /var/lib/pgsql/data and you need to start the other
>> one up to see what's in
>> the other location.
>>
>> On Fri, 28 Jul 2006, Mingzuo Shen wrote:
>>
>>> Thanks Scott.
>>> That is a much clearer way of putting it.
>>> That old PostgreSQL runs just fine,
>>> in one place, but I have 50 GB of files in
>>> another place. PostgreSQL is not reading it.
>>> How can I persuade this PostgreSQL,
>>> or any PostgeSQL, to read that 50 GB of files.
>>> Or any independent tool to read the files.
>>>
>>> Tom Lane mentioned "vacuum".
>>> If only I knew the database name,
>>> I could try "psql dbname".
>>> But I don't know the database name either.
>>> I did run "vacuum" in my new testdb.
>>>
>>> Yeah. I guess the previous DBA put those files
>>> on a different file system,
>>> and then forgot about them, probably with
>>> good reason. But as I said, the previous DBA
>>> is no longer available.
>>>
>>> Imagine I send just those files to you,
>>> and you try to get some text out of them.
>>> I do not have the SQL used to create
>>> the tables, no table structures.
>>>
>>>
>>>
>>> --- Scott Marlowe <smarlowe@g2switchworks.com>
>> wrote:
>>>
>>>> It sounds like the current postgresql is running
>> in
>>>> one directory, and
>>>> you're looking in another directory.  If you can
>> see
>>>> how postgresql was
>>>> started, does it have a -D switch that shows the
>>>> directory?  My guess is
>>>> you could chmod 000 the master directory you're
>>>> looking at right now and
>>>> postgresql could still startup, because it's not
>>>> where you think it is.
>>>
>>>
>>> ---------------------------(end of
>> broadcast)---------------------------
>>> TIP 3: Have you checked our extensive FAQ?
>>>
>>>               http://www.postgresql.org/docs/faq
>>>
>>>
>>
>> --
>> Jeff Frost, Owner     <jeff@frostconsultingllc.com>
>> Frost Consulting, LLC
>> http://www.frostconsultingllc.com/
>> Phone: 650-780-7908    FAX: 650-649-1954
>>
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 5: don't forget to increase your free space map settings
>
>

--
Jeff Frost, Owner     <jeff@frostconsultingllc.com>
Frost Consulting, LLC     http://www.frostconsultingllc.com/
Phone: 650-780-7908    FAX: 650-649-1954

Re: Read db files directly

От
Mingzuo Shen
Дата:
Sorry I did not make it clear.

I still do not see the db, tables, etc.

vacuumdb, pg_dumpdball, psql, etc
that requires a normal PostgreSQL do not
see the db.
Tried them all, many times.

--- Jeff Frost <jeff@frostconsultingllc.com> wrote:

> So, is the working postmaster running in
> /var/lib/pgsql/data?
>
> Have you taken Tom's advice and run 'VACUUM
> pg_database' ?  Or even just
> 'vacuumdb -av' ?
>
> On Fri, 28 Jul 2006, Mingzuo Shen wrote:
>
> > Thanks Jeff!
> >
> > No, cannot do that.
> > Because the folder has only the "oid" files.
> > Don't know that to call them.
> > All file names are numbers.
> > Except the three following:
> > pg_internal.init
> > pgsql_tmp (empty folder)
> > PG_VERSION
> >
> > /seagate400/1061329089 is the actual location
> > of those 50 GB worth of files.
> > I have a symlink like so:
> > /var/lib/pgsql/data/base/1061329089
> > ->
> > /seagate400/1061329089
> > Restarted many times.


Re: Read db files directly

От
Andy Shellam
Дата:
I know exactly what you're trying to do - it's a similar thing to
"attaching database files" in MS SQL 2000.  You've got a set of data
files for a single database from a previous PGSQL install and want to
get another PGSQL to see it and use it as a normal DB so you can extract
the data, right?

I'm guessing PGSQL needs an entry in pg_database with an oid value of
what your data directory you want to "attach" is called.  This is
particularly risky, and obviously this shouldn't be done on a production
or mission-critical database cluster.  (I don't even know if it'd work,
but it's how I'd see it based on my MS SQL 2000 knowledge.)

I've just tried it, and it's difficult if not impossible to insert a new
record into pg_database.  Another thing I'd thought of was to create
your own new database, note it's OID ("SELECT * FROM pg_database WHERE
datname = 'new_db_name';"), then remove it's corresponding data
directory and instead symlink it to your /seagate/1061329089 directory,
keeping the symlink name as the oid from the above query (i.e.
/var/lib/pgsql/data/base/<new DB OID>  --> /seagate/1061329089.)

Like I say this is risky, and these are just a couple of ideas I've
thought of.  (Note these are right for an 8.1 server, don't know what
version you're running so they may or may not work.)

I've just tested my above "create new DB and symlink it" method by doing
this:

1. Create new DB called "test_db"
2. Get it's OID using SELECT oid FROM pg_database WHERE datname =
'test_db'; = 65960
3. Remove the <pg_dataroot>/base/65960 directory
4. Find the OID of an existing database using SELECT oid FROM
pg_database WHERE datname = 'another_db'; = 16384
5. In <pg_dataroot>/base, symlink 65960 to 16384
6. In psql, switch to "test_db"  - "\c test_db"
7. Run a SELECT query on a table that exists in another_db - SELECT *
FROM public.alias;
8. Switch back to another_db - "\c test_db"
9. Run the same SELECT query in 7 - SELECT * FROM public.alias;
10. Check the data bought back is the same - YEP

Obviously worthy of note is that the server versions of the one you're
working on, and the one you're data directory is from should be the same
(this is given in PG_VERSION in your databae directory.)

Hope this works for you!

Andy.

Mingzuo Shen wrote:
> Sorry I did not make it clear.
>
> I still do not see the db, tables, etc.
>
> vacuumdb, pg_dumpdball, psql, etc
> that requires a normal PostgreSQL do not
> see the db.
> Tried them all, many times.
>
> --- Jeff Frost <jeff@frostconsultingllc.com> wrote:
>
>
>> So, is the working postmaster running in
>> /var/lib/pgsql/data?
>>
>> Have you taken Tom's advice and run 'VACUUM
>> pg_database' ?  Or even just
>> 'vacuumdb -av' ?
>>
>> On Fri, 28 Jul 2006, Mingzuo Shen wrote:
>>
>>
>>> Thanks Jeff!
>>>
>>> No, cannot do that.
>>> Because the folder has only the "oid" files.
>>> Don't know that to call them.
>>> All file names are numbers.
>>> Except the three following:
>>> pg_internal.init
>>> pgsql_tmp (empty folder)
>>> PG_VERSION
>>>
>>> /seagate400/1061329089 is the actual location
>>> of those 50 GB worth of files.
>>> I have a symlink like so:
>>> /var/lib/pgsql/data/base/1061329089
>>> ->
>>> /seagate400/1061329089
>>> Restarted many times.
>>>
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 4: Have you searched our list archives?
>
>                http://archives.postgresql.org
>
> !DSPAM:14,44cb30f5143291875220417!
>
>
>

--
Andy Shellam <mailto:andy.shellam@mailnetwork.co.uk>,
the Mail Network <http://www.mailnetwork.co.uk/>

NetServe Support - we don't go the extra mile; we go the whole distance!

p: (+44) 0 845 838 0879 / +44 0 7818 000834
w: www.mailnetwork.co.uk <http://www.mailnetwork.co.uk/>
e: support@mailnetwork.co.uk <mailto:support@mailnetwork.co.uk>




Re: Read db files directly

От
Mingzuo Shen
Дата:
Thanks so much Andy!

"attach" is a better word.
I was asking how to make PostgreSQL "connect"
to my old folder (fs dir, file system directory).
Or something that would read a folder directly.

psql -V
psql (PostgreSQL) 7.2.2

Yeah, I tried something like what you descibed
before I asked the list for help.
Create my testdb, find its fs dir name.
Then symlink that to the old fs dir in question.
that did not help.

also tried in my testdb, created a table t1.
Find out its file name,
then "cat" the contents of one of the old files
over the new file.
This time "select * from t1" returns an error.
I suspect that is because the columns of t1
is different from the columns of the old
table. But of course I do not know the columns
of the old table.

So my question is still,
is there something that can read
such fs dir directly, without needing info
from any other table such as pg_database,
pg_class. All you get is the fs dir.
at least to give me the list of tables,
and better yet give me the columns of tables,
and best of all to give me the rows in text.
If I were to write such a program,
where roughly in the PostgreSQL source code
would I want to start looking?


--- Andy Shellam <andy@andycc.net> wrote:

> I know exactly what you're trying to do - it's a
> similar thing to
> "attaching database files" in MS SQL 2000.  You've
> got a set of data
> files for a single database from a previous PGSQL
> install and want to
> get another PGSQL to see it and use it as a normal
> DB so you can extract
> the data, right?
>
> I'm guessing PGSQL needs an entry in pg_database
> with an oid value of
> what your data directory you want to "attach" is
> called.  This is
> particularly risky, and obviously this shouldn't be
> done on a production
> or mission-critical database cluster.  (I don't even
> know if it'd work,
> but it's how I'd see it based on my MS SQL 2000
> knowledge.)
>
> I've just tried it, and it's difficult if not
> impossible to insert a new
> record into pg_database.  Another thing I'd thought
> of was to create
> your own new database, note it's OID ("SELECT * FROM
> pg_database WHERE
> datname = 'new_db_name';"), then remove it's
> corresponding data
> directory and instead symlink it to your
> /seagate/1061329089 directory,
> keeping the symlink name as the oid from the above
> query (i.e.
> /var/lib/pgsql/data/base/<new DB OID>  -->
> /seagate/1061329089.)
>
> Like I say this is risky, and these are just a
> couple of ideas I've
> thought of.  (Note these are right for an 8.1
> server, don't know what
> version you're running so they may or may not work.)
>
> I've just tested my above "create new DB and symlink
> it" method by doing
> this:
>
> 1. Create new DB called "test_db"
> 2. Get it's OID using SELECT oid FROM pg_database
> WHERE datname =
> 'test_db'; = 65960
> 3. Remove the <pg_dataroot>/base/65960 directory
> 4. Find the OID of an existing database using SELECT
> oid FROM
> pg_database WHERE datname = 'another_db'; = 16384
> 5. In <pg_dataroot>/base, symlink 65960 to 16384
> 6. In psql, switch to "test_db"  - "\c test_db"
> 7. Run a SELECT query on a table that exists in
> another_db - SELECT *
> FROM public.alias;
> 8. Switch back to another_db - "\c test_db"
> 9. Run the same SELECT query in 7 - SELECT * FROM
> public.alias;
> 10. Check the data bought back is the same - YEP
>
> Obviously worthy of note is that the server versions
> of the one you're
> working on, and the one you're data directory is
> from should be the same
> (this is given in PG_VERSION in your databae
> directory.)
>
> Hope this works for you!
>
> Andy.