postgresql-server-8.4.4-1PGDG.el5 - ERROR: could not write block 503414 of temporary file: No space le,ft on device

System info: Lancelot 1874-T dual Intel Xeon(Nehalem-DP) SATAII 24GB
RAM, CentOS release 5.4 (Final), postgresql-server-8.4.4-1PGDG

hi,

We installed a fresh version of postgresql-server-8.4.4-1 using RPMs
from PGDG. We are seeing the following errors, if the system trys to do
'anything' sort of update, insert, etc.:

> 2010-07-26 11:06:59 MST  LOG:  database system was shut down at 2010-07-26 10:51:41 MST
> 2010-07-26 11:06:59 MST  LOG:  autovacuum launcher started
> 2010-07-26 11:06:59 MST  LOG:  database system is ready to accept connections
> 2010-07-26 11:08:18 MST science_admin metadataFATAL:  password authentication failed for user "science_admin"
> 2010-07-26 11:09:33 MST postgres postgresFATAL:  password authentication failed for user "postgres"
> 2010-07-26 11:09:55 MST arcsoft arcsoftFATAL:  password authentication failed for user "arcsoft"
> 2010-07-26 11:10:00 MST arcsoft arcsoftFATAL:  database "arcsoft" does not exist
> 2010-07-26 11:10:43 MST arcsoft postgresLOG:  statement: ALTER USER postgres WITH PASSWORD 'nsa_core';
> 2010-07-26 11:11:06 MST postgres postgresLOG:  statement: ALTER USER operations WITH PASSWORD 'Mosaic_DHS';
> 2010-07-26 11:11:15 MST postgres postgresLOG:  statement: ALTER USER pipeline WITH PASSWORD 'Mosaic_DHS';
> 2010-07-26 11:11:23 MST postgres postgresLOG:  statement: ALTER USER science_admin WITH PASSWORD 'nsa_devel';
> 2010-07-26 11:11:31 MST postgres postgresLOG:  statement: ALTER USER security_admin WITH PASSWORD 'nsa_devel';
> 2010-07-26 11:11:48 MST postgres postgresLOG:  statement: ALTER USER voi WITH PASSWORD 'nsa_devel';
> 2010-07-26 11:15:24 MST system_admin metadataFATAL:  password authentication failed for user "system_admin"
> 2010-07-26 11:19:36 MST postgres postgresLOG:  statement: ALTER USER system_admin WITH PASSWORD 'nsa_core';
> 2010-07-26 11:21:03 MST postgres postgresLOG:  statement: ALTER USER system_admin WITH PASSWORD 'nsa_devel';
> 2010-07-26 11:21:19 MST system_admin system_adminFATAL:  database "system_admin" does not exist
> 2010-07-26 11:33:33 MST system_admin metadataERROR:  could not write block 503414 of temporary file: No space le
> ft on device
> 2010-07-26 11:33:33 MST system_admin metadataHINT:  Perhaps out of disk space?

The postgresql is installed on /var/lib/pgsql, but it has plenty of space::

> -bash-3.2$ df -h
> Filesystem            Size  Used Avail Use% Mounted on
> /dev/sda3              16G   13G  1.7G  89% /
> /dev/sda8             635G  979M  601G   1% /data0
> /dev/sda7             2.0G  241M  1.7G  13% /home
> /dev/sda5             7.8G  1.5G  5.9G  20% /u1
> /dev/sda2              16G  184M   15G   2% /var/log
> /dev/sda1             487M   22M  440M   5% /boot
> tmpfs                  12G     0   12G   0% /dev/shm
> /dev/md0              2.7T  202M  2.6T   1% /data1


The pg_stat_tmp is hardly growing::

> -bash-3.2$ date
> Mon Jul 26 14:37:38 MST 2010
> -bash-3.2$ pwd
> /var/lib/pgsql/data
> -bash-3.2$ du -ks pg_stat_tmp/*
> 56    pg_stat_tmp/pgstat.stat
> -bash-3.2$ du -ks pg_stat_tmp/*
> 56    pg_stat_tmp/pgstat.stat

In summary:

1) installed new version of postgresql-server-8.4.4-1.
2) restored a binary dump of a database from another system running same
version. The restore succeeded without error.
3) started using the system

We saw this same symptom in an CentOS5.2 machine, but we thought it had
to do with the version of CentOS, so we repeated the steps to a system
with more resources and running CentOS5.4.

We have postgresql-server-8.2.14-1PGDG.rhel5 on other CentOS5.3 systems
and we do not see this error.

I read this post:

> http://archives.postgresql.org/pgsql-performance/2010-04/msg00192.php

I understand that autovacuum is enabled by default, I have not modified
them:

> -bash-3.2$ grep vacuum postgresql.conf
> #vacuum_cost_delay = 0ms        # 0-100 milliseconds
> #vacuum_cost_page_hit = 1        # 0-10000 credits
> #vacuum_cost_page_miss = 10        # 0-10000 credits
> #vacuum_cost_page_dirty = 20        # 0-10000 credits
> #vacuum_cost_limit = 200        # 1-10000 credits
> #autovacuum = on            # Enable autovacuum subprocess?  'on'
> #log_autovacuum_min_duration = -1    # -1 disables, 0 logs all actions and
> #autovacuum_max_workers = 3        # max number of autovacuum subprocesses
> #autovacuum_naptime = 1min        # time between autovacuum runs
> #autovacuum_vacuum_threshold = 50    # min number of row updates before
>                     # vacuum
> #autovacuum_analyze_threshold = 50    # min number of row updates before
> #autovacuum_vacuum_scale_factor = 0.2    # fraction of table size before vacuum
> #autovacuum_analyze_scale_factor = 0.1    # fraction of table size before analyze
> #autovacuum_freeze_max_age = 200000000    # maximum XID age before forced vacuum
> #autovacuum_vacuum_cost_delay = 20ms    # default vacuum cost delay for
>                     # autovacuum, in milliseconds;
>                     # -1 means use vacuum_cost_delay
> #autovacuum_vacuum_cost_limit = -1    # default vacuum cost limit for
>                     # autovacuum, -1 means use
>                     # vacuum_cost_limit
> #vacuum_freeze_min_age = 50000000
> #vacuum_freeze_table_age = 150000000


Your comments/suggestions are appreciated.
--irene
---------------------------------------------------------------------
Irene Barg                    Email:  ibarg@noao.edu
NOAO/AURA Inc.                 http://www.noao.edu/noao/staff/ibarg
950 N. Cherry Ave.            Voice:  520-318-8273
Tucson, AZ  85726 USA           FAX:  520-318-8360
---------------------------------------------------------------------