jack <jack4pg@a7q.com> writes:
> I am trying to load about 1.4 billion records into a postgresql table.
> The data is in about 100 individual CSV files which I load individually into the table using the COPY command.
> After the data is loaded I perform updates to the data, and this is when I get errors. Some updates work, some crash.
> I am using psql and an SQL script file (update.sql) to perform the updating (UPDATE table SET field=UPPER(field),
etc.).
> The errors are all "Server closed the connection unexpectedly"
What appears in the postmaster log when this happens?
It would also be instructive to run "top" while you trigger
the problem, and see if there is obvious distress like the
server process's memory consumption ballooning.
It smells like a resource-exhaustion problem. Which probably
shouldn't be happening on a machine with 128G RAM, but maybe
something is misconfigured to give up way below that. In any
case, you'll never find the cause from the client side.
> Last attempts have been with ubuntu 22.04.04 and postgreSQL 16.
16 dot which?
regards, tom lane